System operational

Past Incidents

24th February 2017

Servers ISP Power outage

At approx 11:40 am EST our ISP (Cyberwurx) had power outage. UPS held machines up for about 20 minutes, but after that - all power was gone, and everything halted. At approx 12:30 am EST power was restored, machines started booting, but one rack was off the grid.

Sysadmin team is working on getting everything up, checking replication status and sorting all kinds of issues. No ETA on recovery.

EDIT: at approx 6 pm EST all servers and services have been restored and everything is operational. We'll need to revisit every single thing to confirm that everything is up and working, but so far - everything that's crucial for normal operation is alright.

Servers Server failure

One of physical servers had a failure this morning. Given machine is a host for multiple virtual machines for DatingVIP, and this caused misfunctionality for most of services, but mostly:

  • inbox
  • sessions handling
  • database access
  • memcache cache
  • redis storage
  • background processing

All virtual machines were restored within 1 hour, services are up and running, but piled up work has to be done so you might be seeing some delay in seeing stats, reports, etc.

Sorry for any inconvenience.

23rd February 2017

Network DDoS Attack

Our services were down for about 20 minutes starting of 9:40 am EST till 10 EST. One attack also happened last night (at roughly 10:50 pm), and also lasted for about half an hour.

There's a new attack going on currently, sysadmins are on the issue.

EDIT: second attack has concluded, everything is operational again.

22nd February 2017

No incidents reported

21st February 2017

Payment Wirecard Scheduled Maintenance

Wirecard will perform maintenance work on the SFTP server (sftp.wirecard.com) on the date indicated below. SFTP server will not be available during given time window.

The maintenance work is scheduled for 21.02.17 11:00 to 21.02.17 12:00 Central European Time. (That's from 5:00 AM until 6:00 AM Server time)

20th February 2017

Database (adult) Adult database issues

Adult system (a1) is expiriencing issues because queries that are meant to be executed on slave are (for yet unidentified reason) returning empty results and falling back to master. This causes slowdown on sites (but everything is operational).

EDIT: after a dozen of changes where we've tried to offload work from master as much as possible - situation is looking stable. Response times are back to normal. Our changes might have helped, it's still unclear how situation got sorted, we'll be investigating further and continue to work on improvements.

19th February 2017

No incidents reported

18th February 2017

No incidents reported

17th February 2017

Network Network attack

At roughly 6PM EST time we started expiriencing network issues due to some sort of attack on our load balancer. Our firewall is dropping given requests, but it's too busy doing it's work, which causes service disruption.

We've been down for rougly 20 minutes, attack stopped, but it started happening again. Sysadmin team is on the issue.

EDIT: After another attack that caused about 8 minutes of downtime, we look stable now. One firewall rule was changed from REJECT to DROP, and NTP was disabled. Both of changes helped.