System operational

Past Incidents

1st March 2017

No incidents reported

28th February 2017

No incidents reported

27th February 2017

No incidents reported

26th February 2017

No incidents reported

25th February 2017

No incidents reported

24th February 2017

Servers ISP Power outage

At approx 11:40 am EST our ISP (Cyberwurx) had power outage. UPS held machines up for about 20 minutes, but after that - all power was gone, and everything halted. At approx 12:30 am EST power was restored, machines started booting, but one rack was off the grid.

Sysadmin team is working on getting everything up, checking replication status and sorting all kinds of issues. No ETA on recovery.

EDIT: at approx 6 pm EST all servers and services have been restored and everything is operational. We'll need to revisit every single thing to confirm that everything is up and working, but so far - everything that's crucial for normal operation is alright.

Servers Server failure

One of physical servers had a failure this morning. Given machine is a host for multiple virtual machines for DatingVIP, and this caused misfunctionality for most of services, but mostly:

  • inbox
  • sessions handling
  • database access
  • memcache cache
  • redis storage
  • background processing

All virtual machines were restored within 1 hour, services are up and running, but piled up work has to be done so you might be seeing some delay in seeing stats, reports, etc.

Sorry for any inconvenience.

23rd February 2017

Network DDoS Attack

Our services were down for about 20 minutes starting of 9:40 am EST till 10 EST. One attack also happened last night (at roughly 10:50 pm), and also lasted for about half an hour.

There's a new attack going on currently, sysadmins are on the issue.

EDIT: second attack has concluded, everything is operational again.