System operational

Past Incidents

17th February 2017

Network Network attack

At roughly 6PM EST time we started expiriencing network issues due to some sort of attack on our load balancer. Our firewall is dropping given requests, but it's too busy doing it's work, which causes service disruption.

We've been down for rougly 20 minutes, attack stopped, but it started happening again. Sysadmin team is on the issue.

EDIT: After another attack that caused about 8 minutes of downtime, we look stable now. One firewall rule was changed from REJECT to DROP, and NTP was disabled. Both of changes helped.

16th February 2017

No incidents reported

15th February 2017

Dating Performance issues

We're seeing NodePing reports about dating being a bit slower than expected.

Some of problematic queries have been moved off from master database to read from Cache and slave instances. This resolved the issue as far as we can see. There's more work to be done, but we shouldn't have any more service disruptions.

Sorry for the inconvinience.

14th February 2017

No incidents reported

13th February 2017

No incidents reported

12th February 2017

No incidents reported

11th February 2017

No incidents reported

10th February 2017

Payment RocketGate under DDoS

RocketGate was under DDoS (distributed denial of service) attack at 4:08pm AST. The attack has not affected rebill transactions, only signups.

Database (adult) a1 master change

We've had downtime of 15 minutes due to master change. This was supposed to be without any downtime (and likely would be), but one machine didn't have new network segment setup.

Maintenance will continue during the weekend (reorganization of master/slave hierarchy), but no downtime is expected.