System operational

Past Incidents

23rd February 2017

Network DDoS Attack

Our services were down for about 20 minutes starting of 9:40 am EST till 10 EST. One attack also happened last night (at roughly 10:50 pm), and also lasted for about half an hour.

There's a new attack going on currently, sysadmins are on the issue.

EDIT: second attack has concluded, everything is operational again.

22nd February 2017

No incidents reported

21st February 2017

Payment Wirecard Scheduled Maintenance

Wirecard will perform maintenance work on the SFTP server (sftp.wirecard.com) on the date indicated below. SFTP server will not be available during given time window.

The maintenance work is scheduled for 21.02.17 11:00 to 21.02.17 12:00 Central European Time. (That's from 5:00 AM until 6:00 AM Server time)

20th February 2017

Database (adult) Adult database issues

Adult system (a1) is expiriencing issues because queries that are meant to be executed on slave are (for yet unidentified reason) returning empty results and falling back to master. This causes slowdown on sites (but everything is operational).

EDIT: after a dozen of changes where we've tried to offload work from master as much as possible - situation is looking stable. Response times are back to normal. Our changes might have helped, it's still unclear how situation got sorted, we'll be investigating further and continue to work on improvements.

19th February 2017

No incidents reported

18th February 2017

No incidents reported

17th February 2017

Network Network attack

At roughly 6PM EST time we started expiriencing network issues due to some sort of attack on our load balancer. Our firewall is dropping given requests, but it's too busy doing it's work, which causes service disruption.

We've been down for rougly 20 minutes, attack stopped, but it started happening again. Sysadmin team is on the issue.

EDIT: After another attack that caused about 8 minutes of downtime, we look stable now. One firewall rule was changed from REJECT to DROP, and NTP was disabled. Both of changes helped.