System operational

Past Incidents

22nd February 2017

No incidents reported

21st February 2017

Payment Wirecard Scheduled Maintenance

Wirecard will perform maintenance work on the SFTP server (sftp.wirecard.com) on the date indicated below. SFTP server will not be available during given time window.

The maintenance work is scheduled for 21.02.17 11:00 to 21.02.17 12:00 Central European Time. (That's from 5:00 AM until 6:00 AM Server time)

20th February 2017

Database (adult) Adult database issues

Adult system (a1) is expiriencing issues because queries that are meant to be executed on slave are (for yet unidentified reason) returning empty results and falling back to master. This causes slowdown on sites (but everything is operational).

EDIT: after a dozen of changes where we've tried to offload work from master as much as possible - situation is looking stable. Response times are back to normal. Our changes might have helped, it's still unclear how situation got sorted, we'll be investigating further and continue to work on improvements.

19th February 2017

No incidents reported

18th February 2017

No incidents reported

17th February 2017

Network Network attack

At roughly 6PM EST time we started expiriencing network issues due to some sort of attack on our load balancer. Our firewall is dropping given requests, but it's too busy doing it's work, which causes service disruption.

We've been down for rougly 20 minutes, attack stopped, but it started happening again. Sysadmin team is on the issue.

EDIT: After another attack that caused about 8 minutes of downtime, we look stable now. One firewall rule was changed from REJECT to DROP, and NTP was disabled. Both of changes helped.

16th February 2017

No incidents reported

15th February 2017

Dating Performance issues

We're seeing NodePing reports about dating being a bit slower than expected.

Some of problematic queries have been moved off from master database to read from Cache and slave instances. This resolved the issue as far as we can see. There's more work to be done, but we shouldn't have any more service disruptions.

Sorry for the inconvinience.