ISP Power outage Friday 24th February 2017 11:40:00


At approx 11:40 am EST our ISP (Cyberwurx) had power outage. UPS held machines up for about 20 minutes, but after that - all power was gone, and everything halted. At approx 12:30 am EST power was restored, machines started booting, but one rack was off the grid.

Sysadmin team is working on getting everything up, checking replication status and sorting all kinds of issues. No ETA on recovery.

EDIT: at approx 6 pm EST all servers and services have been restored and everything is operational. We'll need to revisit every single thing to confirm that everything is up and working, but so far - everything that's crucial for normal operation is alright.