We've had a partial outage on various systems today. The issue took some time to identify and to sort out.
We've started experiencing increased load and hanging connections on slave databases. Issue was caused by our "user filtering" feature (used across the platform). Specific type of filter wasn't handled optimally within code, created invalid query which caused database to hang (a1).
Since database was stalling, our webserver and PHP backend were maxing out on available resources (waiting to complete existing requests). That caused sites across the platform to go up and down (as resources became available or exhausted) within some 100 minute time frame.
Issue is resolved and everything is operational.