On April 27, 2025, there was a brief outage within our infrastructure between 11:12 PM and 11:17 PM. The cause was a memory overload on one of our hosts, which became unresponsive. Our monitoring system triggered an alarm within a minute of the outage and notified the responsible on-call staff member.
Timeline of the Incident:
- 11:12 PM – The host becomes unresponsive due to a memory overload.
- 11:13 PM – The responsible on-call staff member begins the investigation.
- 11:14 PM – The monitoring system triggers an alert.
- 11:16 PM – The affected host is identified as the root cause. A live migration of affected virtual machines is initiated. Non-critical VMs are powered down to release resources.
- 11:17 PM – The infrastructure is back online, and the migration is ongoing.
- 11:19 PM – The host is fully back online. A live migration of the remaining VMs is conducted, which may lead to minimal performance dips in the minutes following.
Cause of the Incident: The incident was caused by a memory overload on a host system. This resulted in the host being unable to respond to requests, temporarily affecting parts of the infrastructure.
Remedial Actions: After being alerted by the monitoring system, the host's overload was identified. Non-essential, non-critical VMs were promptly shut down, and a load distribution among the remaining systems was conducted. Simultaneously, a live migration of the affected virtual machines to other hosts was initiated. This quickly alleviated the host and restored the infrastructure.
Reflection and Preventive Measures: To ensure such an incident is avoided in the future, we are implementing the following measures:
- Adjusting Monitoring Thresholds: We will adjust the threshold for RAM consumption in the monitoring system so that an impending overload can be detected earlier, allowing timely response.
- Automatic Load Distribution: Moving forward, we will introduce automatic load distribution for our non-critical VMs to preemptively prevent exceeding critical thresholds.