News: 1623373449

UPDATED WITH ROOT CAUSE A single Availability Zone in Amazon Web Services’ EU-CENTRAL-1 Region has experienced a major incident.

The company’s [1]status page says the incident began at 1:24PM PDT (8:30PM UTC) on June 10th and initially caused “connectivity issues for some EC2 instances”.

Half an hour later AWS reported “increased API error rates and latencies for the EC2 APIs and connectivity issues for instances … caused by an increase in ambient temperature within a subsection of the affected Availability Zone.”

[2]

By 2:36PM PDT AWS said temperatures were falling but that network connectivity remained down.

[3]

[4]

But an hour later, the cloud colossus offered the following rather unsettling assessment:

While temperatures continue to return to normal levels, engineers are still not able to enter the affected part of the Availability Zone. We believe that the environment will be safe for re-entry within the next 30 minutes, but are working on recovery remotely at this stage.

A 4:12PM update reported that staff were still unable to enter the site for safety reasons.

At 4:33PM network services were restored, an event AWS said should lead to swift resumption of EC2 instances. A 5:19PM update stated “environmental conditions within the affected Availability Zone have now returned to normal level” and advised users that “The vast majority of affected EC2 instances have now fully recovered but we’re continuing to work through some EBS volumes that continue to experience degraded performance.”

Kinesis Data Streams, Kinesis Firehose, Amazon Relational Database Service, and AWS CloudFormation also wobbled.

[5]

AWS's most recent status update concluded “We will provide further details on the root cause in a subsequent post, but can confirm that there was no fire within the facility.”

Which leaves the question of just what made the data centre too dangerous to enter?

While we lack any evidence on which to base an assertion, The Register has reported on [6]erupting UPSes and [7]tiny puffs of smoke leading to hypoxic gas being released into data centres.

[8]

The whole point of hypoxic gas release into data centres is to deprive fires of oxygen. And as humans need oxygen, it can be a while before engineers can return to a data centre.

The Register mentions this as it fits the facts offered in this incident, and with AWS’s language about “environmental conditions” preventing entry.

We will update this story if new information about this incident comes to hand.

UPDATE 02:45 UCT June 11th. AWS has updated its incident report (and mostly proven our analysis correct) by revealing that the incident was caused by "failure of a control system which disabled multiple air handlers in the affected Availability Zone."

The air handlers cool the data center, so once they stopped working "ambient temperatures began to rise" to unsafe levels, so AWS servers' networking kit shut down.

"Unfortunately, because this issue impacted several redundant network switches, a larger number of EC2 instances in this single Availability Zone lost network connectivity," the update adds.

And now for the bit that we get to be smug about:

"While our operators would normally have been able to restore cooling before impact, a fire suppression system activated inside a section of the affected Availability Zone.

"When this system activates, the data center is evacuated and sealed, and a chemical is dispersed to remove oxygen from the air to extinguish any fire."

AWS staff had to wait for the local fire department to arrive and attest that the building was safe. Once that signoff was secured, AWS says "the building needed to be re-oxygenated before it was safe for engineers to enter the facility and restore the affected networking gear and servers."

Safe working conditions were restored and so was most of the hardware and services.

But it appears some kit was damaged, as AWS stated "A very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved."

The cloud giant also let clients know that the fire suppression system that activated remains disabled. ®

Get our [9]Tech Resources



[1] https://status.aws.amazon.com/#EU_block

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YMLf5TqKW5p-iF0QA2L9CAAAANQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YMLf5TqKW5p-iF0QA2L9CAAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YMLf5TqKW5p-iF0QA2L9CAAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YMLf5TqKW5p-iF0QA2L9CAAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2018/07/16/who_me/

[7] https://www.theregister.com/2014/02/04/loud_data_centre_gas_release_sounds_harm_disks/

[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YMLf5TqKW5p-iF0QA2L9CAAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[9] https://whitepapers.theregister.com/