IBM blames ‘external' network provider, incorrect routing, traffic flood for its two-hour cloud outage
(2020/06/11)
- Reference: 1591841696
- News link: https://www.theregister.co.uk/2020/06/11/ibm_cloud_outage_report/
- Source link:
IBM has blamed a third party for [1]yesterday’s hours-long outage of its entire cloud. And while it says no data loss or attack was detected, it's still not a good look: major clouds are supposed to be more resilient than this.
A brief notice on the IT titan's [2]cloud status page offered the following explanation for the breakdown:
IBM is focused on external network provider issues as the cause of the disruption of IBM Cloud services on Tuesday, June 9. All services have been restored.
A detailed root cause analysis is underway. An investigation shows an external network provider flooded the IBM Cloud network with incorrect routing, resulting in severe congestion of traffic and impacting IBM Cloud services and our data centers. Mitigation steps have been taken to prevent a reoccurrence. Root cause analysis has not identified any data loss or cybersecurity issues.
That’s rather vague verbiage but is consistent with the sort of traffic flood that can happen when an inadvertent border gateway protocol (BGP) blunder directs packets to the wrong place. BGP hijacking or misconfiguration is a known problem, and you’d think an outfit like IBM would be alive to that sort of error, and have defenses or countermeasures in place to mitigate it.
Yet IBM may not be great at handling traffic spikes: when it ran Australia’s e-census it [3]mistakenly identified a flood of inbound connections as a denial-of-service attack, and had to pull the plug on a router to sort things out.
IBM to power down Power-powered virtual private cloud, GPU-accelerated options [4]READ MORE
Another possibility is that a supplier to IBM’s cloud messed things up. We know that IBM uses Akamai for its content delivery network, offers Cloudflare-as-a-service, and has an expansive relationship with AT&T. The Register does not suggest any of those companies had a role in the outage, but all do have sufficient scale to play a part in a global outage.
Whatever the cause, hyperscale clouds are supposed to be sufficiently resilient to handle unexpected nastiness of this sort. That IBM succumbed is not reassuring, given the tech goliath says it is now a cloud business. That’s a tricky position to sustain given its cloud remains rather less elegant to operate than rivals’ services. ®
Sponsored: [5]The Service Mesh Era: Architecting, Securing and Managing Microservices with Istio
[1] https://www.theregister.com/2020/06/09/ibm_cloud_outage/
[2] https://cloud.ibm.com/status?query=IBM%20Cloud%20-%20June%209%20Outage%0D%0A%0D&selected=announcement
[3] https://www.theregister.com/2016/08/10/censusfail_not_an_attack_not_a_hack_data_safe_no_heads_to_roll/
[4] https://www.theregister.com/2020/06/04/ibm_vpc_power_shutdown_notice/
[5] https://go.theregister.com/tl/1956/-8476/the-service-mesh-era-architecting-securing-and-managing-microservices-with-istio?td=wptl1956
A brief notice on the IT titan's [2]cloud status page offered the following explanation for the breakdown:
IBM is focused on external network provider issues as the cause of the disruption of IBM Cloud services on Tuesday, June 9. All services have been restored.
A detailed root cause analysis is underway. An investigation shows an external network provider flooded the IBM Cloud network with incorrect routing, resulting in severe congestion of traffic and impacting IBM Cloud services and our data centers. Mitigation steps have been taken to prevent a reoccurrence. Root cause analysis has not identified any data loss or cybersecurity issues.
That’s rather vague verbiage but is consistent with the sort of traffic flood that can happen when an inadvertent border gateway protocol (BGP) blunder directs packets to the wrong place. BGP hijacking or misconfiguration is a known problem, and you’d think an outfit like IBM would be alive to that sort of error, and have defenses or countermeasures in place to mitigate it.
Yet IBM may not be great at handling traffic spikes: when it ran Australia’s e-census it [3]mistakenly identified a flood of inbound connections as a denial-of-service attack, and had to pull the plug on a router to sort things out.
IBM to power down Power-powered virtual private cloud, GPU-accelerated options [4]READ MORE
Another possibility is that a supplier to IBM’s cloud messed things up. We know that IBM uses Akamai for its content delivery network, offers Cloudflare-as-a-service, and has an expansive relationship with AT&T. The Register does not suggest any of those companies had a role in the outage, but all do have sufficient scale to play a part in a global outage.
Whatever the cause, hyperscale clouds are supposed to be sufficiently resilient to handle unexpected nastiness of this sort. That IBM succumbed is not reassuring, given the tech goliath says it is now a cloud business. That’s a tricky position to sustain given its cloud remains rather less elegant to operate than rivals’ services. ®
Sponsored: [5]The Service Mesh Era: Architecting, Securing and Managing Microservices with Istio
[1] https://www.theregister.com/2020/06/09/ibm_cloud_outage/
[2] https://cloud.ibm.com/status?query=IBM%20Cloud%20-%20June%209%20Outage%0D%0A%0D&selected=announcement
[3] https://www.theregister.com/2016/08/10/censusfail_not_an_attack_not_a_hack_data_safe_no_heads_to_roll/
[4] https://www.theregister.com/2020/06/04/ibm_vpc_power_shutdown_notice/
[5] https://go.theregister.com/tl/1956/-8476/the-service-mesh-era-architecting-securing-and-managing-microservices-with-istio?td=wptl1956
Anonymous Coward
They tried that excuse with the Australian Census too.
Never mind its up to them to ensure redundancy, resiliency, scalability, and alternate paths/zones/providers etc etc etc.
IBM has never taken cloud seriously, they bought out (the market leader at the time) Softlayer, because their own offering was rubbish, and pissed the opportunity away, while Amazon and Microsoft built up their market share and capability.
Too late now, except for some large corporates who are "owned" by IBM, no one is interested in IBM's "version" of what they think cloud is and should be.
IBM's Cloud Takeover
Anonymous Coward
Internal emails promote IBM's Takeover of the Cloud... From the article it seems more like IBM folk responsible are trying to Take Cover...
BGP hijack?
Wan't there a BGP hijack of mobile data last year around this time for 2 hours? Odd coincidence. Ah well, at least it wasn't a round 2 on its anniversary...