Google reveals version control and not expecting zero as a value caused Gmail to take an inconvenient early holiday
- Reference: 1608533826
- News link: https://www.theregister.co.uk/2020/12/21/gmail_outage_cause/
- Source link:
Google’s shorter explanation for the mess is: “Credential issuance and account metadata lookups for all Google user accounts failed. As a result, we could not verify that user requests were authenticated and served 5xx errors on virtually all authenticated traffic.”
The longer version, detailed in Google’s [2]incident report , kicks off by revealing “The Google User ID Service maintains a unique identifier for every account and handles authentication credentials for OAuth tokens and cookies. It stores account data in a distributed database, which uses Paxos protocols to coordinate updates. For security reasons, this service will reject requests when it detects outdated data.”
The G-cloud uses “an evolving suite of automation tools to manage the quota of various resources allocated for services” and the User ID service was moved to a new quota system in October.
The changeover wasn’t perfect. Google admitted: “previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0 (zero). An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service and triggering this incident.”
Google told BGP to forget its Euro-cloud – after first writing bad access control lists [3]READ MORE
While Google checks for quota messes, the tests in place at the time of the incident “did not cover the scenario of zero reported load for a single service”.
“As a result, the quota for the account database was reduced, which prevented the Paxos leader from writing. Shortly after, the majority of read operations became outdated which resulted in errors on authentication lookups.”
So down it went at 03:46 PT on the 14th of December. Google engineers were paged two minutes later, identified the root cause and a fix by 04:08, then disabled the quota system in one data centre at 4:22. Five minutes later they liked what they saw from that change, so made the same change across the G-cloud.
Error rates returned to cloud-as-usual level at 04:33, but Google Calendar kept sending errors “up to 05:21 due to a traffic spike following initial recovery.” And some Gmail users “experienced errors for up to an hour after recovery due to caching of errors from identity services.”
While there’s plenty of egg on Google’s face over this one, you’d probably rather be in the ad giant’s position of being able to sort this out in 45 minutes than Facebook’s position of just having flubbed a three-month deadline.
That deadline was set by the European Union with this [4]September 10th adjustment to the ePrivacy Directive which expanded it to cover messaging services. Such services had until today, December 21st, to change some aspects of their operations.
“In order to comply with the law, we needed to adjust the way our services work, such as further segregating messaging data from other parts of our infrastructure,” [5]Facebook said today, in a post titled “Changes to Facebook Messaging Services in Europe”.
Those changes have reduced functionality of messenger and Instagram.
“We prioritized core features, like text messaging and video calling, and have made sure the majority of our other features are available,” the post says. “However, some advanced features like polls that require the use of message content to work may be disrupted as we make changes to align with the new privacy rules. We’re working to bring back features that we can as quickly as possible.”
Which sounds an awful lot like Facebook’s code and infrastructure are complicated enough that the time since September 10th wasn’t enough to do the whole job. The Register hopes it’s sorted before Friday, because no developer should be forced to miss Christmas. Not even Facebook developers. ®
Get our [6]Tech Resources
[1] https://www.theregister.com/2020/12/16/google_mail_outage_responds_with/
[2] https://status.cloud.google.com/incident/zall/20013#20013004
[3] https://www.theregister.com/2020/12/16/google_europe_outage/
[4] https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELLAR:f9ee4b32-f353-11ea-991b-01aa75ed71a1
[5] https://about.fb.com/news/2020/12/changes-to-facebook-messaging-services-in-europe/
[6] https://whitepapers.theregister.com/
Deliberate delay to put users against EU
It's another propaganda tactic employed by Facebook. "If you miss features, it's because those Grinches from EU. Santa Zuck was ready to bring you candies, but EU blocked that"
In order to comply with the law, we needed to adjust the way our services work
In other words, what we are doing is so blatantly shitty, that countries are now having to pass laws to stop us.
That's what a staging setup is for
If one wants to do critical infrastructure right, all new code and configurations have to make their way from a testing, to a staging setup before getting even close to production. Deployment is then a simple matter of copying the tested-and-verified configuration on staging to production. Staging and production environments have to be identical for this reason to prevent any surprises.
It sounds more like Google practices the time-honoured tradition of 'production is staging', however. To have an old version of the software lying around suggests that they are not using an automatic deployment script (unless it's an intern called 'Gary') and instead probably have a haphazard semi-automatic (or even manual) procedure for updating production.
Working for large companies this behaviour is not unusual, though. Most places I have worked for never did a 'testing' environment, instead using 'staging' as testing environment, and used production for staging. It does increase code throughput and makes it seem like everything is moving faster instead being stuck in 'staging' for weeks while issues are discovered and fixed. The trade-off with omitting staging is the influx of tickets and angry phone calls the hours and days after deployment to production, of course.
Stuff that slipped past unit tests and local testing would end up in production and explode in spectacular fashion, to the point of devices rebooting (watchdog timer) and functionality being suddenly in broken due to environment detection gone wrong or such simple issues.
Omitting a staging phase in deployment is like omitting the 'are you sure?' dialogue box before a disk-erasing operation. Better keep those backups updated (and tested).
code and infrastructure are complicated enough that the time since September 10th wasn’t enough to do the whole job
Or maybe they were just hoping it would go away, followed by a mad rush as oh crap we really have to obey somebody else's laws ...?