Microsoft sheds some light on perplexing Outlook blank email incident: Word was to blame
- Reference: 1621269429
- News link: https://www.theregister.co.uk/2021/05/17/outlook_incident_report/
- Source link:
It was on the evening of May 11th, UK time, and in the middle of the working day in North America (18:24 UTC, 19:24 BST, 11:24 Pacific), that many Office 365 customers using Outlook on Windows observed a frustrating problem: blank emails (or maybe showing just one line), even though a few lines of the email could be previewed in the list of messages. Attempts to create or reply to an email allowed a line of text to be typed, but when the user pressed enter, the text disappeared.
Admins steeped in the strange ways of Office and Windows tried the usual range of fixes, rebooting PCs, reinstalling Office, poking through the event viewer, digging deep into Outlook options to disable hardware graphics acceleration, but generally without success. The only things that typically worked was rolling back the last Office update, or running Outlook in safe mode. It was a huge waste of time.
[2]
Word Web Layout, using for viewing or authoring HTML content, is a critical component in Outlook
"Nice job, MS!! I just spent 3 hours on a machine trying to figure this out!" [3]said one admin. "Thanks I had nothing else to do today... roll back all my clients' outlook.. and who do i bill for my time?" [4]said another .
Many questioned how the company could roll out an update to the world (users across the globe were impacted) which had such an obvious and catastrophic impact. "Do you not run even basic tests before releasing these shoddy updates?" [5]said a user .
[6]
[7]
[8]
The only mitigation is that the fix came relatively quickly and was rolled out as if by magic. "Ok so you fixed it... I am not sure how... when I have disabled both windows update and office update?" [9]said a perplexed user.
Microsoft recorded the problem as starting at 18:24 UTC on May 11th and fixed by 2:00 UTC (03:00 BST, 19:00 Pacific) on May 12th.
The technical fault was assigned the number EX255650 and categorised as an issue with Exchange Online – odd, since it also impacted Outlook when used with Exchange on-premises, and Microsoft's practice of putting the issue details into the Office 365 admin center [10]annoyed those who could not access it.
What was the problem and how was it fixed?
Microsoft has revealed some details. Initially the company said it was "a recent change to systems that facilitate text display management for content within the Outlook client."
The recently released post incident report states: "While mitigating another problem in Microsoft Word, a configuration rollout was deployed for specific 16.0.13929.xxxxx build versions. However, this had an unexpected side-effect on the Outlook client and Web Layout View used by Microsoft Word."
[11]
Word's Web Layout view is little used, and users are only likely to encounter it if they open an HTML document in Word, or foolishly attempt to author web content there and save as HTML.
Why did this impact Outlook?
The reasons go back to the days of Outlook 2003 and earlier, where HTML email content was rendered by embedded Internet Explorer.
If that sounds like a recipe for security issues, that's because it was. It has led to problems like [12]CVE-2002-0481 , "an interaction between Windows Media Player (WMP) and Outlook 2002 allows remote attackers to bypass Outlook security settings and execute JavaScript via an IFRAME in an HTML email message."
In Outlook 2007, embedded IE was replaced by embedded Word, using Word's HTML support, upsetting authors of marketing emails who had to tailor their content to suit those limited capabilities, but making Outlook significantly more secure.
The same is true for authoring emails: Word is wheeled in, though with some strange Outlook-specific behaviour. For those who wonder why Outlook makes it so hard to format emails sensibly when doing things like quoting parts of an email received in the reply and typing in-line comments, or putting the background of the incoming email behind the reply: it is the interaction with Word that is to blame.
[13]
This interaction turned from annoyance to disaster in the faulty configuration that gave rise to EX255650.
How the heck was that done?
Microsoft's impact report is far from comprehensive, but there are clues about what happened. A configuration, in this context, is something "which Microsoft 365 apps use to determine which features, components, or specific code paths are enabled or disabled. We can target these rollouts to specific builds, apps, platforms, or audiences. Configurations can be synchronized up to every four hours by Microsoft 365 apps," the company said.
That four-hourly call home presumably explains why Microsoft was able both to introduce and to fix the issue with so little sign, from a user perspective, that anything was changing.
Windows just did some background stuff and bam, Outlook breaks; then bam, Outlook is fixed. Microsoft has not revealed the exact detail of what setting was responsible, nor has it yet said what was the problem in Word that it fixed.
How was it not noticed during testing?
Microsoft said that it is "performing an extensive review of our coding to understand how the issue occurred and why it was missed during the testing and early deployment cycle," and there may be a further update to follow on this matter, when the preliminary report is replaced with a final one.
We can speculate though that some small corner of Microsoft did not take account of the key role Word Web Layout plays in Outlook.
In the report, the company noted that for Word, "the default view is not Web View, so only a small percentage of users use it. Other views, such as Page View and Reading View, were not impacted."
In Outlook though, "web Layout is the default and only view available."
That said, even breaking Web Layout in Word is something that should have been spotted – though combining rapid response to security problems with thorough testing of updates for unexpected side-effects is a challenge.
Some of the less appealing aspects of Microsoft's platform were exposed by this bug. One is that Outlook, despite its high value in integrating email, calendar, contacts and tasks, remains full of legacy code that can cause problems.
Another is that today's Windows and Office 365 users are one bad update away from hours of lost productivity.
One thing to note though: users who followed a full cloud model, using the web browser rather than desktop Outlook, were not impacted at all. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2021/05/12/outlook_outage_patch/
[2] https://regmedia.co.uk/2021/05/17/weblayout.png
[3] https://twitter.com/ctcitnetter/status/1392249665442172931
[4] https://twitter.com/MSFT365Status/status/1392208966231355392
[5] https://twitter.com/ToxophiliteLeon/status/1392225511187812361
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YKLne631FVxo4C7U2tDK1wAAAJg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKLne631FVxo4C7U2tDK1wAAAJg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKLne631FVxo4C7U2tDK1wAAAJg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://twitter.com/bunneyd/status/1392427473133002753
[10] https://twitter.com/ColinBocking/status/1392408491583631361
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKLne631FVxo4C7U2tDK1wAAAJg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2002-0481
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKLne631FVxo4C7U2tDK1wAAAJg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
Re: "users who followed a full cloud model ... were not impacted at all"
Nor were those who refuse to use Microsoft's untested (apart from by the victims users) shit!
"when I have disabled both windows update and office update"
So even if you disable updates, MS can force an update on you. This time it fixed something (which probably didn't need fixing because you had updates disabled), next time it could screw up things.
Re: "when I have disabled both windows update and office update"
... could screw up things ? The Grammar Nazi in me says 'would'
Re: "when I have disabled both windows update and office update"
...'would'...
I think it should read " will screw up things". It is a certainty, not an option when using windows and/or office products. Causes for a screw up are plenty.
Not just Windows users
Recently I have had multiple alerts on Macs asking for, among other things, that the users reauthorize their accounts. One user had two O365 accounts, and both of them asked him to reauth twice each in the course of three hours. As one of his O365 accounts wasn’t ours, it’s not likely that it’s our problem. (The user was on his personal machine, connecting using MS Outlook. He also had several IMAP accounts on the same system, all of which worked properly.) Users on Windows had the blank email problem; one such user had the reauth problem as well. He was also on a personal machine. There may be a trend here. All users recently updated MS Office.
HTML for email was a bad idea. Creating dependencies between an email client and a word processor was a worse one. The colored pencil crowd loves HTML so we're stuck with it, though.
s/pencil/wax crayon/
Tell me about it. Most of the worst parts of the internet can be traced back to the crayon fairies.
If it's not having special fruit-based hardware, activating creepy ad-tech trackers "to provide valuable insights into our customers", pushing aggressive cookies, HTML and "Twitter engagement".
So I take it M$ are going to be paying effected customers for their loss of service?
That has got to be a lot of workers, globally, reliant upon email unable to work and your support wasting it's time trying to fix someonething that is completely outside of their control.
With M$ beta testing via their subscribers and able cripple your company regardless of your update options you should be wondering what you would actually loose with a change to local office suit and text only communications, especially when so many are free to use without slurp and actually under your control
I wonder if effected companies can claim off their insurance or have write it off to tax.
How was it not noticed during testing?
The post is required, and must contain letters.
Oh, ok then, see icon--->>
The Old New Thing
Half the article sounds like one of Raymond Chen's apologias for all the legacy shit in Windows.
How was it not noticed during testing?
Thats the publics job, init?
Nprmal.dot
I had an issue a few years back when outlook menu items were missing it was caused by a corrupt NORMAL.DOT, for this issue in the mentioned article, i was up till 0100 trying to fix this when i did a news search on Outlook and discovered the Muppets at MS had caused the issue. Why do these morons not test their code ?
Suffice to say by morning everything was working.
"foolishly attempt to author web content there and save as HTML"
Blimey, is it still that bad? Yes, it's Word - of course it is!
I once had someone upload Word-generated HTML to an intranet server that overwrote earlier (and properly formatted HTML- well, for 2001 anyway) versions of the pages. A stern talking to was required.