Microsoft patches Y2K-like bug that borked on-prem Exchange Server
- Reference: 1641254350
- News link: https://www.theregister.co.uk/2022/01/03/exchange_servery2k22_flaw/
- Source link:
“The problem relates to a date check failure with the change of the new year,” states a [1]January 1st post to the Exchange Blog.
Exchange’s malware scanning engine is the source of the problem, as Exchange checks the version of that software and then tries to write the date into a variable. But that variable’s maximum value is 2,147,483,647 and the value Exchange tries to write - 2,201,010,001, to reflect the date of January 1st, 2022, at midnight – exceeds the variable’s maximum threshold.
[2]
The malware engine therefore crashes when it reads the variable.
[3]
[4]
Deprived of its malware-scanner – an important element of a mail server - Exchange queues mail instead of sending it.
[5]Let us give thanks that this November, Microsoft has given us just 55 security fixes, two of which are for actively exploited flaws
[6]Story of the creds-leaking Exchange Autodiscover flaw – the one Microsoft wouldn't fix even after 5 years
[7]Microsoft Exchange Autodiscover protocol found leaking hundreds of thousands of credentials
A wag on [8]Reddit named the mess “The Y2K22 bug”, a reference to the infamous Y2K bug caused by early programmers adopting a date format of DD/MM/YY to use less memory than would be required by a format of DD/MM/YYYY. Unfortunately, that decision meant that many systems would assume that the 1st of January 2000 was the first day of the year 1900, which would have made for some merry messes.
The Y2K bug was remediated after years of expensive effort.
Fixing the Exchange mess needs some work, too. While Microsoft has released a [9]script to solve the situation, the company’s Exchange team has warned “it will take some time to make the necessary changes, download the updated files, and clear the transport queues.”
[10]
Microsoft’s [11]post about the bug and how to address it was updated seven times between publication on January 1st and the time this story was posted. Chatter in Microsoft forums and elsewhere suggests that the fix can be tricky to apply, and sometimes fails.
All of which means Exchange Admins have an incredibly fun start to 2022 - who doesn't love a deluge of polite inquiries from users miffed that they can't access email? Thanks, Microsoft. ®
Get our [12]Tech Resources
[1] https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YdPUfmxGAJ5mbenf1144egAAAQA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdPUfmxGAJ5mbenf1144egAAAQA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdPUfmxGAJ5mbenf1144egAAAQA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/11/09/microsoft_spreads_patch_tuesday_joy/
[6] https://www.theregister.com/2021/09/27/microsoft_exchange_autodiscover/
[7] https://www.theregister.com/2021/09/22/microsoft_exchange_autodiscover_protocol_found/
[8] https://www.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update/
[9] https://aka.ms/ResetScanEngineVersion
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdPUfmxGAJ5mbenf1144egAAAQA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447
[12] https://whitepapers.theregister.com/
Re: Seriously?
This isn't "legacy" - it's a horrible hack, as they're trying to encode a character based representation into a numeric variable and seems to be the result of some horrible mental confusion. This should have never been written in the first place. They need an absolute thrashing.
I would not call them a "coder" any more than someone that "programs HTML"
Re: Seriously?
> [ ... ] they're trying to encode a character based representation into a numeric variable [ ... ]
That's not what's going on here.
It's pure stupidity on the part of the implementor - namely Microsoft.
They decided to use a signed 32-bit integer instead of an unsigned 64-bit integer to represent a date - namely the type time_t .
Maximum possible value of a 32-bit signed integer : 2,147,483,647 . Evidently, 2,201,010,001 won't fit, so it overflows.
Maximum possible value of a 64-bit unsigned integer : 18,446,744,073,709,551,615 .
If they changed signed to unsigned but kept the 32-bit integer size in their patch - which is very likely, given Microsoft's legendary QOI - they're going to run into the time_t problem in 2038.
They only had 22 years to think about this, so we gotta cut them some slack.
Congrats to Microsoft for making everyone re-live the party atmosphere of 12/31/1999. Cue mandatory Prince song here.
Re: Seriously?
The current fix: Represent 2022-01-02 as 2021-12-33. Using the same logic they are good until 2024-12-19 12:47. If they switch to uint32_t they get to 2042-12-31 23:59 without trickery and 2042-94-96 72:95 works out as 2050-01-07 01:35. I could live to see this bug back again at the start of 2043 but 2050 is probably past my sell by date
"the fix can be tricky to apply, and sometimes fails"
So not much different that most other Microsoft or Oracle patches, then.
There's still time...
Since the suffix "K" in computers refers to the value 1024, "2K" would thus be 2048. Any "Y2K" issues are still 26 years in the future, right?
*Runs away before someone clubs me silly with a ClueBy4*
Another date bug?
(Yawn.) Wake me in 2038.
There are two things to observe here (in naming updates with yymmdd####)...
1. changing the type from the equivalent of "int32_t" to "uint32_t" would solve the problem for several thousand years.
2. Does everyone understand that by reserving four digits for an update counter *within a day*, Microsoft effectively said "our software is sooooo bad, we might actually have to post a thousand or more updates *in a single day*"? Sheesh! :)
It's been a problem for 20 years already
Post-dated spam has been coming in for at least the last 20 years. It didn't cause failure of delivery in the version of Exchange I was using -- but it did cause failure of aging and deletion, with post '22 mail never matching the deletion criteria.
Seriously?
This sounds like a result of some legacy code, a good few years old which was expected to be replaced and then was forgotten about. Maybe the coder who marked it in his/her calendar or ToDo list left and no one else knew about it.
As mentioned in the articles, it's very Y2K-like, no one expected the code to still be in use after it's "Use By" date.