Getting that syncing feeling after an Exchange restore
- Reference: 1656924130
- News link: https://www.theregister.co.uk/2022/07/04/who_me/
- Source link:
Our tale comes from a reader we'll call "John" (because that is not his name) and takes place in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5.
John had earned his stripes in the technical support department of a mail-order parts company (remember those halcyon days prior to ecommerce?) and was promoted to into the giddy level of "the IT team."
[2]
"Which comprised of me and one other guy," he noted.
[3]
[4]
Still, it did mean John got to do all manner of exciting work, such as the data processing for the catalog mailings in SQL Server. When this colleague (who was minding the domain and dealing with the foibles of Exchange) left, John was handed the keys to the server room. He even got to go on some courses for a MCSE certification, thanks to a boss who reckoned himself a bit of a technical whizz.
Time passed and nothing much went wrong. The lights stayed on. Computers hummed. John was clearly doing a splendid job.
[5]
Right up until, as he put it, "the first domino fell."
"The Exchange server," said John, "which doubled as a Fax gateway (because email wasn't really a thing yet and almost everyone got their account statement via fax – that weekly fax job was literally critical to the company's cash flow), started blue screening."
A look at the code on the Blue Screen of Death (BSOD) hinted that the motherboard might be at fault.
[6]
"The boss asks me if I'm up for the task of the Exchange rebuild," recalled John. "I tactfully suggest we get in outside help – 'I know a guy who's great with Exchange!'"
Deal! The plan was to do the work on Friday night. All simple enough – back up the Exchange Server to tape. Back it up again, just in case. Then get cracking on the hardware.
All went well. The new motherboard went in. "We pulled out the 8-way Fax card marveling at its grotesqueness and stuck it on the new motherboard."
By midnight Windows was back. Exchange was back. Everything was service packed. Just the tape restore of the data…
…which worked fine. A few obscure ESEUTIL commands were required and… Exchange wouldn't start. Odd. The other backup was tried. Again, Exchange wouldn't start and bleated about a corrupt database.
By now it was 2am, and John decided to call it a night. After all, there was the whole weekend left to get things running.
It isn't hard to guess what happened next. It turned out that ESEUTIL had sufficient options that the potential for error was vast. "Nothing worked," said John, "nobody had a mailbox anymore."
This was bad. But not a disaster. Backups ran every day (a full one on Sunday and then daily incrementals) so restore those, right? They were restored… fingers were crossed… Exchange was restarted…
No luck. Repairs attempted using ESEUTIL took hours, but nothing worked.
At this point it was 4am on Sunday morning and tools had to be downed for a few hours in favor of sleep.
"Tomorrow arrives," said John, "and this time we try to bring back the machine with the old motherboard because we are running out of ideas. Screwdrivers out. Four hours later same result. Back to the new motherboard because we found some more ESEUTIL settings. Alas. Turns out every single one of the 36 backup tapes were corrupt."
[7]You need to RTFM, but feel free to use your brain too
[8]Know the difference between a bin and /bin unless you want a new doorstop
[9]Whatever you do, don't show initiative if you value your job
[10]Brute force and whiskey: The solution to all life's problems
A weekend to sort things had turned into a mere seven hours before staff were due to arrive for work.
It being midnight on Sunday, desperation set in.
"What's better than no email service at all? Empty mailboxes!"
And so there was one last effort. Exchange was reinstalled and configured from scratch. The directory was restored. Sure, everyone would come into work to find their mailbox empty, but at least it would be possible to send and receive messages from the outside world.
At 6am, the duo stood at the front of the building, ready to dispense the bad news to staff as they arrived.
"One by one the message is delivered 'your email works but you don't have any of your old emails'."
The responses were mixed: "The best of which was the warehouse manager: 'No worries mate, it was all useless shit anyway.'"
Finally John's boss (the one who reckoned himself a technical whizz) turned up. Half an hour of questions and explanations followed before he accepted John's apologies: "Well, OK, you did everything you could. It's OK. I've got a backup of all my contacts on this…"
He held up his fancy new Palm PDA, one that had dispensed with the cable for a wireless link.
"He smugly walked into the office into and prepared to get all his mailbox back," recalled John.
"Except the sync settings in the Palm software said 'take the most recent'."
"What's newer than Friday's full mailbox? Today's empty one"
And with that the duo beat a hasty retreat. Very much as case of "I'll get my coat…"
Ever received a slapping from the synch settings? Or seen a Friday night just-a-job explode into a weekend of catastrophe? Of course you have, and you should share your tale with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/Who,%20Me?/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YsK6QzXR3ipccktfq9XycQAAANg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YsK6QzXR3ipccktfq9XycQAAANg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YsK6QzXR3ipccktfq9XycQAAANg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YsK6QzXR3ipccktfq9XycQAAANg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YsK6QzXR3ipccktfq9XycQAAANg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/06/27/who_me/
[8] https://www.theregister.com/2022/06/20/who_me/
[9] https://www.theregister.com/2022/06/13/who_me/
[10] https://www.theregister.com/2022/06/06/who_me/
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Exchange...
...was never easy to back up and restore. I remember when the first "bricks level" backup utilities arrived. Ah yes...years of a product before you had any easy way to restore a single email from backup.
I haven't worked hands on with Exchange for years, and even then the most recent was moving from on-prem to hosted online. And in my personal opinion, it's one service that is better looked after by someone else. :-)
Re: Exchange...
Exchange is still no better.
The improvements have been in backup software (eg Veeam). I'm a particular fan of "instant recovery" where everything instantly runs from the backup storage with an option to migrate to the production drives allowing for an effectively instantaneous restore.
I just can't understand why did John lose the HD, did he reinstall Windows from scratch? If so, why? Replacing a motherboard for another one in most cases just requires some new drivers installed after booting from the windows recovery disk. I did several of those in my days as one-man IT support team.
" Alas. Turns out every single one of the 36 backup tapes were corrupt." "
At which point you just have to think about what you have done to piss off Sir John Cosmos* and wonder whether his plan of ruining your week will just stop at these corrupted backups or will he do his best to knobble your boiler on a cold morning as well.
* the universe - as it feels impolite to call the universe the universe without giving it it's proper title.
Of course the backups work
I did a health check at a customer's site, with a standard list of questions
Q:Do you backup? A:yes..
Q:How do you restore? A:We get John to do it.
Q:Where's John? A: He's on vacation this week... Oh he's turned his phone off.
Q:Can you show me the evidence that the backup/restore has worked?
A:Yes look. Oh Shot. We've been backing up and test-restoring the prototype database - not production!
They had moved from an evaluation system to production, and had not changed the backup jobs.
I added another question to my checklist
Q:Are you backing up the correct things?
That syncing feeling ...
Which way is it syncing, how is it set up where and when to sync? Sync is either great or a vision of impending doom - it never goes "a bit" wrong.
It was the second half of the 90s
I was working in $BIG_COMPANY's European IT organisation that managed several hundreds of servers. One of those servers was in Germany (I was not) and needed rebuilding to swap the existing discs with newfangled arrays. It was an extremely important server too, of course (bad things don't happen to crash-and-burn boxes) and a shitstorm would ensue should anything untoward happen to it.
That Saturday morning everything looked fine, a customer engineer was onsite in Germany taking care of the hardware swaps. Diagnostics ran successfully, the result of the backup jobs had been checked, two full backups had been made, nothing could go wrong - famous last words.
Rewind to a couple of months earlier: I had suggested to my boss to implement backup verification because 'a backup that cannot be restored isn't a backup'. The suggestion was put in the freezer because that would be too time consuming, and perhaps we'd revisit that idea later. We did indeed.
The first restore failed. So did the second. Blood pressure was rising. My faithful onsite helper in Germany fetched tape set after tape set and in the wee wee hours of the morning we had a functioning box again, and I seem to remember that the data loss was marginal - if any.
On Monday I had a little chinwag with my boss and told him that either we implement backup verification NOW or this was the last box I'd ever rebuild. Shortly after that we had the process in place. Those were the days.
I remember Exchange Server 5.5 well. The BOFH and I lost so many weekends to that shitty piece of software back when I was a PFY. If you had the "regular" version, rather than the "super expensive" version, there was an artificial limit of 16GB for the mailbox database. As soon as it reached 16GB, it went offline, meaning that you couldn't even delete any emails to reduce it below the 16GB limit and make it work again.
There were only three ways to fix it. First we had to backup the server in case things went wrong which took a couple of hours. Then either run the "compact datastore" program which took many hours and usually didn't work. Or just truncate the file manually and then run the the "repair datastore" program, which also took many hours. Or restore from the last backup before it went down. Whichever option we tried invariably resulted in some data loss.
Why did it have to screw up so badly and generate data loss and necessitate hours of repair work when it hit this artificial limit? Couldn't it just have popped up a nice message saying "your datastore is full, please delete some emails"?
We tried so hard to find a free open source replacement for Exchange, but there was a nasty circular dependency that Outlook only works properly with Exchange and Exchange only works properly with Outlook. And none of the Open Source mail clients had the convenient calendaring and scheduling functionality of Outlook.
This should be called “Me too”
Not “Who me?”
Exchange 5.5/NT4 was why I packed the MS crap in, started on Cisco, got into networking and wound up doing my CCIE.
Yes, I've been bitten by that. Lost email server, recovered it, local backup sync'd to empty server. ARGH!!!!!!
Luckily, I whipped the power straight out, imaged the disk, and recovered about 99% of the data from the image.
"Except the sync settings in the Palm software said 'take the most recent'."
"What's newer than Friday's full mailbox? Today's empty one"
Looks like a poor Outlook for John...