I was a part-time DBA. After this failover foul-up, they hired a full-time DBA
- Reference: 1756712349
- News link: https://www.theregister.co.uk/2025/09/01/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Derek" who told us that around 2012 he worked as a part-time database administrator for a company that operated two MySQL instances – one in Boston, USA, and the other on the charming Mediterranean island of Malta – and set up symmetrical replication between them.
Derek thought this was a bad idea because Malta is not regarded as a global connectivity hub, and he felt the goal of symmetrical replication – swift mirroring of any changes made in either location – might be hard to achieve if something damaged a submarine cable to the island.
[1]
He was right for the wrong reasons, and learned that when the company's Boston datacenter tested its automated systems for switching to backup power in case of emergency.
[2]
[3]
That test failed, leaving the Maltese database handling mission-critical, high-volume transaction applications.
[4]Teen interns brute-forced a disk install, with predictable results
[5]Pay attention, class: Today you'll learn the wrong way to turn things off
[6]Tech bro denied dev's hard-earned bonus for bug that overcharged a little old lady
[7]Intern did exactly what he was told and turned off the wrong server
Once the Boston bit barn came back online, Derek knew he would need to use the REPAIR TABLE command to catch up on changes made in Malta and restore synchronization between the two databases.
He did so but forgot the LOCAL option that would ensure he only repaired the Boston database. To be fair, the company's manual only mentioned this on page two.
Remember that symmetrical replication? Derek had just started repairs to both the Boston and Malta databases, and the latter didn't need to be repaired. It needed to run, unimpeded, to keep the business humming.
[8]
After a frantic hour of repairs and backtracking, Derek restored operations.
"The moral of this story is read the second page of the manual," he told Who, Me?
This story has a happy ending of sorts because after Derek's mistake, his employer recognized it probably needed a full-time DBA.
[9]
Yes, Derek should have RTFM. Has failing to do so led you into trouble? Here's a clear instruction you can't miss: if you have a story in this vein [10]click here to send email to Who, Me? We could use a few more stories right now so get clicking! ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aLVuu3e0vgfYniR_C-EHiAAAABQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLVuu3e0vgfYniR_C-EHiAAAABQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLVuu3e0vgfYniR_C-EHiAAAABQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2025/08/18/who_me/
[5] https://www.theregister.com/2025/08/11/who_me/
[6] https://www.theregister.com/2025/08/04/who_me/
[7] https://www.theregister.com/2025/07/28/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLVuu3e0vgfYniR_C-EHiAAAABQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLVuu3e0vgfYniR_C-EHiAAAABQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
swift mirroring of any changes made in either location
Well there's the problem, Taylor isn't known for her DBA skills
Tay Tay is better known for her security skills.
RTFM. Has failing to do so led you into trouble?
About 50/50; reading the (wildly inaccurate) manual and proceeding on the basis of its assumed accuracy has produced some unexpected consequences.
Documentation can be out of date; copied from another platform where unique but unredacted features existed; contain feature were to be implemented but never were; just plain wrong.
Re: RTFM. Has failing to do so led you into trouble?
Im just impressed his company *had* documentation!
Re: RTFM. Has failing to do so led you into trouble?
It doesn't help that some of the manuals out there are pretty dreadful. Oracle is one example. A single page jumps all over the place and you need to keep your wits about you to make sure you're still reading about the same product.
Paging Doctor Strange
The warnings come AFTER the spells!
Blind Spot
My particular blind spot, if following a list of detailed instructions, is to fail to notice or absorb the second of two operations included in the same line.
We have a machine-logger where the data can be downloaded by the end-user, but one line of the instructions calls for two actions and the second is often missed, resulting in a call for help. From personal experience, I know how to break the news gently, helping the customer to understand his failing in good humour ..... I have re-written the instructions but they have not been adopted, probably because the message contains more than one operation in the same sentence.
You should do this
I remember getting into trouble when the documentation said "you should do .... " when it meant "you must do...". The word 'should' implies recommendation but not mandatory.
Fortunately it was only a test system we could easily recreate. My supervisor got the documentation changed.
Re: You should do this
See https://www.rfc-editor.org/rfc/rfc2119
Re: You should do this
Too many people are afraid to use a straight imperative: "do X". This avoidance is generally fine in a social situation, but not in an operational one; the listener/reader needs to know whether he is receiving a request or an order.
Re: You should do this
gov.uk is full of "you may" and "you might". If anyone is supposed to give clarity, it's them.
"Yes, Derek should have RTFM. Has failing to do so led you into trouble?"
The real trouble starts when the manglement read the sales lies blurb and believe it.
No, take a step back - when the vendor reads their specification and believes it. Works in test is not the same as works in production. Works in production on original platform and works in test on ported platform is not the same as works in production on ported platform.
Seen it done at a hardware level
We had VAXen boxes with mirrored disk pairs. There was a disk issue that caused on of the disks to drop out of the mirrored pair. The system carried on as it was supposed to do, no user impact at all. Digital came and replaced the drive with a new one.
The System Manager then readied himeself to bring the new disk into the mirror and have it catch up. Unfortunately he got his source and destination wrong and before I couled yell out NOOOO, he mirrored the brand new, empty disk all over the remaining working copy of the production data.
A real oh sh1t moment.
Yes, Derek should have RTFM. Has failing to do so led you into trouble?
Well were the Maltese Cross?