Lost your luggage? That's nothing – we just lost your whole flight!
- Reference: 1696231866
- News link: https://www.theregister.co.uk/2023/10/02/who_me/
- Source link:
This week, meet "Leopoldo" who some decades ago landed a stint as a database admin with a certain unspecified national airline. This was his first job as a DBA, so naturally good old Leo did not expect to be handed any serious responsibility – a nice junior position, thanks, and opportunities to learn on the job.
He was assigned to shadow the Senior DBA for a couple of weeks, which seemed fine. Then he was told that at the end of the introductory period he would be taking over the Senior DBA's duties, as he was leaving the airline. Leo did the career equivalent of making sure he could find the barf bag.
[1]
Day one went OK. He went around watching what the Senior did and making notes. The database, which held millions of records, kept track of every reservation, every flight, every passenger's details – the lot.
[2]
[3]
Naturally there were two instances of the database: the production instance, on which all the reservations, cancellations, changes etc. were actually made for this global airline; and the dev instance, in which any back-end changes were made, along with maintenance and pruning of unnecessary records.
All good so far.
[4]
On day two, the Senior decided that Leopoldo should "reverse shadow" him. That meant he would follow Leo around and tell him what to do, and then watch as he did it.
So the Senior DBA told Leo to open a database, define a set of records, and delete. Leo proceeded to bash in the requisite commands, hit Enter, and wait.
And wait.
[5]No, no, no! Disco joke hit bum note in the rehab center
[6]Chap blew up critical equipment on his first day – but it wasn't his volt
[7]Watt's the worst thing you can do to a datacenter? Failing to RTFM, electrically
[8]Sure, give the new kid and his MCSE power over the AS/400. What could possibly go wrong?
Feeling turbulence increasing, Leo wondered aloud whether it should be taking this long. The Senior replied that no, it should be pretty quick to make a change like that on the dev database. "Wait," he asked, rather later than he ought, "what instance were you in?"
A review of the last commands typed in confirmed their worst fears: Leopoldo had just dumped a large chunk of the production database.
[9]
Right on cue the phones started ringing. In every terminal in every airport around the world, the airline was suddenly unable to create, check or cancel tickets. The manager stormed in, demanding to know what was wrong and insisting that full operation be restored immediately.
And that was when the full horror emerged: the backup server, one of the departing Senior DBA's responsibilities, hadn't been operational for weeks.
Thankfully there were some more recent exports of the database around, and with 12 hours of diligent work it was possible to return the airline to something approaching full operation. But a major embarrassment.
Don't cry for Leopoldo, though. When he returned to work the next day – fully expecting his stint with the airline to be over before it had begun – the manager told him that he understood it wasn't his fault. He blamed the Senior DBA for poor supervision – not to mention gross dereliction of the backups – and told Leo to get on with the job.
Ever found yourself in Leo's position, doing the wrong thing because someone higher up the chain was a numbskull? Tell us all about it in [10]an email to Who, Me? and we'll anonymously make you famous. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZRqUw-MgRGkWJeoNiBAwYwAAAJE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRqUw-MgRGkWJeoNiBAwYwAAAJE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZRqUw-MgRGkWJeoNiBAwYwAAAJE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRqUw-MgRGkWJeoNiBAwYwAAAJE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/09/25/who_me/
[6] https://www.theregister.com/2023/09/18/who_me/
[7] https://www.theregister.com/2023/09/11/who_me/
[8] https://www.theregister.com/2023/09/04/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZRqUw-MgRGkWJeoNiBAwYwAAAJE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
If the Italian name is a hint, plus "some decades ago" and "national airline", then there's just one airline that fits, and it's dead already.
> then there's just one airline that fits, and it's dead already.
You means it's plane to see...
Aircraft Landing In Tokyo, All Luggage In Amsterdam?
Your luggage has been retained for your security. Please donate 10 6 Lit. to the Fund for Hierarchical Organised Families so we may safely release your luggage?
The 'retirement' of Pope John Paul II actually made one of my favourite jokes obsolete.
Why does the Pope kiss the ground after each landing?
That's something only Alitalia customers understand.
Don't cry for Leopoldo, though. When he returned to work the next day – fully expecting his stint with the airline to be over before it had begun – the manager told him that he understood it wasn't his fault. He blamed the Senior DBA for poor supervision – not to mention gross dereliction of the backups – and told Leo to get on with the job.
I bet he was in quite a flap the next day...
It was certainly a turbulent start
Thankfully the manager cooled his jets before the next morning
Spoiler alert...
This one command you must not enter
40 years ago, I heard about a senior operator training some junior staff, and he typed the command "PURGE SYSTEM ALL" on the product console, saying this is one command you must never , ever use. Then, like he does a thousand times a day, he pressed the enter/return key,
One outcome of the incident review was that they tightened up the authorisation checks for all commands - and many of us lost what little permissions we had.
A colleague did a customer audit/review, and recommended that the production console background was a different colour to others. A year or so later, someone issued issued a command on the wrong system. My colleague went along to do a root cause analysis, and found none of his recommendation had been implemented. When the feed back got to the board level, they found that there was an action "Get audit/review of system" Tick, this had been done. There was no action "Implement feedback",so nothing was done. Head's rolled, and several people were redeployed the next day.
Re: This one command you must not enter
SQL Server Manglement Studio allows for different coloured highlighting by server, but defaults to MS-preferred bland. It would be better for it to automatically allocate a different colour scheme for each server it connects to, with the usual options to change it manually and perhaps one to disable automatic colour selection to keep everybody happy.
There’s nothing like a bright red colour scheme to remind you that this is the !!!!!!LIVE!!!!!! server, and the yellow one isn’t.
Did the outgoing Senior DBA ...
get a golden parachute?
Deletes eh?
I had to intentionally delete a large number of records from a production db, and very nearly messed up. How? Stay with me…
Inherited a system where a new database was created for each new client, each one linked back to a master db. One client reported problems; an investigation showed the daily data file that they supply to us to load into the client db was the cause, and had been wrong for a week or so. Fixing that was pretty straightforward with a few minutes faffing in UltraEdit, correcting it in the db was a bit more involved. Rolling it back would put it out of step with the master; rolling that back would put all other clients’ dbs out of step. A carefully crafted delete across this one client db and the relevant data in its master db would fix it though, so I set about it.
The delete statement in the test db was fine. Dropping that into the live system and swapping delete for select to eyeball it and confirm that it was good showed all was well, as did wrapping it inside a transaction to roll it back on completion. It was safe, so instead of diving in I went for a coffee first, knowing that it just needed one last sanity check before running it. Which I did, but as I reached across the kb to hit F5 in SQL Server Studio my wrist caught the mouse and did a perfect click drag of the cursor across the delete statement, omitting the Where clause. As any fule know SSMS runs whatever’s highlighted, and it did, happily setting off to delete everything instead of the required few hundred thousand records from the client db. I clocked it almost immediately and hit stop, then silently prayed that SQL Server would live up to its ATOM promise of all or nothing. After an age of watching the rolling back message, it succeeded. Blimey.
I’ve tried to recreate that mishap but could never get a syntactically correct statement - Where what?, invalid table name, incomplete where clause etc. Good job there was enough data there to require a long enough execution time.
You have to be careful in this kind of codebase
You have to be careful what you type working with airline data - all the areas of the system involved in calculating luggage allowances are case sensitive.
I wonder whether that airline would survive a reservation of our lil' [1]bobby tables of the '); drop table passengers;-- family.
[1] https://xkcd.com/327/