That script I wrote three years ago is now doing what? How many times?
- Reference: 1698046269
- News link: https://www.theregister.co.uk/2023/10/23/who_me/
- Source link:
This week, our hero is someone we've chosen to Regomize as "Indiana" who has reached deep into the archive of a top secret storage facility to bring us this tale of a lost RAID.
Indy was doing a bit of sneaky sub-contracting on the side for a small local business that couldn't afford a full-time tech. One day, Indy's client asked him to have a look at their server, which was behaving strangely.
[1]
It didn't take Indy long to work out that the RAID controller had died, and therefore the bespoke data retrieval system they had developed was useless.
[2]
[3]
Tragically, they could not only not afford a full-time tech, they also could not afford a new RAID controller at that particular moment. The business was basically going to fall over unless a quick and dirty – and cheap – fix could be found.
Indy therefore whipped up a clunky but effective solution. He reconfigured each of the disks in the RAID as a standalone, and wrote a script that essentially created a system of five rotating backup copies of the database. It was well short of RAID, but it allowed some measure of redundancy.
[4]
The danger, of course, was that each of the five copies of the database looked to the system like it was the production database. To avoid the obvious pitfalls that would create, Indy set up a shortcut on the desktop for all staff to use – so that only one of copy of the database would get opened and edited.
That went just fine. For a while.
[5]One door opens, another one closes, and this one kills a mainframe
[6]Scripted shortcut caused double-click disaster of sysadmin's own making
[7]Lost your luggage? That's nothing – we just lost your whole flight!
[8]No, no, no! Disco joke hit bum note in the rehab center
Fast forward three years, and our hero gets a call from the tech who now worked full time for said company. Something's gone wrong, and no-one understands why.
Indy got a sinking feeling in his gut like he'd eaten bad dates. From what the tech was describing, there were multiple copies of the database all over the system – many more than just five. No-one was using the desktop icon to start the database, so new copies were spawning all the time and filling the drives to capacity.
And no-one knew exactly how long this had been going on – only that the entire system had become gradually more corrupt.
[9]
Naturally, hero that he is, Indiana immediately owned up to … absolutely nothing. He still has no idea what went wrong with the sync script, because the last thing he wanted to do was actually go and look at the thing for himself. That's how you get your face melted off.
He agreed with the tech on the phone that this was very strange and mysterious. Probably the best thing to do was to figure out what was the most recent version of the data (if possible) then wipe the system and restore.
Oh, and maybe spring for a new RAID controller.
Here at Vulture Central, we doff our Fedoras to those who, like Indy here, somehow escape from their messes in the nick of time (and hey, three years ain't bad for a script that was doing the work of a busted RAID controller). If you've ever barely scraped out of a disaster that was maybe a little bit of your own making, let us know in [10]an email to Who, Me? and we'll salute your exploits – anonymously, of course. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZTZERfT2TlZgEsKI4L8-rgAAABM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTZERfT2TlZgEsKI4L8-rgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTZERfT2TlZgEsKI4L8-rgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTZERfT2TlZgEsKI4L8-rgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/10/16/trainee_learned_an_important_lesson/
[6] https://www.theregister.com/2023/10/09/who_me/
[7] https://www.theregister.com/2023/10/02/who_me/
[8] https://www.theregister.com/2023/09/25/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTZERfT2TlZgEsKI4L8-rgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.come
[11] https://whitepapers.theregister.com/
Re: Roll your own ... everything
It went as well as expected ie flaky, badly written & undocumented.
So, just like MSMQ then?
Re: Roll your own ... everything
It annoys me how people forget that staff actually cost money. And it gets worse once management start penny pinching. Why the heck do you worry about spending an extra £50 on higher spec monitors for your staff when you're paying your staff five or six figure salaries?
Penny saved,...
You get what you pay for, and then some, for not following the instructions.
Software RAID
Many years ago we had to put in some very small "servers" which were basically PCs running Windows Server - because there was no budget for a RAID controller, we just dropped in a 2nd HDD and set up software RAID to mirror the primary drive. Not an ideal solution, but it was quick to set up, didn't need any drivers and was sufficient for the needs of the client at the time.
Re: Software RAID
Did that in some old MicroServers from HP. Far cheaper to buy those than a NAS, put a boot drive in place of the CD and then configured a software RAID
Took and absolute age to rebuild if a disk failed.
At least someone hacked the bios to support hot swap
We have a management system, hosted on AWS. It manages a few hundred machines, and has various scripts it can run on this machines to perform various functions. One script that I wrote zips up several logs on the machine it runs them on, then "attaches" them to the machine record on the server, where they are available as downloads.
I developed this script, but due to structural changes in my team, never used it, and never really even tested it properly. It stayed on the server, largely forgotten about by me until our server started crashing. The hosting company said that the attachments table for the database had reached it's quota (several gigabytes), and they blamed this script.. The script needed to be triggered by a system admin, as it was triggered by a certain event that could not occur normally (basically, using the agent software, we had to trigger the event manually, and it had a custom name). The other admins denied all knowledge of the script, even though they could see it (and how to call it) in their admin consoles.
Somehow, this script, despite generating and uploading gzip files that were around 100k on each run had filled up the attachments table.
When I was notified of this, I disabled the script, and the hosting company purged the attachments table, but it kept happening. Eventually, I got the hosting company to delete the script, and purge the table. This is when it settled down.
I actually believe the other sys admins when they say they didn't use it. They wouldn't have needed to use it 10s of thousands of times in a few months, we only have about 300 computers on the system and don't get tens of thousands of calls.
I think what happened is that I was testing the script, something went wrong, and the system did not register it had run the script, so did so, repeatedly. But because it wasn't reporting this on the admin console I have access to, and made the logs available for download (as it should have), I didn't know this, and thought it had completed sucessfully.
Thankfully, the hosting company has since agreed to give us a test instance of the server, so any future development can be properly tested before it goes to the live server. This is good, actually, because I need to write a few scripts we can run locally to automated some aspects of the management. We have the required API access for this, but I'm hesitant to let scripts I've written API access to the live data without rigorous testing.
On the flip side
Back in the mid/late 1980's my boss was asked to produce something for the Ferrari F1 team. (We did lots of techy things for the teams.) He decided on a TMS320 based solution (fastest DSP at the time), that I (as the only employee of the company - the boss worked for another shell) would design the hardware and software for. The problem was that the official TI assembler was about the same price as my yearly wage.
So, I had to write my own macro-assembler for the TMS320. It was a great learning experience and meant I knew the chip inside out.
The hardware struggled a little with noise on the dual layer PCB as 30MHz clock was quite fast at the time, but this turned out to be from the radio producing company next door! (It worked perfectly on a Sunday morning!)
I later rewrote a Javascript interpreter that was taking too many time and memory resources on a 4MHz 8086 Internet set-top box where memory was used for screen refresh for 50% of the time.
Getting "deep and dirty" is something everyone should do at least once to get a proper understanding of what is involved.
(Like every driver should spend a night 'on the motorway cones' to get an appreciation of that. I helped change the contraflow at the M2 jnct 5 one night. It is bloody scary standing next to cars doing 80+ in the 50mph limits imposed in such situations! - obviously pre-speed camera days.)
Re: On the flip side
I was once chewing the fat with the heat of IT about why some of my co-workers didn't understand why what they were trying to do wouldn't work. The boss pointed out I was the only person who had any qualification in IT (I had a degree - a third rate one from a third rate uni - but it still counts as a degree). Everyone else had fallen into IT from very different carears (including the boss!) so they didn't have the holistic view of how a computer worked.
To fully appreciate your world, you have to understand how the areas around you work. It's so easy to design the perfect system that no-one uses because you don't understand the context in which your system works.
Re: On the flip side
Getting "deep and dirty" is something everyone should do at least once to get a proper understanding of what is involved.
It was something they were very keen on at Newcastle Uni when I did my degree, even knowing that hand writing machine code wouldn't be something we'd likely need in our professional careers - an understanding of the fundamentals was considered a key skill.
I met a student from a another, supposedly more prestigious, University and discovered their entire computing course seemed to consist of learning as many high level languages as possible with very little emphasis on understanding programming techniques.
vmware disks and oversubscription...
I still can't tell this one...
That's a lovely piece of writing
Professor with a hat icon, that's the closest I can get.
Indiana immediately owned up to … absolutely nothing
Obviously, he didn't want to embark on a last crusade.
Re: Indiana immediately owned up to … absolutely nothing
Of course not, because otherwise he will have to do additional films, I mean jobs, for them down the line which will get worse, just like the last two films!
Roll your own ... everything
Worked on a project many years ago where the best tech for our situation was Microsoft Message Queue (MSMQ). Only problem was management didn't want to spring for the license fees. The directive was for us to build it ourselves. It went as well as expected ie flaky, badly written & undocumented. The irony was that MSMQ was included (free) in the next release of Windows.
Also ironic was that everyone their own (badly written & undocumented) string & date classes.