Do you come from a land Down Under? Where diesel's low and techies blunder
- Reference: 1623656949
- News link: https://www.theregister.co.uk/2021/06/14/who_me/
- Source link:
"Bruce", as the somewhat unimaginative Regomiser has dubbed our reader, is today's contributor and tells us of an event that occurred back in the 1990s.
He was working for a telecoms firm notable for its paging network. "Used to be able to look at the paging messages in real time in plain text from emergency services," he cheerfully told us before unnecessarily adding: "You'd be amazed by what people can 'accidentally' put up their bumhole."
[2]
However, this firm also dealt with the then infant mobile telephone market and "this new-fangled internet thingy (which I was involved in)," he said.
[3]
[4]
"We had a power cut in the entire North Sydney area," he said. "Lightning strike, bloke with a digger, particularly randy wombats wanting a root? No idea."
[5]That thing you were utterly sure would never happen? Yeah, well, guess what …
[6]Thanks, boss. The accidental creation of a lights-out data centre – what a fun surprise
[7]Big red buttons and very bad language: A primer for life in the IT world
[8]The future is now, old man: Let the young guns show how to properly cock things up
No worries, though. There was a colossal UPS able to take the load until the generator could kick in. It would keep things ticking over, at least for the emergency services that depended on the network, for around 40 minutes.
The minutes ticked by, and Bruce started hedging his bets by shutting down non-essential services in order to eke out every last bit of juice from the UPS. "Sorry to all those trying to get to alt.sex.gerbils on USENET at the time," he added, also unnecessarily.
Finally the mighty generator kicked in! It coughed, it spluttered... it proceeded to be a wholly useless lump of machinery.
[9]
This was bad. It was also unexpected. Maintenance procedures had been followed religiously. Every month the checklist had been completed, and every month the generator had been fired up. One step was, however, missing: "Fill generator with diesel."
And, of course, the final emptying of the tank had just so happened at the worst possible time.
Fortunately for the lives that might have been endangered by the outage, our story has a happy ending.
[10]
While Bruce dealt with helpdesk calls from customers of the ISP side of things wondering where their connectivity (or "gerbil pr0n" as Bruce put it, perhaps revealing a bit too much about how much detail could be seen in the logs) had gone, a panicked relay of Jerry cans to the nearest petrol station ensured the generator was coaxed into life with just a few minutes remaining before that critical network fell over.
Checklists and Standard Operating Procedures are great until you discover a missing step at the most inconvenient of times. Ever had to improvise when a SOP let you down? Or were you the author of a process that omitted an all-important instruction? Confess all with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YMcowbbK47rfPH6kqszLfwAAAIA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YMcowbbK47rfPH6kqszLfwAAAIA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YMcowbbK47rfPH6kqszLfwAAAIA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/06/09/who_me/
[6] https://www.theregister.com/2021/06/07/who_me/
[7] https://www.theregister.com/2021/05/24/who_me/
[8] https://www.theregister.com/2021/05/17/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YMcowbbK47rfPH6kqszLfwAAAIA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YMcowbbK47rfPH6kqszLfwAAAIA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Re: looking at the DNS log
Checked a pr0n filter log on a company firewall.
Only once. Never again.
Re: looking at the DNS log
From a former company, overheard conversation between CEO and IT Manager:
CEO: You mean it's possible to check exactly what sites our employees are visiting from the firewall logs? Why don't you do this then?
ITMan: Because if I did, I'd have to tell you what I found. Do you really want me to? Really?
CEO:
Re: looking at the DNS log
Alternative answer:
ITMan: Because if I did, I would also see what sites you visited. Do you really want me to? Really?
Expect the same answer from CEO.
No pr0n without diesel
Give 'er a good filling and crank 'er 'ard!
Re: No pr0n without diesel
Easy if she's self-priming, less so if you have to prime 'er by hand first...
Alternative steps
1) On the quiet, install camera to find out who's siphoning diesel out of the storage tank.
2) Review footage.
3) Present footage to CEO, and wait for a sudden resignation. In our case, of the building services manager, who was earning a bloody sight more than I was, but still felt the need to steal a few quid's worth of diesel once a week.
Optional step 4: While officially declining to give a reason for manager's sudden departure, make sure that unofficially the entire company knows exactly why they went, and exactly how they were caught. This last step can dramatically reduce all sorts of thefts, not just of diesel.
Re: Alternative steps
Some years ago (about 2005 or so) I was working in the automotive third party business and we had a contract with one of the large waste disposal companies in the UK.
They were using a lot more diesel than they should be (big trucks but even so...) so a bright individual came up with an idea to fit a RFID tag at the fuel cap of each of their vehicles and put a reader on the fuel nozzle.
The reasoning was that the pump would only operate if it read a valid tag so we implemented the system for them such that a hash of the tag was available from what was basically a password hash file.
Diesel consumption went down by about 25% almost immediately.
No points for guessing who were most put out by the new improved fuel delivery system.
Happened to me as well!
At what was once described as Europe's Biggest Data centre . have no idea if this was true, but as a 23 year old, it was pretty damn big. Think a large aircraft hanger filled with mainframes, tapes, printers and drum storage. The local power builders managed to put a JCB through the heavy duty three phase power supply, which mist have brought tears to the operator.
We ran a motor-generator set, with a large fly wheel to keep the generator going until the diesel kicked in. Which it duly did, coughed spluttered and died. We had forgotten that diesel can wax, which ours, unused for years, duly had. So the injectors clogged and the thing refused to work. The embarrasing thing is that we were an oil company, and we were supposed to know about such things!
Re: Happened to me as well!
Generators that run all the time are easy, standby ones need care. I love how it is called "polishing" the diesel.
Me too!
When I worked for a large facilities management company, they decided to do a disaster recover test at our site. The power company shut down the electricity supply, the UPS's told the genny to kick in, which it did, and management were all beaming smiles, showing the assembled clients how safe our site was. Beaming smiles right up until the genny ran out of diesel after about 10 minutes. Seems the blokes who drove the nightly backup tapes to the off site storage had been filling up the van from the genny tank.
Re: Happened to me as well!
"The embarrasing thing is that we were an oil company, and we were supposed to know about such things!"
Brilliant - I've only been at one company that did DR processes well, and it doesn't come cheap (but it does come with great reliability, which is correctly more important in certain sectors).
The closest I've come to the failed diesel generator is working next door to a company that, supposedly, had it happen.
Given the number of people here who have claimed to have had it actually happen to them... I am more inclined to believe the reports.
My first gig
20+ years ago, I was managing a computer lab for high school students... this was before we had a content filter! Not enough mind bleach... not enough!
Re: My first gig
This kind of thing is why I no longer fix people's personal computers on the side... *shudder*
Gee…
[P]articularly randy wombats wanting a root? I didn't even know wombats ate roots!
(They actually do, even if that's not quite what Bruce—no, not that one, the other Bruce—meant by this.)
Re: Gee…
Generally they eat, root, shoot and leave. Ooooo Matron!
Checklists
The Air Force have a saying that "Checklists are written in blood"
Someone else has invested heavily in adding each item.
At my last company we moved into a new building which had a huge flywheel generator thingy on the roof. It was frequently tested, so everyone was under the impression it would do the job if it came to it.
Obviously one day a JCB (or similar) went through some nearby power lines and it was time for emergency power... or it would have been had someone remembered to flip the switch from "testing mode" to "do something when the power goes out mode". Oops.
This didn't effect me as I wasn't in the nice shiny building, but the old building a lot of the company didn't know existed which housed the computers/phone switches etc where we toiled in the basement. Ironically it couldn't be made a machine room down there as "it might flood".
Make sure you connect the sense wire to the right side
One (large) place where I worked had an automatic diesel generator set (ISO-container sized IIRC) set up to come on automatically on detecting loss of incoming mains, and off again once mains was restored.
I can't remember whether this was on a deliberate test, or a real mains failure, but the first time it detected loss of power and switched itself on... it immediately detected that the power was back, and switched itself off again.
looking at the DNS log
back when dial up was a thing. Some URLs where telling, others not so much. Content was invariably "OMFG!". Never did it again.