News: 1612511706

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

The Linux box that runs the exec carpark gate is down! A chance for PostgreSQL Man to show his quality

(2021/02/05)


On Call This week's episode of [1]On Call , as ever, comes with a warning: Be careful moving that beige box, for you may not realise what it does.

Register reader "Jim" was the recipient of today's super-urgent callout, which occurred during his final week of paid employment ahead of a well-earned retirement.

[2]

Jim was a 20-year veteran at his company, having worked in the development team that ran the multinational firm's overseas factories. Describing himself as the resident PostgreSQL evangelist, he'd been given "the talk" by his boss and, like so many in the IT world, found his department was to be dissolved and the work sent abroad.

Luckily, he was near enough to retirement to opt for a life not spent toiling under The Man and used his remaining six months of employment to wind things down. He had also spent the last few years telecommuting, thus avoiding being shoved into a maze of cubes with the rest of the old guard.

[3]

"I wasn't the only one telecommuting unofficially," he protested before admitting, "even though I might have been the most flagrant."

And so it was that on his very last Monday, an 8am call woke him at home. On the line was an impossibly young IT employee seeking help with the server responsible for the security gate to the underground carpark at the Silicon Valley HQ.

"This kid tells me, the garage security gate doesn't work, corporate mucky-muck is pissed he can't park his Mercedes in his reserved space, and demanded IT mobilize all resources to fix this RIGHT NOW."

Takes from the taxpayer, gives to the old – by squishing a bug in Thatcherite benefits system [4]READ MORE

We suspect that Jim meant the bigwig was jolly annoyed ("pissed off") rather than on the vodkas, but then again, our grasp of American can sometimes let us down.

Unused to being roused at such an early hour, and before he had a chance to hit the morning coffee, Jim asked for as many details as his colleague could muster: IP addresses, passwords, and so on. His knowledge of PostgreSQL and Linux had made him the man of the hour.

Donning the garb of the telecommuter (hoodie and sweatpants), Jim trudged to his home office to fire up his PC and connect to the corporate VPN.

And there it was – a slightly whiffy version of Linux squatting on the network. He could SSH to it with root. It ran the security gate and was physically located in a security office. Somebody, it transpired, had unplugged it to rearrange the plugs and the elderly thing had declined to reboot properly.

So now the gate wasn't working and the bigwigs' Mercedes, BMW, and other execu-barges were at risk of either bird guano or an accidental keying from a soon-to-be-outsourced employee.

"It was an ancient version of Linux," said Jim, "with a really old version of Postgres and a bunch of weird Japanese software that ran the gate controller. Best I could tell on first look, the gate software was shutting down because it couldn't connect to PostgreSQL, and PostgreSQL was exiting because _____ no logfiles, all had been set to /dev/null.

"Uh oh."

It took two days to fix. It turned out that PostgreSQL logging was disabled so Jim had to sort that out. Then he found that invalid changes had been made to the configuration files. Since the box had never been rebooted those hadn't been a problem… until now. And so it went on.

Eventually, with mere days remaining until a well-earned farewell beer with his fellow veterans was scheduled, the resurrected server was ready for user acceptance testing.

"OK," he admitted, "kid ran out to the parking lot and waves his security card at the gate kiosk, and IT OPENS! Win win!"

"Did I mention, they told me the last guy who ran this box was laid off six months earlier?"

Sadly, Jim never saw the line of bossmobiles filing into the carpark. Instead he pretty much took his final Friday off and headed for a local outdoor craft beer garden and farewell session with his colleagues.

"Which was indeed epic."

[5]

Ever found yourself taking a call with mere days of your notice remaining? Or did you depart knowing you'd left an accidental timebomb ticking behind you? Share the time you were called out for somebody else's dirty deed with an email to [6]On Call . ®

Get our [7]Tech Resources



[1] https://www.theregister.com/Tag/on-call

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YB0lTFs0CZdsN-NYL4Z0LQAAAMs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YB0lTFs0CZdsN-NYL4Z0LQAAAMs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://www.theregister.com/2021/01/29/on_call/

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YB0lTFs0CZdsN-NYL4Z0LQAAAMs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] mailto:oncall@theregister.com

[7] https://whitepapers.theregister.com/

Jenny with the Axe

I'm thinking that the person who "managed" that box before did not deserve being laid off. He deserved being fired.

tfewster

I'm awaiting the "Who,me?" angle from the ex-admin. Told he was being laid off, so set up a time bomb that would impact manglement.

That's assuming it wasn't deliberate

Dabooka

Could it be the final action of an annoyed ex-employee?

ITWSBT

Temporary fix

b0llchit

Fired? This is probably a case of temporary fix that has become more permanent than the regular change of day and night.

And most likely, the Boss himself ordered the temporary fix and denies any responsibility or knowledge, even though he has been warned extensively.

Tom 7

Or possibly trained. No, almost certainly trained.

chivo243

With all the money spent on excumobiles or excubarges, I'm guessing there wasn't much budget leftover for a non mission critical system like the garage door opener.

Had a call...

GlenP

...from a previous employer some months after they'd made me redundant, "Do you know the admin passwords to the CAD machines?"

I was tempted to point out that even if I did know them the Company had decided my role was no longer required so obviously didn't need me to tell them the passwords. It was irrelevant, I'd never had those passwords as the systems were purchased, installed and managed through a third party not by IT.

Re: Had a call...

Anonymous Coward

I was made redundant from my previous job. It was an interesting experience, as the consultancy meetings about my role all went like this:-

HR/Management: "We think your projects are finished, so your role is no longer required."

Me: "They are nowhere near finished. The network speed dictates that the data moves for one client alone in Project C will require two months, even before we begin processing that data for ingestion at the new destination. There is months of work left in my projects, at least."

HR/Management: "We disagree."

Me: "You're disagreeing with your own network infrastructure. Good luck with that."

On my last day, my boss was out of the office (for very understandable personal reasons - not out of malice or ill will). He sent me an email: "Hey, we haven't started the main phase for Project C yet. Could you please provide us a quick high-level outline of how you'd handle each client during this phase?"

Naturally, I forwarded the email to my personal email address immediately. I then responded with a professional and courteous email.

I did consider taking this to tribunal, but I was lucky and got new job offers almost immediately. So I decided to simply move on with life. Asking for time off because you're suing your previous employer is not a good look at a new job! Also, it's good to just move on and not dwell on past unpleasantness.

But they were very lucky that I did get a new job so easily. Otherwise, they would have had a very awkward time explaining why my projects were still running, but my role was no longer there to run them...

(Posted anonymously, as I have no wish to identify the previous employer. The company's management may be crap, but the people I worked with were good and I have no wish to endanger them.)

Re: Had a call...

GlenP

I did consider taking this to tribunal

In the role I mentioned I had grounds for taking them to a tribunal but as they paid me a reasonable amount above statutory, untaxed money in lieu of notice, a few extras such as a "stationery allowance" as I wouldn't be in the office, etc. I didn't bother. I had a new job within 2 weeks anyway so the timing was right.

Re: Had a call...

Sam not the Viking

Had a night-time call from my previous employer, actually their security/alarm company, six months after I had left. A break-in had been detected and the procedure was that someone needed to attend to investigate and be there when the police arrived (if a second trip was set). Explaining that I no longer worked there the security company explained that they knew that but none of the 'official' responders were answering their phone, 'Could I help?'

No. I sent an email to my old boss later in the day.

Re: Had a call...

Ozumo

There's a typo in your last line. "email" should read "invoice".

Pascal Monett

That was very stupid from the security company, and it could have landed them in court.

An ex-employee has no authority to be on company premises after hours - certainly not when an alarm had been tripped.

They could have unwittingly been setting you up - or wittingly.

In any case, refusal was the only appropriate answer.

Unoriginal Handle

Worked for an ISP many years ago. Was involved with auditing user accounts and configs on sensitive switches and routers which controlled a fair chunk of the ISPs and their customers connectivity.

On being told I was made redundant, I handed the SecurID fob in to a senior colleague the same day, and got an email confirmation from them to confirm. No way I was going to open myself up to all sorts of problems if there were problems.

My foreign manager, on the other hand, is a completely different story and couldn't understand why I wouldn't fly to Europe for a few days, in my consultation period, when I had job interviews lined up...

Evil Auditor

Or did you depart knowing you'd left an accidental timebomb ticking behind you?

Maybe a timebomb nut not accidental: I used to maintain a certain software. My departure had been scheduled and was known for about ten months prior. Several earlier attempts to hand-over the software remained unanswered and in my very last week suddenly they started to panic. Second last day I travelled to HQ and tried together with them to lift the DB. It failed, repeatedly. And my willingness to help ceased with approaching the departure of the train that brought me home in time for supper. Don't know what happened afterwards and neither do I care.

If you asked them I can imagine it would be something like: this a**hole left chaos behind. Good riddance!

I got a wierd call one evening

Pascal Monett

It was the very evening of my last day on a contract in a large Luxembourg company. The contract had been slightly difficult because, on my first day, six months before, I quickly assessed that the IT manager's right-hand man had something against me and was not shy of being clear on that point.

Well, I didn't need to work with him, so it didn't bother me much. I learned the hard way that you don't need to work with someone to be bothered by them - especially when they work above your glass cieling.

Suffice it to say that I was not sorry to see the end of that contract. Then I got the call, which went something like this :

Caller : "Hi, it's me. Um, we've had a problem with the database you were working on, would you happen to have a backup of the code ?"

Me : " Hi. What happened ?"

Caller : "Well the server crashed and, when it came back up, that db was corrupted. We need your backup."

Me : "I'm sure I have a local copy on the desktop of my account."

Caller : "Yeah, but Security was quick to erase that before we could assess the problem."

Me : "Well, there's the backup of my account. Surely you can restore that."

Caller : "Well, no, there is an unplanned maintenance on the backup server - we can't use it. We really need your copy."

Me : "I'm terribly sorry, but I would remind you that the NDA I signed specifically forbids me from taking any code from your premises. I have no backup copy. You're going to have to wait for the backup server's unplanned maintenance to end."

Then I hung up. The caller had been a guy I worked with, who had generally been nice to me - or so I thought.

This had obviously been an attempt to frame me for doing something that I definitely should not (and didn't). Had I been stupid enough to take the code, and even more stupid enough to admit to it, I would have undoubtedly found myself in very hot legal water, and possibly the end of my budding career.

This event taught me two things early on : 1) never trust anyone when someone high up doesn't like you, and 2) always stick to the rules.

Useful lessons.

Outsourcing

Anonymous Coward

Ah yes, outsourcing. That oh so brilliant idea that a revolving door of "programmers/sysadmins" in another country can be just as effective as the people with 20 years experience with the system in question in the name of cost saving. If outsourced resource is just as capable as the home country workers but cheaper, wouldn't the same logic apply at C-level?

Execu-barge

PerlyKing

Not quite the same thing, but at a former workplace the company's car park was secured with one of those ramps that rotate up out of the ground. I got to work one day in time to see a large black BMW beached on top of it, as if someone in a hurry had tried to tailgate their way into the car park. Ouch.

Re: Execu-barge

chivo243

We have bollards? Not sure if that translates well, but you know, those big posts that rise up out of the ground behind the security gate? One poor soul got lifted by the bollard, due to their slow egress... the timing of the raising mechanism was quickly adjusted!

The weakest link

ColinPa

A customer told me of an incident where he worked. They were a top z/OS shop, with two hot sites etc, they tested failover every 2 weeks etc. One of the critical applications told people what fresh fruit etc to put in the containers. It was very slick. The little problem they had was when the drivers came to pick up the containers and collect the delivery instructions, and the printer went down. This printer had been in the cubicle for 10 years. The driver would arrive, the person in the booth would print the instructions and off goes the driver.

When the printer went down there were lorries backing up along the road. As they could not buy a replacement printer with coax attach, it took over a day to get it working, as it was out of wifi range etc.

For want of a nail....

"I thought that you said you were 20 years old!"
"As a programmer, yes," she replied,
"And you claimed to be very near two meters tall!"
"You said you were blonde, but you lied!"
Oh, she was a hacker and he was one, too,
They had so much in common, you'd say.
They exchanged jokes and poems, and clever new hacks,
And prompts that were cute or risque'.
He sent her a picture of his brother Sam,
She sent one from some past high school day,
And it might have gone on for the rest of their lives,
If they hadn't met in L.A.
"Your beard is an armpit," she said in disgust.
He answered, "Your armpit's a beard!"
And they chorused: "I think I could stand all the rest
If you were not so totally weird!"
If she had not said what he wanted to hear,
And he had not done just the same,
They'd have been far more honest, and never have met,
And would not have had fun with the game.
-- Judith Schrier, "Face to Face After Six Months of
Electronic Mail"