News: 1626679809

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

How to keep your enterprise up to date by deploying the very latest malware

(2021/07/19)


Who, Me? Start your week with a trip back to the early days of the World Wide Web with a tale of imaging peril and malware malarky from the files of [1]Who, Me?

Today's story comes from a reader whom we shall call "George", for that is not his name, and an experience at what he delicately called "a very large retail store" during the closing years of the last century.

Back then, deployments in George's organisation were based on machine images in order to keep a consistent Standard Operating Environment (SOE) over the fleet. A machine would be set up with the required configuration, which would then be imaged and reused.

[2]

Simple stuff. Until the events of today's story.

[3]

[4]

It was a Friday afternoon. "I had completed the build on the latest SOE on a spare workstation," George told us. The plan was to take that pristine build and deploy it over the 30 new workstations that were due for delivery next week. Job done, he headed home, doubtless looking forward to cloning his work over the sparkly new hardware.

However, he had reckoned without the antics of the 24x7 operations team, who had taken a shine to this particular workstation. "They'd do their personal stuff here," George said, "banking, online shopping, searching for rental accommodation and on this occasion…" something somewhat salacious.

[5]

Blissfully unaware of what his freshly built computer had seen over the weekend, George arrived on Monday, raring to go.

"I clone this workstation into the new image," he told us, "and proceed to deploy that pristine SOE into 30 brand-new workstations for the Advertising department."

By Tuesday morning he was ready to refresh the dated hardware currently in use with the shiny new gear. Everything went swimmingly – the users were delighted with their new workstations and, having deployed 30 new machines in a day, George was the hero of the hour and the apple of his boss's eye.

[6]Ah, I see you found my PowerShell script called 'SiteReview' – that does not mean what you think it means

[7]One good deed leads to a storm in an Exchange Server

[8]Hmmmmm, how to cool that overheating CPU, if only there was a solution...

[9]Updating in production, like a boss

Up until noon on Wednesday.

"A weekly scheduled scan starts across every workstation looking for viruses and unwanted nasties…"

[10]

No, we don't know why some IT departments insist on crippling computers during working hours either, but it's good to know that some things never change.

George's console lit up like a Christmas tree as the scanner found all manner of horrors on all of the new workstations.

"My boss gets these alerts at the same time as I do," he said, "they are still dinging on my console and I'm sitting there dumbfounded as my boss looms into my space."

"I really like a manager who can take the time to understand an issue, help identify the cause and work towards a resolution before they go looking for the person who needs to have the conversation that starts with 'I'm going to rip you a new one'," George observed.

As do we, George, as do we.

Thirty workstations in a day was nothing compared to the speed at which he and some colleagues managed to switch the new (and festooned with malware) shiny with the old and busted. It took less than an hour for the befuddled users to be back in business with last week's hardware.

"The users never knew the real reason why we did this odd workstation shuffle," said George. "They got the new machines back again a few days later."

As for how it happened, it transpired that operations team had been omitted from the internet content policies of the organisation, something that was swiftly rectified.

Along with a few extra checks and processes around image deployment…

Ever done the IT equivalent of trampling mud (or worse) over a favourite grandparent's new carpet? Or fired up an exciting new bit of kit only to find something awful has happened to it? Share your story with an email to [11]Who, Me? ®

Get our [12]Tech Resources



[1] https://www.theregister.com/Tag/who-me

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YPVNQvGEyvtSAN2hx@IR2QAAAM8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPVNQvGEyvtSAN2hx@IR2QAAAM8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPVNQvGEyvtSAN2hx@IR2QAAAM8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPVNQvGEyvtSAN2hx@IR2QAAAM8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2021/07/12/who_me/

[7] https://www.theregister.com/2021/07/05/who_me/

[8] https://www.theregister.com/2021/06/28/who_me/

[9] https://www.theregister.com/2021/06/21/who_me/

[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPVNQvGEyvtSAN2hx@IR2QAAAM8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[11] mailto:whome@theregister.com

[12] https://whitepapers.theregister.com/



Potemkine!

No, we don't know why some IT departments insist on crippling computers during working hours either

Maybe because some users keep shutting down their computer when they end their day, whatever the numerous times you said to them not to do it to enable the scans during the nights?

At the end, either you disturb a lot of users or you have computers never scanned blinking in red on your monitoring app. Some choose the former, some the latter.

Anonymous Coward

Wouldn't Wake On Lan be a sensible option to use?

that way machines that had been turned off could be powered up in the wee small hours to run their scan and shut down again

Korev

> Maybe because some users keep shutting down their computer when they end their day

Maybe because they want to conserve power which saves the company money and is better for the planet

big_D

It has been company policy at most places I've worked in the last decade to turn PCs and peripherals off when leaving the office.

The cost of wasted electricity for hundreds or thousands of PCs left running over night soon mounts up.

Anonymous Custard

Wouldn't Wake On Lan be a sensible option to use?

Not when the laptops in use are shut down, undocked and locked away safely in a drawer or cupboard for the night.

Apparently the policy here is more concerned about the cleaners than malware...

This is one of those things...

Anonymous Coward

You do once. And only once in a career.

Leaving a freshly minted machine all alone without proper access controls where any Tom, Dick and Harry (it's never the women I note...) can mess with it.

In my work place an unlocked machine results in SFW leather fetishism image searches involving a serious gray beard and outdoor settings.

There is not enough mind bleach...

Anon because I'm supposed to work for a respectable company...

Re: This is one of those things...

big_D

I'm guessing proper access controls were in place. But the operations team were excluded from the Internet content filter.

Rookie mistake

PTW

Why did he not image it the moment it was complete? Even on PoETS day, this can wait for the image to complete

this apha code is not fit for production!

ColinPa

When I worked in product development I was contacted by some one (in the same company) who wanted to ship our product as part of theirs. I was asked could I get them an early driver. I said go through the official channels, but we can provide some alpha level code so you can develop your install procedures. We fixed the code so it had a drop dead date of 2 months away. "No problems mate". He got the install working, so was happy.

Two months later my senior manager got a call from someone angrily complaining that our product was rubbish - it kept starting and shutting down with no error messages. They were going to ship their product next week, so this was a critical problem which we had to fix. The demo to their executives was a disaster ... etc

Instead of getting the golden master of our product, they had cloned the software from my friends machine, without his knowledge and were going to ship it. I believe there was a "senior manager to senior manager" discussion about the incident, and I believe the person who caused all the problems was asked to "review his options" and moved to pastures new.

I learned that if you are going to give someone a b*llocking - be very sure of your facts, and keep it low key.

Re: this apha code is not fit for production!

Pascal Monett

I work in Luxembourg.

Check it out on the world map. It's a very small country. You can cross it in an hour on the highway.

It's even smaller in the workplace. I learned that very early on in my career.

If you want to chew someone out, you had better be damn sure you're right, because your reputation can be trash quicker than you can blink.

Re: this apha code is not fit for production!

Rich 11

It's a very small country.

You're being generous there. In matter of fact, Luxembourg's front drive opens onto Belgium and the gate at the bottom of the garden leads to Germany. If you want to visit France you have to jump over the neighbour's fence.

Not quite the same

Roger Kynaston

A friend of mine was demonstrating the marvels of ghost for rapid deployment of multiple workstations when he had a Bill Gates moment and the PC he was cloning to dind a classic NT blue screen. We all laughed a lot, not least since I had just made the jump to supporting UNIX by then (Solaris and DEC Alpha for the nostalgic).

Been there - on a Nuclear Power Plant

I Am Spartacus

similar story. We had built the first image for a power control plant that was to hook up and manage a number of nuclear power stations. Thanks to the local telecoms muppets we were way behind schedule and hadn't started our clone of the standard image on to the workstations. Until one evening, the system came online, and we started doing a long suite of tests with the power station. And left for the night.

When I got in next morning all hell had broken lose. People were going mental and the site managing director wanted to rip me a new one. It seems that we had sent a "SCRAM" command to the reactor. For those not knowing, this shuts down the reactor hard and fast, and can take months to recover from. I asked for an hour to find out what was going on.

With the hour up, we went back in to the meeting for round two of lets beat up the IT techies. Where I asked if "Jerry" was joining us (not his name). Jerry was there. He was from the company building the reactor control equipment that we connected to. His work was behind schedule, caused by our work being behind schedule. So I asked him how come all the PC's in the control office were all up and running. He then somewhat sheepishly admitted that he had, over night, cloned our working machine that was doing tests and installed the image over all the new machines that we had not yet configured. He then started doing some commands to see if the reactor was behaving.

The problem was that he had not changed the unique addresses of the workstations. This is pore internet, so they were sort of like IP addresses but exactly the same. The protocol allowed the boxes to come up. So Jerry booted all 8 boxes. And sent a command to the reactor.

Now the protocol was: Send a command. Reactor sends the command back for confirmation. The PC then replies, yes, that's what I want, and the reactor says OK, here is the result. Trouble was, one PC said yes, that's fine but 7 said - no, we didn't ask for that. So the reactor does another round of checks. And agsin 1 says yes and 7 say no. The reactor then assumes it's under attack and shuts itself down. Hence the scam command.

Fortunately we didn't cause too much damage as we were only talking to the control test rig, not the actual reactor. But it did get me the missing lock on our computer room and Jerry a flight home. Oh, and we got our names on rather splendid mural at the entrance hall for having saved the say!

The explosion icon, well, its obvious.

Lockdown

elsergiovolador

Reminds me of an organisation that deployed brand new laptops for their developers. The machines were so locked down, they couldn't install almost any software required to do their job. For a couple of weeks those who had a personal laptop, had to work on that and those with desktops worked from home...

One guy offered to install everyone a fresh Ubuntu so they could get on with it, but management said what if those machines get hacked later on and we will be on the hook for allowing that? Yet, they had no problem with people working on their own computers for a time being...

Never have children, only grandchildren.
-- Gore Vidal