DIMM techies weren’t allowed to leave the building until proven to not be pilferers
- Reference: 1740126608
- News link: https://www.theregister.co.uk/2025/02/21/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Harry" who once worked for a supergiant global tech services company as it helped a major UK government agency to drag its tech into the 21st century.
"I was one of those responsible for digital security in two large datacenters, which in turn supported multiple sites," Harry explained.
[1]
As the project progressed, Harry learned that servers in the datacenters regularly ran out of memory, slowed down, then crashed.
[2]
[3]
"Server memory was expensive, so critical servers were prioritized to receive extra memory," Harry told On Call.
That upgrade program did not produce the expected performance improvements.
[4]
"The odd thing that repeatedly occurred was that we would upgrade a machine that crashed for lack of memory, then it would start crashing again a few weeks later at which point we would find the memory had not in fact been upgraded."
This was a very odd situation, so Harry dug into the paperwork and discovered the memory modules had been ordered, paid for, and delivered.
This being a government job, Harry's employer handled part of the gig, and another contractor did the memory upgrades.
[5]
Harry checked with that contractor, who insisted the upgrades had been made, and commented that perhaps the memory was being stolen.
[6]Techie cleaned up criminally bad tech support that was probably also an actual crime
[7]I was told to make backups, not test them. Why does that make you look so worried?
[8]User said he did nothing that explained his dead PC – does a new motherboard count?
[9]Techie fluked a fix and found himself the abusive boss's best friend
That was an interesting prospect, so Harry and his colleagues wrote a script that ran when a techie logged a complete memory upgrade job in the change management system.
The script looked up the quantity of memory the server should contain after the upgrade, then interrogated the machine to check of the quantity of installed RAM matched records.
"If that did not happen, the change could not be closed, the engineer wouldn't be allowed to leave the site, and people would be alerted."
When those alerts arrived, Harry's approach was to ask the techie if they could just check the memory was seated correctly – a cunning way of telling them they'd been rumbled as DIMMs sometimes need a little extra encouragement to connect all their finicky little pins to memory sockets.
"From then on, we never had issues with memory upgrades," Harry told On Call.
How have you detected, and deterred, light-fingered colleagues? Get your own light fingers moving across a keyboard and [10]click here to send On Call an email so we can tell your story on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z7hc2ArroCZoV3csRxcNAgAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z7hc2ArroCZoV3csRxcNAgAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z7hc2ArroCZoV3csRxcNAgAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z7hc2ArroCZoV3csRxcNAgAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z7hc2ArroCZoV3csRxcNAgAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/02/14/on_call/
[7] https://www.theregister.com/2025/02/07/on_call/
[8] https://www.theregister.com/2025/01/24/on_call/
[9] https://www.theregister.com/2024/12/20/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
I used to work somewhere where they always seemed short of laptops; however, there were none missing in the system. One of the guys on the team seemed to have some very similar laptops available on his ebay account, they suspect (but never proved) that he was pinching them before the computers' details were entered into the CMDB.
Seen that
The guy selling was even using the companies post room to send the kit out !
Couple of the managers got kit purchased via ebay and sent to addresses and then off to HR
I mean, knicking and selling via you own (identifable) account is silly, but using the company post room takes the biscuit
A former colleague did something similar. Ordering new laptops for the company, selling them to his friends. An audit discovered the discrepancy, and a prison sentence hopefully changed his ways and ended his career.
Theft Ring
At a very large place I worked, a group of three PC techies were selling some of our new stuff on eBay. I don't know how they diddled the property database, but I do know how they were caught.
We had an on-campus system which allowed us to send alphanumeric pages to peoples' company-provided pagers -- just sit down at a PC or laptop, go through the company menu to open up a Telnet session to Telecom's server, key in the person's/peoples' login ID(s), and send your text.
This theft ring was sending each other alpha-pages about the status of their auctions &c., which the system retained in logs for 30 days. During routine maintenance, someone spotted these log entries, and kicked the issue upstairs.
As the miscreants had retained these alpha-pages on their pagers, there was a record of their activity beyond the 30-day purge horizon of the server.
The ringleader had a methamphetemine addiction; I don't know about the other two, nor whether we ever got our stuff back.
One of our stores monkeys devised a foolproof scam to order up toner and other associated printer consumables like drums etc then resell them via his personal eBay account and dispatch them from our stores using the company courier account.
He'd have gotten away with it except that he was an idiot who didn't realise that he was an idiot and more than doubling the monthly bill for printer consumables might possibly get noticed.
Using his real name as his eBay handle was also less than smart.
A similar thing happened with a family friend who was working on a building site in some capacity. The site was infested with rabbits and it was discovered that some of the building site consumables; something like 5/8" steel nuts or something, made absolutely ideal catapult ammo. This together with lax on-site security and the fact that the bunnies were well fed and good eating meant that pretty soon everyone on the site was taking pot-shots at the rabbits at all hours.
This all came to a grinding halt when someone from Head Office turned up to find out why that one site was getting through double the usual volume of these steel nuts.
It didn't save the rabbits, though. The builders were hooked on poaching by then and simply switched back to pebbles.
Did the technicians look a bit Sheepish when asked about the RAM?
It looks like theft at Harry's work was RAMpant
Ewe'd never believe it.
Baah, humbug.
Some people RDIMM
You'd hope that those contractors wouldn't be used again after that, but they probably won the contract again by just submitting another implausibly low bid.
Our techies took a couple of PCs out of the store room to build. One of the PCs failed to power on. It wasn't the first time components had come loose in transit, so the techies opened the box to check connections and components and noticed something missing: The CPU. When I say missing, I mean missing: It wasn't loose in the case (or box) it was MIA.
We checked a couple more boxes and found a few more components missing. Police were duly called and questions were asked. Nothing came to light.
As an aside, someone mentioned that they had seen one of the facilities people in town over the weekend.
More questioning and the Police turned up at the facilities person's house and basically said to them: "We know you stole those items. Either you give them to us now, or we come back with a search warrant."
They knew the game was up and handed over the missing items.
Details simplified as the culprit was convinced of theft.
"Details simplified as the culprit was convinced of theft."
"the culprit was convinced of theft."
The plod resorted more robust interrogation techniques? (Wrecking bar and the "A to Z" come to mind.)
During the great DRAM shortage of '88 the prices for these components meant the temptation was pretty great. Whole university computer labs were raided and the computers relieved of their DRAM chips. PCs were hastily wrapped in steel chastity belts and wired in alarm loops.
I would have thought a less clueless techie might have exfiltrated the RAM by placing them in a floppy disc mailer and dropping it in the company's outward mail bag.
I think there is something missing here from the story. "someone mentioned that they had seen one of the facilities people in town over the weekend." There must be more to it than that. Were they selling PC's or just doing some shopping?
DIMMs
It's true DIMMs sometimes needed a bit of persuasion to seat them properly. Unfortunately I once allowed my boss to do a RAM upgrade - did you know it's possible to insert a DIMM the worng way round and force it into the socket? Fortunately the magic smoke only came from the memory so it cost for the replacement, not for a new motherboard.
It will fit in
I once worked with someone who was having a problem with DIMM's and installation. DDR2 , DDR3, voltages, slot location differences.....
anyway, I had a look and I could see the additional DIMMs in the machine, but it was not recognising them. I had tried to pop them out, which they did and then went to reseat and there was no way they were going in. Close inspection and the slot and locator peg were not aligning and then I checked the labels and saw they were wrong.
"How did you get them into the slot".... thinking, they do not fit, should not have got them in
"Oh, I had to push really hard"....
Badly configured
My team took over support from another team, and part of this involved an older Network 4 server - being used for storage by some older Macs.
It ran, albeit slowly. We were going to do some upgrades and get some new kit, but this old box I knew we could re-use for some archiving and exchanging of data with non Mac users. I saw the machine had spare memory sockets so convinced my manager to get more RAM as part of this upgrade as it would be beneficial.
Also observed this was a Dual Processor box.
RAM arrives, install it and go onto the console to do a few more checks, RAM is there all good, running a lot better - great. Then I noticed that the server was in single CPU mode.... Odd as the team that looked after it previously and their main engineer was a Netware guru, upto date CNE blah blah. He liked to tell everyone.
Got to the repurposing of the machine, reinstalled Netware, this time telling it to use 2 CPU and boy did that machine then fly.
Configure correctly and up the RAM and who knows what you can get kit to do
Many years ago
I turned up at work early one morning in the early '90s to find the car park cordoned off, and several police cars parked in it.
After finding somewhere to park, and eventually getting to my desk, I was told that the overnight security staff had been restrained, and many PCs had had their RAM stolen from them (not difficult to restrain two oldish men who wouldn't put up much resistance).
Being a 16 floor building, they had started from the bottom up, ripping the desktop PCs apart to get at the RAM (quite literally, many were IBM PS/2s with just plastic cases and locking tabs), and had got to about the 6th floor before deciding to bail out. Fortunately the systems I looked after on the 11th floor were not touched. I did have some servers on the third floor, but these were caged, and behind a double door 'airlock' that the overnight security would have had difficulty getting into.
Re: Many years ago
Lucky them that the machines were IBM PS/2. Had it been early '90s Olivetti computers, they would have bled to death before reaching floor 6.
With their razor-sharp metal sheets inside, there wasn't a single computer that I touched without leaving a blood sample behind. Occasionally, I had to wipe some drops off and, probably, in doing so, collecting even more cuts. With hindsight I wonder why I never wore garden gloves when installing anything inside those bloody machines.
Re: Many years ago
I recall a similar incident about the same time - could be the same one, there was a spate of break ins too - where the intruders would just open the PCs and take the RAM.
Maybe it was in Computer Weekly, tabloid sized print publication that always seemed to be lying around tea break rooms in those days.
This was when RAM prices had shot up due to a rumoured flood at one of the fab plants, or that was the story going round.
We also had a boss who had a maxed out RAM installation in his PC with a 32 bit Windows (95 maybe) and not able to use all of it without some config fettling.
When an important client machine was short of RAM, his PC was raided. He never noticed.
One time we had a courier deliver a load of PCs (Back in the day of CRT monitors so not the svelte things we have nowadays) As he was unloading them we mentioned to him that he should be delivering them to the loading bay not the front desk. He swore and carried on saying he didn't have time to reload and drive around.
Once he had finished unloading we started to count the boxes. Again, he swore saying he was late and we were holding him up.
We persisted.
We had two different people count and they both got the same number: Three boxes short of what the delivery note said we should have.
We asked him to check his van and he said he'd unloaded everything for us. So we said we've amend the delivery note to show the actual number of delivered boxes and contact the supplier to let them know some boxes had been lost in transit.
The driver decided to take a fresh look inside his van, and, you'll never guess, but he magically found the three missing boxes!
Everyone knew what was going on, but no-one said anything. We took the three missing boxes and signed the delivery note.