News: 1656059409

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

The perfect crime – undone by the perfect email backups

(2022/06/24)


On Call A tale of theft, fraud and understanding the meaning of "Delete" to end your working week. Welcome to a legally questionable episode of [1]On Call .

Our story is another from a reader Regomized as "Ellen" and once again concerns Digital Equipment Corporation's finest. In this case, DEC's ALL-IN-1 office automation suite of the 1980s.

ALL-IN-1 was quite the thing back in the day. By modern standards it was pretty rudimentary, but with its email and word processing functionality it must have seemed like a whole new world. It was also highly customizable.

[2]

Ellen was employed at a financial institution that we will not name and tasked with administering a network of ALL-IN-1 computers, with accounts in every branch. "Normally," she said, "it was pretty pain-free stuff allowing for long lunch breaks."

[3]

[4]

Until, typically, the inevitable happened and Ellen's pager went off. And of course it had to go off during the lengthy (for the UK) Easter holiday. On the LCD screen was a request to go into the office. She ignored it. The pager chirped again. And again. It appeared there was no avoiding a bank holiday visit to the workplace.

So in she trudged, only to find the office relatively quiet. The urgency of the message had made her suspect the Vax had dropped over but nope. While there were a few people milling around, a glance at her homegrown status page showed all services up and running.

[5]

"Before I could say 'Why am I here?', I was called to the top floor, which was an elevated place I very, very rarely had cause to bother. This time it was straight into a boardroom. Lots of very seriously worried people in there."

And lots of very big cheeses.

[6]How one techie ended up paying the tab on an Apple Macintosh Plus

[7]The swift in-person response is part of the service (and nothing to do with the thing I broke while trying to help you)

[8]Can't get that printer to work? It's not you. It's that sodding cablin.... oh beautiful job with that cabling, boss

[9]Oh Lord, won't you buy me a Mercedes-Benz? Detroit waits for my order, you'd better make amends

Was it possible for her to look at some specific email accounts? For a specific branch of the bank?

Sure. But what for?

There was some shuffling of feet. "Anything strange."

[10]

This was distinctly unhelpful, and Ellen pointed out that such a vague search would be akin to hunting for a needle in a haystack. When one doesn't know what a needle looks like. Or even a haystack.

There was more huffing and puffing and a piece of paper was pushed across the desk. It was a Non-Disclosure Agreement, replete with some hefty penalties should the vow of Omertà be broken.

The reason for the call-out was then disclosed. It transpired that an employee had been overly helpful and run a job on the mainframe early – on Friday rather than the next working day, which would have been Tuesday (Easter – remember?)

This had thrown up a discrepancy…

"It seemed a rather large sum of money was missing," Ellen told us, "but they didn't know why."

Attempts to contact the manager of the branch were unsuccessful. Officially he was on vacation. However, the security officer had managed to get in touch with the manager's spouse, only to be told he was away on a course for work.

There was no such course.

"Alarm bells are now going off," said Ellen. "Missing money, missing bank manager. We can all see where this is going."

Now knowing what she was looking for, Ellen commenced the hunt. It didn't take long: "There were a host of increasingly lurid emails between the bank manager and the head cashier," she said. "They had planned this, knowing that the bank would not be audited for several days, and had removed a rather large sum from the branch's float.

"The head cashier was also missing and uncontactable."

Ellen dutifully handed over the evidence and the miscreants were apprehended while attempting to board a flight from the local airport.

Our story, however, does not quite end there.

The bosses were impressed with Ellen's detective work, and surprised that the duo had left such an easy trail to follow.

"I had to explain," said Ellen, "that they had tried very hard to cover their tracks, but we had this thing called a 'backup'…"

The backup tapes hadn't made it offsite yet, so she'd been able to get them loaded and pick through the data in search of an explanation. She was naturally very chuffed at her cleverness.

The board, however, was not.

"You mean," came the horrified question, "when we press delete, the emails don't actually delete, they get saved to backup tape?"

Yep – that was pretty much size of it.

Pandemonium now ensued.

"But when I say delete, I want them deleted!" shrieked a senior manager, "You have to delete them not just on the online system but also in all the offline backups!"

Ellen was about to point out both the technical and legal challenges involved when another boss stepped in and gently pointed out that the same senior manager had accidentally deleted a folder of vital emails just the other week, and they'd been quietly restored by IT before anyone had noticed. Backups were good.

"Someone at the meeting seriously suggested that we add an extra button to the keyboard. One would be 'Delete, but I may want this back' and another would be 'Delete, now and forever'."

The idea, while serious, was thankfully never implemented.

Ever turned your IT skills into weapons for crimefighting? Or had to gently explain to a red-faced executive that sometimes Delete doesn't always mean Delete? Tell your tale with an email to [11]On Call . ®

Get our [12]Tech Resources



[1] https://www.theregister.com/Tag/on-call

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YrWLSPB-3PJV58aSTc1LNAAAAIk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YrWLSPB-3PJV58aSTc1LNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YrWLSPB-3PJV58aSTc1LNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YrWLSPB-3PJV58aSTc1LNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2022/06/10/on_call/

[7] https://www.theregister.com/2021/05/07/on_call/

[8] https://www.theregister.com/2021/04/30/on_call/

[9] https://www.theregister.com/2021/05/21/on_call/

[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YrWLSPB-3PJV58aSTc1LNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[11] mailto:oncall@theregister.com

[12] https://whitepapers.theregister.com/



Panic stations

SminkyBazzA

Sounds like a certain senior manager also needs their email backups reviewed for naughty business, literal or otherwise.

One wonders what will happen when ...

jake

... major governments of the world discover that when BigHighTechGossipMonster deletes all that personal data that they have been illegally hoarding for the last couple decades, all that really gets deleted is the current working copies. The carefully curated off-site backups live on forever, in all their redundant glory, just as they were designed to do.

"Delete" = "Hide"

Mast1

I heard a story that the Psion organiser (early PDA) saved data in UVEPROM, ie unerasable while in device. "Deletion" involved flipping the 8th bit in an ASCII code, which was interpreted as not to read the file. This approach relied on the EPROM being much larger than anyone could reasonably fill purely by typing. Apparently the Psion was a favourite of the criminal fraternity who recorded their deals on them. Once apprehended by the police, correctly read, the Psion provided a nice audit trail.

Re: "Delete" = "Hide"

phy445

Isn't this essentially the same on MS-DOS based systems? My memory is that they just added a character to the start of a filename so that they were flagged so as not to show up in directory listings but remained until someone sent the command to clear space. I'm fairly sure that the clearing process was (is?) just removing them from the record of what is on the disk (hence the existence of undelete utilities).

Re: "Delete" = "Hide"

GlenP

Essentially, yes, also the FAT table retained the sector links.

IIRC it changed the first character rather than adding a character (only 8 remember) so to undelete you had to know, or guess, what the first character should be*.

I can't recall the exact details but I was once able to reconstruct a Word temporary file via Norton even though undelete failed. The user, a young student placement, had spent all day typing a document then failed to save it - one of the times IT managed to save the day and prevent a lot of tears.

*I used to work for an Apricot dealer, swapping between PC compatible and Apricot versions of DOS. The latter used A: for the HD so it wasn't unknown to type Format A: on an Apricot then Y followed by "Oh sh*t!" Ctrl-C and Undelete usually sorted it.

Re: "Delete" = "Hide"

Greybearded old scrote

I take the view that delete is a hangover from the days of floppy disks and "enormous" hard drives that wouldn't hold one of my photographs.

My projects tend to have a soft delete and perpetual undelete, just because storage is cheaper than lost data. Although that wouldn't be appropriate in a GDPR adjacent application.

Stuart Castle

Not really a use of my IT skills, but when one of our users reported their laptop had been stolen, I looked up the footage from the nearest camera, which had actually recorded the theft. Unfortunately said thief had long gone, and the room was open to the public for a special event, so we had no access records.

I dutifully phoned 999, reporting the crime. Within a few minutes, I got a call from our local Police station, so offered them access to whatever they needed to solve the crime. Their response? "Here is a crime number. Please investigate the crime, and let us know if you find anything." The user was obviously not impressed, but chuffed when we were able to to claim on the insurance and buy her a new laptop.

GlenP

We had a visit from the Police at work, asking if they could view our CCTV footage for a particular night as a crime had been committed nearby. Due to the ancient system we had then it wasn't easy to just view it, but I did offer to drop it onto a memory stick, which they were delighted about - promising to collect it in a couple of days.

That's the last we ever saw of them - so obviously it wasn't that important and I wasted my time trying to help!

The most they might have got anyway were times vehicles passed, the cameras didn't cover much beyond our immediate property boundaries and were poor quality.

A similar story involving core store

Alan J. Wylie

A previous post of mine:

[1]"You mean that the top secret contents of memory don't get lost when we power off?"

[1] https://forums.theregister.com/forum/2/2017/11/03/on_call/#c_3336077

Manager and Cashier

chivo243

I wonder what their plan was? Just to nick the cabbage and have a nice vacation, or pull a Steve Miller classic, Billy Jo and Bobby Sue... oh oh take the money and run!

CEO investigation

Anonymous Coward

Anon, of course.

Background was a 9BUSD turnover international company, publicly listed. The share, if I recall correctly was tanking to the point, after the facts, I remember it was almost every day claimed to be, depending on whether it was an odd or even day of the month, "the highest rise" or "the highest fall" of the index. LOL. Yes, share value was pure white noise, depending on a single investor's sneeze ...

I was once summoned by top mgmt to investigate the CEO's email, for all backups in the past. A lawyer would come personally to pick up all gathered data.

So I spent one month+ restoring every backup from years before, after rebuilding a mail system (complex feat, the solution was phased out for years).

In the end, I was able to recover a fair share of data (50 %), but nothing was found that I heard of, by the lawyer.

I had the time to briefly look at the emails, and saw this bollocks CEO was spending a fair share of time planning for his kids' career. Not really a surprise the company had done so bad during his 25 years tenure !

Hopefully, the company is now mostly recovered, but this was a scary insight into some dodgy CEO life ... And I've stopped investing in any share ever since.

ahem...

chivo243

"You mean," came the horrified question, "when we press delete, the emails don't actually delete, they get saved to backup tape?"

Being a senior manager, you did sign off on implementing this system, you read the documentation, did you understand it??? LOL

Re: ahem...

Jeroen Braamhaar

Two kinds of people:

those that manage what they do not (want to) understand

those that understand what they don't manage

sometimes supplemented by the third kind:

those that don't manage what they don't understand (but collect big consultancy fees for)

Re: ahem...

KittenHuffer

I could have sworn that CEO the 3 rd kind was a film directed/written by Steven Spielberg in the late 70's!

I once did ...

KittenHuffer

... some forensic auditing at a garage, following parts being ordered from suppliers, to receipt, and booking out to workshop or bodyshop jobs. And then proving that the parts were not for the make/model of vehicle in the workshop/bodyshop. This was then used in the court case of the mid-management employee that was ordering/stealing the parts to order.

The system was old enough that it printed out an audit trail of transactions at the end of each day, so the actual w**k was wading through stacks of printouts taller than the PFY that I was.

Eliminating the impossible! ----------->

Re: I once did ...

My-Handle

And some of those middle managers can really be impossible....

Sorry, just came from the BOFH thread.

J.G.Harston

So, when they delete something, they want you to send a truck to the off-site storage, load it up with tapes, bring them back on-site, load each tape in turn,track down the file on each backup tape, and remove it, thereby changing history. What's his name, O'Brien?

Turning up at work to a bunch of police

Steve Kerr

In a previous job turned up for a shift with a bunch of police that wanted to talk to me - scared the crap out of me.

Turns out, they wanted to listen to a voice recording of a particular line (lots of dealer lines & also some switchboard lines were recorded) and management told that to come to the building and I'd do it but didn't tell me that they were arriving.

These were analogue tapes that had something like 48 lines recorded and the tapes ran for 24 hours at a time.

Anyway, I ran thru a tape over a time period for one particular external line whilst they listened (and obviously I did too).

What transpired was there were some bomb threats being phoned in to reception, in listening to a scratchy recording, could hear a woman in the background and a child, it was one of the managers in the midsts of a mental breakdown phoning in bomb threats. Don't know what actually happened to him in the end. I was asked by the police about what I thought I heard.

Have had to listen in to a bunch of other dealer conversations which included hearing the sex lives of several of the dealers - oops!

Get it up, keep it up... LINUX: Viagra for the PC.

-- Chris Abbey