Lost in translation and adrift in cloud storage
- Reference: 1585551785
- News link: https://www.theregister.co.uk/2020/03/30/who_me/
- Source link:
Our story concerns "Dirk", who at the time of our tale was hard at work in a Netherlands IT department. Dirk himself didn't actually speak much Dutch; his first language was English but that was more than enough to get by with in the land of clogs, windmills and dykes.
Having enjoyed a few relatively peaceful months in the job, Dirk told us he was "getting on with tidying up the systems and dealing with the technical debt that had accumulated since the last Big Cleanup a number of years previously."
"One such system," he said, was "cloud-based storage." The vendor was a well-known giant of the industry, but for the purposes of this story, Dirk called it "Poodle"
The task that day was dealing with obsolete accounts, not just from a security standpoint, but also due to licencing costs and, he said ruefully "It seemed like a good idea at the time."
Dirk ploughed through obsolete accounts until he came to one with an odd name: "beheerder". Perhaps an amusing play on "beheader" by some long gone techie? Or something to do with "herding" files? Dirk checked in with his predecessor, who gave the digital equivalent of a shrug. The account had been around since things had been set up back in the day, but nobody had used it in ages.
Indeed, it had been well over a year since anyone had actually logged in using the account. The password was reset and the mailbox scrutinised.
"Perhaps some more notice should have been made of the fact that at one point it was receiving system error messages," sighed Dirk, "but what use is hindsight?"
Anyhow, the messages seemed to have stopped recently and the rest was just junk. To keep things spick and span the account was deleted and, as was standard practice, all files transferred to Dirk.
He thought no more about it until a fortnight later when he decided "to move all those transferred files to the new-fangled Poodle 'Shared Drive'." Heck, everyone else's documents were due to moved there at some point ("with the correct access permission set," he added.)
Dirk clicked "Ignore" on the standard "Files accessed by other users may be affected." This would prove to be a fateful click.
After 20 minutes, one of his colleagues to call to complain that they couldn't access one of their files.
A coincidence, surely. But Dirk began to feel the arse-swooping sensation of dread that something awful might be happening.
Perhaps a little later than he should, Dirk took a closer look at the files being moved. He looked a little closer since some were still in progress. He poked further, down two or three folder levels and saw something that looked familiar.
"Horrifyingly familiar. In fact, it almost looked exactly like the core folder structure that everyone in the office used."
A little more investigation and Dirk realised that he could skip the "almost."
As his stomach sank through the floor, Dirk realised "that over the next few hours file access will disappear for all users."
"It wasn't immediate," he added "because file storage is cloud-based and sometimes it takes its own, undefined, time to do large file moves."
The helpful interface afforded no way to stop the hellish process of borkery. In a panic he called the cloud vendor, observing: "They're rather large and helpful in inversely proportional quantities."
And, of course, they couldn't help. For whatever reason, "there didn't appear to be a way to pause or stop the oncoming slaughter. Everyone was going to lose access to their files."
Having told the increasingly anxious users that there was a "small" problem, Dirk peered at the unfolding carnage he had inadvertently wreaked upon the company's files:
"Some were still in the old location, and some were in the new location."
He tried to move the files again. And again. On the third try he got some response; the system helpfully told him that it was "moving files", but without any kind of countdown of progress bar.
"Empty folders were being created," he told us, "and since the system was still progressing the previous move requests things got very complicated - sometimes files were moving and sometimes not. And sometimes the files were copied before they were moved. On the plus side Poodle Drive would happily create folders before deciding whether or not to copy the files, so that's something isn't it?"
What had happened was that the user "beheerder" had created the root folders back in the day, and other users had created subfolders. A little too late, Dirk discovered that the Move function he had used "only affects folders that you have ownership over. This doesn't include subfolders you didn't create."
"Suddenly," he said "there was a smorgasbord of folders spewn around the system. Some stuff stubbornly in the old location, and some stuff in new locations. Hundreds of gigabytes of data in fact. Smeared everywhere."
The good news was that was a backup, so nothing was actually lost. The only challenge was decide to "Restore" or "Restore with File Permissions." Fearing what would happen to existing files, he restored without those permissions. At least then he'd be able to see all the files and put them back correctly. He could then tidy up the folders and set the correct access levels.
"It took a month," he said.
"Hindsight says that I should have checked out the other option too, but maybe next time(!)"
Our tale ends happily. The users were eventually happy. The storage was tidy. Dirk, while he obviously paid wasn't overtime for his efforts, survived. He also learned to take things a little slower and think a little harder about what those messages were telling him.
Heck, he's even learning Dutch.
Apparently, "beheerder" means "administrator".
Ever been bitten on the behind by cloud storage, or gaily skipped past a message box without fully understanding what it was telling you? Sadly, it is all too common. Share your tale of woe with the sympathetic vultures at [2]Who, Me? ®
[1] https://www.theregister.co.uk/Tag/who-me
[2] mailto:whome@theregister.co.uk
google translate anyone?
I am native Dutch, if I was working in another country I would run any admin accounts through google translate to see what they were. There must be a root/admin account somewhere and it should be marked as such.
Re: google translate anyone?
1) google translate still is sort of... unreliable
2) it was doubly so waaay back
3) who creates admin accounts with a localised name? I'm not a native English speaker, and have worked in countries where English is not the main language. I have not come across that in the last decade at the very least. Not even two decades ago in a very small group - I'm not saying it does not happen (it does), but, hey, who in their right mind... (ok, wrong question)
4) his predecessor basically shrugged - not sure if he/she was dutch speaking - maybe? Should have spotted that.
Oh, and hindsight is 20/20, as they say...
Re: google translate anyone?
> who creates admin accounts with a localised name?
Maybe the localisation system? I don't know this particular system, but if you install Windows to use Finnish, it helpfully shows nearly everything in the UI in Finnish. The underlying admin user name might not be altered, but what the GUI shows is.
My son accidentally set the language to Spanish when taking his new Windows PC in use. That was fun (and lucky it was not Hebrew or Chinese). It still occasionally shows some texts in Spanish, even after trying to tell it we want to run it in Finnish. Probably would require a reinstall.
Re: google translate anyone?
You don't even need to go to google translate, just highlight beheerder on the page, and chose search in google from the context menu, and the translation to administrator is right there.
Apparently, "beheerder" means "administrator".
After many years in NL, this one brought a chuckle... ja hoor!
Re: Apparently, "beheerder" means "administrator".
Throughout the whole article I was reading it as "beeherder", and thinking "I know those Dutch are crazy, but WTF?"
More caffeine required.
Re: Apparently, "beheerder" means "administrator".
@jonathan keith - 'I was reading it as "beeherder"'
Maybe not so different, they're both difficult jobs, and you're going to get stung!
With the lockdown, sorry, but no beer.
Reason #3641 ...
,,, why one should learn the fine art of homebrewing.
.. and that, for anyone still wondering, is why, when seemingly everyone else was flocking to the bog roll aisle, I was quietly cleaning out all the local supermarkets' (admittedly rather meagre) supplies of Weissbier.
Got to get the survival priorities right, after all.
[instead of bog roll] I was quietly [buying] supplies of Weissbier.
I take it your bidet is fed from a local storage tank, rather than mains water, then?
If it had been....
....catheerder it might have been more obvious.
Re: If it had been....
kat herder?
Icon: my wife's opinion of cats
The problem is not beheerder
One can argue about Google Translate and its relevance back then, or how the previous admin should have been a bit more knowledgeable, but the real problem has nothing to do with the admin name.
The real problem is that this Dirk launched a file operation without checking what he would be transferring. It's that second of inattention that always gets you, and it got him.
When you are the admin, you never initiate a file operation without checking and double-checking what you're doing. With your access, it's way too dangerous not to. He didn't check, and the mayhem lasted a month. I'm guessing that was a learning experience for him.
Re: The problem is not beheerder
Rather like fire safety - always ensure you have two exits available.
Re: The problem is not beheerder
In the DIY business, it's "measure twice and cut once." An aphorism many, across many disciplines, could do well to acknowledge...
And there coimes a point in IT
where you must learn to distrust the automated function and perform it manually and in a phased approach. This was one of those times.
Beeherder
I was offered that as a job once wiith three times my normal wages but I didn't take it.
It was a sting!
Security?
I'm under the impression that renaming root /admin is a security feature.
Re: Security?
That's what I thought - sure I've read somewhere that you rename the admin account to something innocuous, and set up a "dummy" admin account for the hackers to waste their time with?
beheerder
Came across such an account some years ago. At that time, for all I knew, this looked just like any another user name following more or less their naming convention. I wondered though (in fact, I got quite shirty), why a person who didn't even belong to the IT department had administrative permissions on the system.