Housekeeping and kernel upgrades do not always make for happy bedfellows
- Reference: 1613376914
- News link: https://www.theregister.co.uk/2021/02/15/who_me/
- Source link:
Our story, from a reader Regomised as "Aapt", takes us back to the days when the DEC Alpha reigned, er, supreme over computer labs and anyone with even a passing knowledge of Linux could fall all too easily into the "expert" bucket.
[2]
Aapt was not short on confidence, be it tinkering with the xconfig file to persuade a monitor to work with Linux or, as we shall see, casually attempting a kernel upgrade. "I was convinced I was the best sysadmin for my lab computers," he told us.
Since finding a decent sysadmin for Linux was problematic back in the last century, and the institute where he worked had yet to do so, Aapt fell into the role. "For 'best sysadmin' read 'only sysadmin', for whatever I was worth," he said, with the wisdom of decades now under his belt. "Thank goodness they didn't ask me to administer any other institute machines..."
[3]
Thank goodness indeed.
Aapt's little world consisted of a lab containing a mighty DEC Alpha workstation, which performed compute and fileserver tasks, and a bunch of x86 machines that he considered poor performers in comparison. The Alpha had, after all, run without a reboot for over year. The same could not be said for the other hardware.
Eventually, however, the time came to upgrade the Linux kernel.
"I didn't back up the OS because, after all, I was going to upgrade it, wasn't I?"
The new kernel was compiled and made ready to be placed on the required partition.
A tidy fellow, Aapt decided that the upgrade would also be a good time to clear out all the system files.
"In a fit of enthusiasm, I set off the dreaded rm -rf on the /usr partition."
For those unaware, the rm command deletes files. Adding the -r option removes directories and their contents recursively. The -f option stops prompts before removal.
"I sat back meditatively," he said, "listening to the crunching sound of the disk head skittering around killing all those bits.
"Fortunately, this took long enough for my meditations to come to the point of thinking about the relative merits of pointing the bootloader to the new kernel vs a complete reinstall from scratch."
Aapt had been through the delight of an Alpha install over a year previously and the experience had "Not Been Fun."
Gut-swooping moment of fear upon him, he frantically jabbed Ctrl-C until the destruction abated. How much of the system had survived?
"The great Penguin in the Sky smiled upon me," he said, after gingerly trying a few commands (presumably not rm -rf this time) to see what remained. Enough, just, to complete the upgrade and update the system files.
"I never told anyone about my nearly successful attempt to delete my OS while it was running," he said.
"I'll just say that this experience substantially reduced any reluctance I may have had to hand things over to the institute sysadmin, when we finally hired one."
We have fond memories from the last century of an overpromoted DBA insisting that using DEL *.* on a running SQL Server would be fine since SQL locked .MDF files and .LDF files. Oh how we laughed when we discovered that backups hadn't been taken first.
How about you?
[4]
Share your tale of the moment you realised you didn't really know what you were doing after all with an email to [5]Who, Me? ®
Get our [6]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YCpUXRtpl9VUTllI2x-mrAAAABQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YCpUXRtpl9VUTllI2x-mrAAAABQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YCpUXRtpl9VUTllI2x-mrAAAABQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] mailto:whome@theregister.com
[6] https://whitepapers.theregister.com/
"let's just get rid of the stupid hidden files, shan't we?"
# rm -rf .*
yes, the promt was # (me doing this in the /root folder as root)
not $
oops.
The secret to intelligent tinkering ....
.... is to keep all the bits!
For computers this means making sure that you've got backups of the bits you're just about to screw with!
Re: The secret to intelligent tinkering ....
I think it takes 'an event' to make most people think seriously about the question "What's the worst that can happen?" before they hit the button.
Re: The secret to intelligent tinkering ....
Yep, it's the curse of the young and over-confident sysadmin. Nothing like a trashed file system or major outage to bring someone down to earth with a bump. I was fortunate when much younger in that I witnessed someone else at my first place of work cock things up and bring down half the network. Seeing the panic unfold around the place as people tried to coax an entire rack of servers back to life taught me an important lesson about backups and testing before doing anything to a live environment...
Re: The secret to intelligent tinkering ....
I was "lucky", I saw it by other people first.
I tend to have the problem that I have so many copies of the bits, I have to shuffle through and find the "latest" copy.
I currently have Carbonine for continual backup, a cloud drive for sync (sync is not backup!), rsync from SSD to spinning rust, rsync from spinning rust to NAS and manual backups from NAS to an external drive.
The early Apricot PCs weren't IBM compatible, one of the differences being they used A: for the HD.
Doing some work on a customer's machine after I'd been using the compatible machine back at the office.
Insert floppy
A:
DEL *.*
S**t!
^C
Search for Norton Utilities disk, UNDELETE...
Fortunately the time between typing DEL,,, and ^C wasn't too long and I knew enough about the structure of the system to be able to figure out the first character of the filenames after UNDELETE had done its best,
That's when I learnt always do DIR before DEL
One thing i instill in all the juniors I mentor is always always be in control of your own destiny when making a change on production systems, never rely on the infra backups being fit for purpose, never assume anything you haven't seen with your own eyes or done with your own fingers is there, even then be sure that you have copied what you think you have. Being able at very worst to just be able to put everything back how it was and walk away for another day is the key to a happy techie.
Delete is written rename
I learned that delete is a two stage operation
1) rename the file
2) next week after the change has been running for a week, delete the file.
Re: Delete is written rename
:) but what is that poor techie 5 years down the line going to do if you haven't left that ancient config file lying around that has the password in for the DB that everyone has forgotten. ;)
Re: Delete is written rename
That's why GUIs have waste-bins.
Re: Delete is written rename
Or Recycle bins.
The only exception to this was the Shredder in OS/2. It did its job too well.
Re: Delete is written rename
Except it is usually the "the files are too big for the recycle bin" files that are permanently deleted that are the ones you probably need!
DOS/VS build problem
One of my first jobs 40 + years ago was in build. I built a major application on DOS/VS running under VM on IBM 360.
Disk space was short and we only had 2 disks/systems - the "current system" and the build system.
I was given an envelope with a short list of instructions which started "delete the old build". So off I went.
About half an hour later we had some developers came round to say there were some files missing... I was working on the live system - not the build system. Whoops. I got a bollocking and the instructions were amended to "1) change to the disk to the value written on the board".
Next Monday I started the next build.... and half an hour later a developer came round and said there were some files missing. I got another bollocking because someone had forgotten to update the board!
The next week I got moved to work on VS1 where I could do no damage. All important disks were read only.
But why clear out /usr anyway? Even if the files were going to be updated the old ones would have been replaced with the new ones. It''s not as if they'd have been left hanging around.
Depending on how far the new kernel deviates from the old kernel, you might very well have new commands with new names that supersede old commands. Installing the new will not get rid of the old, which can raise all kinds of merry hell.
This was especially true in the early days of Linux, when people were still in shell-shock after the UNIX Wars and weren't quite certain which direction Linux was going to move off in.
I feel for "Aapt"'s pain
I did something fairly similar once on a Sun 3 workstation (which really dates this event for those in the know) running SunOS. We had an considerable accumulation of files in /tmp that where starting to cause issues (SunOS insited on putting /tmp on a separate partition that never seemed to be large enough), so one morning, nice and early before anyone started work, I decided to have a clean up. Having logged in as root, I then entered the dread command "rm -rf / tmp/*" - yes I had accidentality inserted a space after the first slash.
A rapid control-c proved futile - too much was gone. I had just enough left to perform a backup of the important files, then spent the next couple of hours reinstalling the OS from tape.
Lesson learnt the hard way.
Rassen frassen
AUTOEXEC.BAT sets TMP=C:\DOS
sometime later, something does DELETE /R/Y %TMP% .....
I started my working career in a specialist software dev house where all the developers had SPARCstations of various flavours. The configs, while generous at the time, where tried by the work we were doing .. particularly disk and memory. One of the other devs, need to free up space on his hard drive and started deleting a stack of files he never 'used' from /, /etc, /bin, /usr, /dev, /mnt and so on.
All was fine until the next reboot when we it didn't. Our sysadmin spent many happy hours in PROM mode rebuilding just enough of the files with echo "xxxx" > 'filename' until he could minimally boot it and restore from back up. Happy days :)
Mistakes are how we learn
I've done "rm -rf /usr" before. Fortunately it was on a personal Linux machine. For whatever reason I'd made a copy of /usr and then came time to remove it. Of course muscle memory kicked in and put the '/' before 'usr'.
Back to the title of my post. I got into Linux in the late 1990s. Back then, you learnt by scouring Usenet groups and using this up and coming search engine called 'Google'. (I wondered whatever happened to them?) You'd try something, break your system and work out how to put the pieces back together. Over the years I've noticed a trend for newer users not to experiment but want the answer spoon-fed to them.
We have fond memories from the last century of an overpromoted DBA insisting that using DEL *.* on a running SQL Server would be fine since SQL locked .MDF files and .LDF files. Oh how we laughed when we discovered that backups hadn't been taken first.
I think the time has come that I can now safely say that I borked an Exchange solid - by deleting all the journal log files, but I mistakeny ASSumed the Exchange database files will be locked.... I deleted those files as well... and no backup in sight....
....was a very interesting week. Company was not amused. This was about 10 years ago.
Learnt my lesson then. Always, always have a backup (and know what you're doing)....
Just over a billion years ago ...
... as the Internet measures time (call it roughly 1984), I received a brand new Sun 2/160. It was a dual pedestal beast, with all of 8 Megs of RAM and a pair of 380 Meg CDC SMD drives. Roughly 65 grand worth.
I decanted it from the boxes-on-pallets, plugged all the cables in, and fired the thing up. Into a beautiful new GUI on the Sony Trinitron monitor, just as advertised. Logged in as root, on purpose as there were no other accounts as yet(!!), using the default password(!!!!) ... and poked around. All was well, near as I could tell.
The plan was to repartition the disks to better suit our needs and then reinstall the OS. So I made absolutely certain I had the correct tapes, and did the one thing I had never done as a sysadmin ... closed the GUI, and from the # prompt ran rm -rf / intentionally. I was curious to see how long it would take to lose it's tiny little mind. It trundled away to itself for a few minutes, but seemingly was still working fine, enough of vmunix and the shell were in RAM and the swap partition to keep doing simple stuff. I was quite surprised, but that wasn't really what I was there for ...
So I shut her down, went and got a cuppa coffee, reached for the tape and went to fire up the machine ... only to discover it didn't ship with a tape drive, despite one being listed on the packing list. It had a lovely beezel that LOOKED like it might be a tape drive, but the space behind it was empty. Oops. So there I was, stuck with 65K worth of dead Sun hardware that I was supposed to demo for the Brass at 4PM.
Fortunately the 1980s Sun had Clues about customer service. One call, and their field service rep had the SCSI tape drive, the requisite cables, and a couple of VMEbus cards, (E)EPROMS and spare OS tapes "just in case" on my desk in under forty minutes. She even hung out and made certain that the system worked properly after we took it apart to install the bits that needed installing, and then partitioned it and re-installed the OS.
I made the 4 o'clock deadline ... and bought the Rep the first of many well deserved dinners.
Silly Con Valley was a very small place back then ... Sometimes I miss it.
Lucky escape without any real problems (except for probably an UUR experience).