It's always DNS, especially when a sysadmin makes a hash of their semicolons
- Reference: 1606119305
- News link: https://www.theregister.co.uk/2020/11/23/who_me/
- Source link:
A Register reader, "Hugo", shared today's story, which takes us back to the late 1990s, when the commercial internet was an optimistic glimpse of the future and outages were a thing that happened when someone accidentally picked up an extension elsewhere in the house.
Hugo was a senior sysadmin for the UK division of a certain global ISP (let's call it "BigNet", for that was certainly not its name).
"BigNet," he told us, "hadn't invested much in tools and automation, and for many things we made it up as we went along.
Panic in the mailroom: The perils of an operating system too smart for its own good [2]READ MORE
"There was no customer web portal for anything and they had to raise a ticket, by email, for things like DNS changes which were then actioned by Customer Services."
Happier and simpler times. Until the day Hugo came into work and found the place in uproar.
"DNS was down for every customer, primary domains and secondary," he told us. "The brown stuff really had hit the rotating air displacement machines."
Hugo sprang into action, pulling the DNS server logs and swiftly finding errors. At first they made no sense whatsoever until an awful, creeping realisation dawned.
Remember how he told us that there had been precious little investment in automation? Included in that sacrifice on the altar of corporate perks were tools to edit the DNS. A few helper scripts were used, which basically invoked Vi to edit the zone files.
Vi, for those spared the editor wars, is a venerable text editor much beloved by Unix admins. Others swear by Emacs (others still have been known to just swear at Emacs, but we'll step away from that argument).
As far as the scripts were concerned, there was some simple templating to assist with creating a Statement of Authority (SOA) record, but no actual validation of the zone occurred. There was also no history or versioning. There was only the date and owner of the file.
It turned out that Hugo had made the last edit, two weeks ago.
"I had probably been working on a perl or bash script," he told us, "and on the same day I edited the zone file for uk.bignet.net which was in the SOA record for every domain we hosted."
He went on: "In Bind zone files, the comment character is a semicolon, but I accidentally used a hash, and whilst Bind loaded the zone file, it decided it was no longer authoritative, and this went unnoticed."
This was all well and good until the default two weeks time-to-live expired. Since every other domain depended on that one being valid and authoritative, that expiration meant Bind stopped serving all the other domains.
Result: chaos.
The fix was trivial. Hugo switched the comment to a semicolon, hurriedly pushed out the update and restarted all the name servers. The relief was palpable as the services came back up.
Hugo's fate was, unsurprisingly, to create a DNS zone file validator to prevent further "accidents". Mindful of his own brush with a pink slip, Hugo upped the paranoia of the tool from merely warning of errors to issuing a full-on stop when validation failed.
The customer services and provisioning team hated it "for reasons I couldn't understand," he said, "until I also ran the checker across all the domains we hosted and found something like 15 per cent of them had basic errors."
Hugo, it seemed, was not alone when it came to cavalier treatment of critical files.
Ever been struck by the curse of the wrong comment character? Or a mistake made weeks ago rearing its head in a most unpleasant way? Share your tale of woe with an email to [3]Who, Me? ®
Get our [4]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://www.theregister.com/2020/11/16/who_me/
[3] mailto:whome@theregister.com
[4] https://whitepapers.theregister.com/
Obviously they meant vi, not Vi. Damn that case-sensitivity. Actually, don't; case-sensitive is the only true way.
Mine was dhcp
Of course I locked down services, just a pity I missed a ] in one edit. Took about 20 minutes to work out why the whole network was down as it took a month for the error to hit. Took 5 to get it sorted and just held up my hands and put similar validation in place. God I hate networking
I learned the hard way not to use local drive mappings in Robocopy scripts. I remapped the destination folder's drive letter to another server later in the day without thinking, and I also ran the Robocopy job as a scheduled task out of hours with no monitoring or oversight, and so didn't spot the mistake whilst it was running. Oops.
The new destination server filled up to the point of failure "only" about half an hour before I got back into work the next morning, so I was able to bring it back up relatively quickly. But it still left egg on my face though.
Back in th eday? Still!
Zone files are edited manually where I am at the moment. validation - schmalidation!
Re: Back in th eday? Still!
They were at $LASTJOB too. There were some concessions to modernity though - edited zone file was deployed by Puppet and named-checkzone run before the thing was put live.
Re: Back in th eday? Still!
Quite right too. Where's the fun in not being able to bring the whole thing tumbling down with a simple typo? Life on the edge has a lot to recommend it.
A global ISP handling their domains for thousands of users with scripts and vi?
Sorry, but that's atrocious no matter what generation that was.
And the unforgiveable bit: No backup or history? It sounds like they just jumped into a single live config and went poking for the solution. Not even a "restore from yesterday", even if they wouldn't have worked? Because even "Restore from last week? No? Restore from last month? That's working? Cool!" would have got them back up without that kind of manual intervention and then they could go comparing files to actually see what changed.
If this was the 70's or something, and the tools simply not available, it's still not great. But the 90's?
You are overlooking some small details like manglement and funds, in the case of the latter more specifically lack of.
As it seems to have been a Unix environment, the tools were there long ago.
Back in the 80s I used to create scripts that invoked various bits of SCCS (together with vi) to provide change control on pretty much every config file across umpteen systems.
With makefiles to look after testing, rebuilding etc. for the more complex ones.
Fiddly to set up, but life is calmer once it's done.
Bollocks! In the right hands and with the right tooling, scripts and the occasional bit of hand-editing is just fine for managing Big DNS. That's how it still gets done. The registrars and ISPs who manage zillions of domains use a back-end database and write their own SQL scripts or whatever to generate their zone files and name server configurations, Most TLD registries do this too.
You can arrange for those tools and scripts to perfectly fit the organisation's IT operations, processes and procedures - trouble ticketing, change control, backups, testing, support handling, etc. That isn't possible with bloatware enterprise DNS "solutions" and crudware IPAM systems. With these you change your processes to fit what these piles of shit offer. Which usually isn't much - apart from a glitzy UI which impresses the IT directors who will never have to use it.
Another problem with enterprise DNS and IPAM systems is you end up with someone who knows how to drive these heaps of cruft - if you're lucky - but knows fuck all about DNS. Or how to configure and troubleshoot a name server.
BTW, DNS wasn't around in the 1970s. And neither was vi or emacs. The 70s and early 80s Arpanet used the hosts.txt file.
Programming interminable comments
During the years I was programming in Coral 66 I had several comment problems.
'comment' comments must end with a semicolon;
But it was easy to omit that terminator, especially if you were used to other programming languages. The result was that the next statement was treated as a comment until its final semicolon. I.e. it was effectively absent. This could cause mysterious problems.
I had a pretty-printer program for Coral that I had written, and it was the pretty-printer which finally revealed the mistake: showing the statement as run-on text at the end of the unterminated comment.
Surely since forever there have always been unintended consequences. It's par for the course ....
....... and results can be brutal.
Mindful of his own brush with a pink slip, Hugo upped the paranoia of the tool from merely warning of errors to issuing a full-on stop when validation failed.
Such considerations have moved on and into other fields of endeavour peddling and pimping/pumping and dumping content to/of the masses. And for probably pretty much very similar reasons .... maintenance of status quo conditions, no matter what the perilous state it may be in. But it is no fix, nor even an effective tool whenever wielded, for it then has invariably revealed a systemic weakness which the program/platform has scant defence against?
The following is a current example of the tool type/program gripe ......
We are unable to post your comment because you have been banned by Slugger O'Toole. Find out more.
Me 2
I did the same thing last week not with bind but a similar config file where I muddled the comment character oops...
vi v's Emacs
Real programmers use [1]butterflies
[1] https://xkcd.com/378/
Corporate edicts can be helpful sometimes
I remember a corporate edict that said all programs we ship must compile/assemble with return code 0. No warnings. There were all the usual moans about wasting time etc. We fixed all the warnings, and it was surprising the number of little/intermittent problems that just disappeared.
As one developer said "I was always suspicious of one bit of code, but never had time to fix it. I was glad when my manager came and told me to fix it"
cd ; rm -rf ./*
Doesn't work out so well if current user is root, current directory is / and
Re: cd ; rm -rf ./*
I'd say it is a good work out for whoever has to reinstall those 50 machines ;)
Comments in Bind?
Yes, it happened to me too, and not 20 plus years ago. But as far as I remember, bind would not reload the zone with the error, or something, so I noticed the error immediately.
Make
Some time ago (prior to the turn of the century) I was writing a lot of diagnostics for a product (using an [1]SGI Indy - fun times)
One fine day, I checked out the latest branch to address a requested feature but when I tried to compile, I got many screens worth of errors.
Turns out that the previous checkout (by users unknown) had loaded the makefile into their editor which converted tabs to spaces .
In that version of make, tabs vs. spaces had meaning so I had to get an older version of the makefile and check it back in.
That made for an interesting morning.
[1] http://www.computinghistory.org.uk/det/11261/SGI-Indy/
Anyone who ever dabbles in vi
Admit it, you've still got a running terminal window somewhere because you can't quite figure out how to close it.
(For the record, I prefer nano. It just works, though I refer to myself as a developer not a programmer so I'm not sure if I fall into the 'real programmer' camp or not)
Double quotes, versus single quotes, versus Microsoft "smart" quotes...that took a while to find.
Silent minus-to-emdash translation by $EMAILAPP recently got us when emailling code snippets.
The option to mail in plain text is buried deep in the menu system (and seems to change location over the [many, frequent, likely pointless] updates) and it has to be off by default because it messes up the corporate email chains.
Smart would have to be the most abused word in the English language at the moment. It is everywhere, and usually can be translated as dumb.
$ Vi
bash: Vi: command not found
$