Client demo in 30 minutes. Just what could go wrong?
- Reference: 1647592086
- News link: https://www.theregister.co.uk/2022/03/18/on_call/
- Source link:
Our hero, whom we shall call Simon though that is not his name, was gainfully employed at a UK telecoms outfit way back in the mid-1990s. Carrying the vaunted title of systems engineer, he was based in the City of London doing pre-sales work for some of the world's biggest finance companies.
High-powered stuff, indeed.
[2]
One of these financial firms held its annual conference in a different international city every year, inviting potential and existing clients to wine, dine, and generally have a good time at its expense. The year our tale takes place, a new wheeze was suggested: why not have one of those new-fangled internet cafes at the conference hotel?
[3]
[4]
Simon was quick to volunteer. After all, the city selected was Madrid, which enjoyed both better weather and better cuisine than the London of the 1990s.
How hard can it be?
The plan was simple: fly out to Spain, form a network of PCs at the hotel, initiate an ISDN connection back to base in the UK, pipe in some internet goodness to this pop-up cafe, and demonstrate your employer's marvelous communications capabilities.
"Being reasonably techie," Simon told us, "I established in advance exactly what I would need in terms of the kit – a Cisco router – and exactly how to configure the ISDN modem that would be provided in Spain."
After spending a few days making sure everything was set up on the UK side, Simon – Cisco router in his luggage, wedged between fresh Marks and Spencer underpants – boarded a flight to Madrid.
[5]
It was Sunday, and the cafe had to be ready by 10am Monday. All went well: the PCs, provided by the big financial firm, were set up, and the router was freed from Simon's underwear and plugged in. A link back to the UK was established, and Simon was surfing the internet as fast as 1990s ISDN technology would let him.
It was a good end to the day. There is no record of whether or not he celebrated with a well-earned adult beverage. We like to think he did. Several.
Sadly, the inevitable going-wrong happened the following morning. It began well. The PCs fired up, and Simon checked that the accountancy websites to which the customer wanted access were working. At 8.30am he treated himself to a pastry and some coffee.
[6]
At 9am, with the customer stood next to him, it all went wrong.
"A quick router reboot, problem still the same," recalled Simon: "I could access any site I wanted providing I knew the exact IP address, but unfortunately I couldn't resolve the domain names.
"By now it's 9.15am and my customer's clients are due to walk through the door at 10am."
With the icy finger of fear jabbing his neck, and under the fixed gaze of the customer, Simon put in a support call to the UK.
"I was informed the DNS server had overheated," he said.
We have redundancy, we have batteries, what could possibly go wrong? [7]READ MORE
Forget an air-conditioned server room: it turned out this DNS server was sat under a programmer's desk in some provincial British town. This would explain the inability to resolve any domain names.
With the big hand making its way inexorably to doom o'clock, when the customer's clients would arrive, an increasingly panicked Simon asked if perhaps the server could be fixed? Or replaced? Or something? Now?
The bored voice from the UK droned, "What's your maintenance contract number?"
Simon snapped. The usual "don't you know who I am?" – in this case, the guy about to screw up the most important demo of the year – cut no ice. Expletives slid off the Teflon tech support staffer, who was secure in the knowledge that hundreds of miles separated him from Simon's wrath.
"Eventually only the threat that my managing director, a well-known American at the UK telco, would put a rocket under somebody's backside had the desired effect," recalled Simon. By 9.55am something finally happened. Replacement or reboot – by this time Simon didn't care – and five minutes later "all was sweet and light once more. Phew!"
Ever been as ready as you can, only to have all your hard work undone by slipshod behavior hundreds of miles away? Or maybe you were that bored voice on the phone wondering if you could squeeze in a tea before rebooting the box? Tell us, with an email to [8]On Call . ®
Get our [9]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YjRmYoMlat5narqOrI5CiAAAAMM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YjRmYoMlat5narqOrI5CiAAAAMM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YjRmYoMlat5narqOrI5CiAAAAMM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YjRmYoMlat5narqOrI5CiAAAAMM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YjRmYoMlat5narqOrI5CiAAAAMM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/03/11/on_call/
[8] mailto:oncall@theregister.com
[9] https://whitepapers.theregister.com/
Re: What's in a name?
We wouldn't have such a tracker problem if DNS was always at fault. Your link is tracker infested and it would be prevented from loading if DNS did not work. Lets all kill DNS to kill the trackers.
Re: What's in a name?
Lets all kill DNS to kill the trackers.
Sounds like a job for Pi-hole!
Re: What's in a name?
Like the one in the Pi on the rack just to my right, doing a sterling job for many years...
Re: What's in a name?
Except when it is a network cable.
Broken tab, and slipped out just enough for an unreliable connection.
Re: What's in a name?
Worse still, the ones that are in but only work when pulled back a bit.
Re: What's in a name?
> Except when it is a network cable.
To the DNS server?
From the World Famous BOHF excuse generator
"DNS server drank too much and had a hiccup"
Ah, DNS
A place i used to work at had a hideously complex solution for DNS.
Some core DNS servers looking after all sorts of name spaces
A couple of ADs, each running DNS, with forwarders setup to the core
External DNS servers replicated the public bits of the namespace.
All tied together with a bunch of scripts, schedules databases and who knows what. The people that put most of this together had left long ago so that the only bits anyone really understood where the AD DNS (not that it is particularly difficult). The rest became a complete mystery and it was not unknown for all sorts of weird things to happen if the core DNS was given a funny look or a keyboard pressed too hard. Clients would "go missing" or only exist on one of the servers. Websites would only resolve if you were inside the border, or the reverse, public facing services would stop resolving and nobody could find out why.
As usual, smart people doing really clever things that are great as long as those people are still around to support it.
DNS can be as simple as you want or as complex........
Non-Techy tries too hard
10+ years ago at a previous employer, we had a nerd in HR, he had all of his home network devices set to static addresses, including the wireless on his mobile phone, which he would bring into the office, with the same IP address as the device handling our DNS**, fun times chasing that one down*. Funny thing is he was almost always the first to complain the internet was down!! When we figured it out, I was happy to drop the bomb on him! It's your own feckin fault! Nimrod...
*(he later claimed it kept him safe while torrenting?)
** this was quickly changed to something different, the network guy learned something!
Re: Non-Techy tries too hard
....he had all of his home network devices set to static addresses....
Given his A+ grade in "Fucking up DNS 101", this is hardly surprising.
Sometimes it can be dhcp
It isn’t always dns
I had someone do similar but they plugged a dhcp server into a production segment. The server was for building and imagine hp thin clients.
Could not work out why nobody could get an up address as it was giving out the same addresses as the lab but with itself as the gateway.
That was a fun morning….
It was BT for me
In the late nineties I set up a similar sounding "suite" of computers for a research conference we had organised. It was hosted at a local hotel and the hotel had an ASDL line (with BT) installed for us. Set up was simple, our university IT bods had supplied a router and configuration instructions and all was running well - the academics could all log onto their emails (most had login details written on bits of paper) the day before the conference started. Next morning there was no connection to the outside world. All the connections checked out, the configuration hadn't been changed. The telephone support that BT offered were confident that the fault was at our end.
I had to get the hotel management to call in the BT engineer. Said management made it clear we would have to meet the costs if there was no fault. Fortunately the hotel's ASDL contract was sufficiently expensive that an engineer arrived inside an hour. He checked all my wiring, configurations, etc. and could find no fault. He then made a call to the relevant exchange and two minutes and some sheepish looks later the system was working again. It turned out someone had unplugged the ASDL line at the exchange end
Re: It was BT for me
I've had a few run-ins with BT where I've reported a fault. They've come back with, "No fault found" but whatever it was has mysteriously fixed itself in the interim period.
Re: It was BT for me
Think you've got your dates or technologies mixed up. ADSL didn't launch until the early 2000s in the UK. Unplugging it at the exchange sounds about right for BT though!
BT certainly didn't have any back in the 1990s, it was the very end of the 1990s when we put in Colossus - which was the backbone that initially delivered the backbone network for Dial-up and leased line connections and later broadband services.
By 2000 all we had was a lot (250,000+) of dial ports and a load of leased lines.
X25 woes
Once upon a time an HPUX server had to be upgraded from 10.20 to 11.
It was top critical as it was handling all the bank transactions of a 9BUSD per year turnover company, through an X400 messaging relying on an X25 line.
No-one had dared to even touch the thing, but my then boss (a notorious cowboy) gave me this "gift" as a new joiner :)
"Easy stuff, mate, just boot on disks, install HPUX 11/X25/X400, then patch all with the repository, end of job". Bastard.
As it happened, all went well, that evening. Except the darn X25 daemon which would flatly refuse to launch.
It took me all night, many tired calls to HP support, with the first lines asking the usual trivial questions, before ending up at the X25 L3 specialist which gave me the solution, one I would have never found alone.
I left the building at 11:30 am, having started the day prior at 8:00 am, completely zombified ...
Re: X25 woes
HP-UX, X400 and X25? Congratulations! You won first prize for untold shit and misery in the IT (non Microsoft) world.
How much?
I remember going to a conference in the US. One of my colleagues, his first trip abroad, did a couple of back to back sessions logging on to the systems back in the UK.
These were the days before wireless where you unplugged the cable from the telephone, and plugged it into your laptop.
All went well - till he came to check out of the hotel and had a phone bill of thousands of pounds (more than the cost of his whole trip).
When he got home and tried to claim his expense, his manager paid (after lots of grumbling) and asked "Why didn't you use the AT&T freephone local number". No one had explained to the poor guy how you work internationally.
"Ever been as ready as you can, only to have all your hard work undone by slipshod behavior"
Yes. Every time BT is involved in anything. If you want something well and truly fscked up, you can rely on BT.
(the guys on the ground are generally fine, it's the management level that balls it up)
What's in a name?
As the old mantra goes, [1]it's always DNS...
[1] https://isitdns.com/