Techie's quick cure for a curious conflict caused a huge headache
- Reference: 1691398751
- News link: https://www.theregister.co.uk/2023/08/07/who_me/
- Source link:
This week, meet "Bruce" who was once and IT Helpdesk Analyst for a "large bluechip healthcare organization" in Australia. That sounds a very impressive job title, but Bruce tells us he mostly spent his days "repairing computer issues, troubleshooting networks and installing and upgrading equipment."
On one of those days, he was replacing a PC in a doctor's clinic. Each machine ran a Windows standard operating environment, and each machine had an asset tracking sticker on the side. Usual practice when setting up a machine was to give it the same name as that printed on the asset tracking sticker. All fair dinkum so far, as we say in Australia.
[1]
But when Bruce strode up to this machine he discovered its name was already in use on the network. How that had happened was anyone's guess, but Bruce decided to fix that problem another time and just for now give the machine a temporary name. Any name. Whatever would do. First thing that popped into his head.
[2]
[3]
He used the name of the company.
Now, you might say that it was likely that name would already be in use by another machine on the network, but evidently it was not. The network accepted the name, the machine booted, and all was evidently tickety-boo, as we say in Australia.
[4]Nobody would ever work on the live server, right? Not intentionally, anyway
[5]Linux lover consumed a quarter of the network
[6]Network died, hard, during company Christmas party, leaving lone techie to fix it
[7]Turning a computer off, then on again, never goes wrong. Right?
Then, the call came. Bruce's boss had received a call from the CIO, who had received a call from the head of infrastructure, who had been called by one of the senior techies. They all had the same question Bruce's boss now put to him: "What did you do to the intranet?"
The entire corporation's intranet was down. Not just in that building, either. As Bruce puts it, this was "a company with 25,000 employees and 800 servers spanning 120 locations and three continents" and "no-one could connect remotely from outside the network, they couldn't access forms for annual leave, purchase order requisition forms, educational information and just about anything else you can think might be on an intranet."
[8]
We have a response to that sort of thing: "Somebody must've messed that up."
It transpired that the corporate disk image, on which every single PC in the company was based, had Microsoft's IIS web server enabled. That's a bad idea and it wasn't Bruce's fault that it was running.
What was Bruce's fault was that he'd given that computer, which the network was seeing as a web server because IIS was enabled, the same name as the corporate intranet. It makes perfectly good sense that the company name was the name of the intranet, of course. It's the first thing any employee would think of, right?
[9]
So Bruce, in trying to get past that little problem of the computer's name already being in use, had redirected thousands of employees' intranet requests to a single PC that, while it was running a web server, was not driving the company intranet.
Strewth.
That was relatively easily fixed by disconnecting the PC, which meant all traffic routed back to the actual intranet. Then it was just a matter of setting the PC up again with a unique name.
And perhaps deactivating the web server on PCs that didn't actually need one.
Ever done a little thing that made a big mess? Tell us all about it in an [10]email to Who, Me? and we'll share your exploits with the world. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZNDAxusgD62FgKj@g2LO6gAAAlA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZNDAxusgD62FgKj@g2LO6gAAAlA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZNDAxusgD62FgKj@g2LO6gAAAlA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2023/07/31/who_me/
[5] https://www.theregister.com/2023/07/24/who_me/
[6] https://www.theregister.com/2023/07/17/who_me/
[7] https://www.theregister.com/2023/07/10/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZNDAxusgD62FgKj@g2LO6gAAAlA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZNDAxusgD62FgKj@g2LO6gAAAlA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Especially in a company that installs a web server on every PC and has no use for it.
That was a brilliant idea there, or a wonderful absence of checking that the image had only what was actually needed.
In either case, if I had been in charge, someone would have been raked over the coals for that (not Bruce, I hasten to add).
"Ever done a little thing that made a big mess"
Who hasn't ?
Just a few weeks ago, I was at a customer site during a migration to Exchange. I was asked to add an Internet Address to all the groups in the Name & Address book. The format was supposed to be I'm not an admin, so although I found the request curious, it is not for me to speak up on such a matter. I created the required script, ran it, and all appeared to be well. For about five minutes. Then calls started coming in to the Helpdesk. People were receiving weird mails that they shouldn't have and didn't before. It just so happens that some groups had a group name that was very similar to some redirections defined on the internet side of the portal, and anything coming in from outside was now sent directly to all the names in those groups. Oops. Needless to say, I was asked to revert the change post haste, which I did in record time. I then left them to devise some new format which wouldn't create a new dogpile of issues. Maybe by checking incoming redirection lists first ?
Re: "Ever done a little thing that made a big mess"
" I'm not an admin, so although I found the request curious, it is not for me to speak up on such a matter. "
Bad things happen when good people stay quiet.
Re: "Ever done a little thing that made a big mess"
A migration *to* Exchange?
That's your problem right there .....
Re: "Ever done a little thing that made a big mess"
Yep, I've done something like that before.
I reset a little wifi router on the office network back to factory settings, so that I could log in and reset the password that someone (cough) carelessly lost. I learned a few things from this small change. A) What a DHCP server was, B) most home routers have this enabled by default, and C) it's a bad thing when you suddenly have a second DHCP server on a network that doesn't talk to the first and hasn't been configured with the correct network information. I found all of this out when PCs on the network started dropping off randomly. The IT Director asked if anything new had been plugged into the network, and I remember the sinking feeling I had when I remembered the reset router...
As this is Australian, I'm just pleased the webserver didn't turn out to be some big nasty venomous spider...
Back when I started my first proper IT job back in 1998 I was sent to the US head office for 6 months training. As part of my training I had to set up and install a Communication Server (a server that talked to third party systems). I was pointed to a clean PC, given the Software and an instruction manual and told to get on with it. Simple enough.
The first step was to install Windows NT Server on the box, which I dutifully did. I then proceeded to install our own software on the newly installed NT Server. While merrily working away installing and configuring our software on the nice new NT Server, I noticed out of the corner of my eye, the service manager race past my desk (which was quite impressive as he was getting on a bit). And then he raced back in the other direction. And then our internal Systems Administrator ran past. And then back again. Soon lots of people were running about the place looking rather agitated. The next thing I know, the Service Manager pops over to my desk, taps me on the shoulder and very calmly asked if I had just installed NT Server…
It turned out the manual I was following failed to mention that when you install the Windows NT Server you MUST set it to a Secondary Domain Controller, otherwise it decides it’s the Primary Domain Controller and takes over from the existing Primary Domain Controller. I had just managed to take down the entire network in our Head Office!
Luckily the company was very easy going so I never got in trouble. I think they were more impressed that I’d managed to install and set everything up from just reading the manual (as the manual failed to mention setting the Server to a Secondary Domain Controller I suspect I was the first person to manage it). Obviously, the manual was very quickly updated.
Don't you just hate it when the technology is actively out to get you.
Sure, naming a single random PC after the company is somewhat maximising the chance of something unfortunate happening, but the chances are still so slim that it'll cause any major issue that it just reinforces the point that computers are vindictive buggers and You Can Never Trust Them.