Brilliant backups that kept data alive for ages landed web developer in big trouble
- Reference: 1773646206
- News link: https://www.theregister.co.uk/2026/03/16/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Neil" who told us he maintained a website for a client he suggested we call "Gerald."
"He would send changes, I would do updates, and he would approve it," Neil wrote. "We had years of smooth sailing."
[1]
"Until the day Gerald called, furious. The website was wrong, he said. Old content everywhere."
[2]
[3]
Neil checked and the site was fine. He verified that all recent updates were in place, and told Gerald all was well.
"Gerald told me what I could do with my perfect website," Neil told The Register .
[4]
Neil soon figured out what went wrong.
"When I'd migrated his site to a new server, Gerald had reviewed and approved it – from home, like a normal person – and I'd left the old site ticking over on the old server," Neil said. He added that he always does this because it costs nothing to keep an old site alive and it makes sense to preserve data until you're absolutely sure it's not needed.
"What nobody had told me was that Gerald's IT support had hardcoded the old server's IP address into the office's internal DNS," Neil explained. "So for two years, every one of his 40 staff opened a browser at their desk and saw the old website. Completely frozen in time."
[5]
But Gerald had never looked at the site from his office during working hours – for two whole years! When he visited the site from home, he bypassed the DNS server and saw the updated site. Once he finally checked it in the office, he saw the old version.
[6]Bug that wiped customer data saved the day – and a contract
[7]Server crashes traced to one very literal knee-jerk reaction
[8]Work experience kids messed with manager's PC to send him to Ctrl-Alt-Del hell
[9]Final step to put new website into production deleted it instead
"When I finally worked out what had happened and explained it to Gerald, his IT team were present," Neil told The Register . "There were two of them. They listened to my explanation, looked at each other, and then – with the quiet solidarity of people who have broken something expensive – explained to Gerald that actually, the problem was that I had left an old website running on another server, which had caused the confusion."
Neil says that was technically true, "in the same way that 'the floor was wet' is a technically accurate description of the Titanic's situation."
"Gerald looked at me," Neil recounted. "And I looked at Gerald. There were two of them and one of me, and they had lanyards."
"You should have deleted the old site," Gerald said.
And Neil did.
But not for another four months, just in case.
Has being too cautious got you in trouble? If so, be a little reckless and [10]click here to send us your story. We'd love the chance to feature it on a future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2abfi1JPvEEuJcfdxPgbP5wAAARc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44abfi1JPvEEuJcfdxPgbP5wAAARc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33abfi1JPvEEuJcfdxPgbP5wAAARc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44abfi1JPvEEuJcfdxPgbP5wAAARc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33abfi1JPvEEuJcfdxPgbP5wAAARc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2026/03/09/who_me/
[7] https://www.theregister.com/2026/03/02/who_me/
[8] https://www.theregister.com/2026/02/23/who_me/
[9] https://www.theregister.com/2026/02/16/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Only if it is not the other's fault. And it is always the other's fault. Therefore, it is always the other's fault. That is why DNS fails.
Shouldn't we call this episode "Who Other?"
It's always DNS...
Doubly so when split DNS is involved.
If the external DNS isn't a subset of the internal DNS this sorry state will always eventually arise. If you control both and use bind, you can "include" the resource records in both the zone files of both views.
When moving services like this I normally turned of the service eg httpd doesn't run, or run the service on another port.
If you aren't responsible for the content and that kitchen contains a surfeit of cooks, deleting content or even taking it offline can have unintended and unforeseen consequences.
Definitely not "at home" to Mr Rm -rf (notoriously the roving agent for Mr Cockup.)
Power down the server, you have backups if you need the data. You do have backups of everything, right?
Or if it's on shared infrastructure, set that specific site to not be one of the enabled websites (depending on what webserver code you're running), so that website is no longer available
Or if you do need to keep the old site available for comparison, and don't want to turn it on to check stuff - set it behind IP/cookie/whatever rules
You leave an old server running (which may not get security updates anymore), and don't even check that the traffic has stopped (checking logs is not hard)?
Yes. You messed up.
And if you do need to keep them both live
Set a background or banner on the old one to make it clear "this is not current" (and maybe add a link to the other one, DNS permitting!).
I use a similar trick when I have multiple VMs, firewalls, etc. for systems that are set up in the same way for different projects so that I know which one I am looking at when working on more than one at a time.
Re: And if you do need to keep them both live
Not even a banner. A simple "old." in front of the domain should be more than enough to indicate what the website is.
I feel incredibly sorry for Neil that it ended the way it did. It's all well and good keeping a back up of the code but it's quite another to spin it up to compare it to the new one.
Re: And if you do need to keep them both live
But Gerald's IT techies had hard coded the DNS, so I assume that www.gerald.com was pointing at the 'old' IP address, even though www.gerald.com should have resolved to the 'new' IP address. In which case renaming the 'old' site to old.gerald.com would have had no effect at all.
I bet Gerald was wild .... in fact he was livid! (points for people who remember that one!)
Re: And if you do need to keep them both live
So you make sure that anyone trying to access the old site can only get to the landing page, then you make the changeover and have someone from the internal IT team test access from both the internal network AND via whatever method they access the site from externally so that you know the new site is live. You don't wait for staff start rolling in and assume it will work. As for the old site, archive it if you have the capacity rather than delete it as that makes sense. Delete when absolutely sure it is no longer needed.
And hardcoding? Avoid if at all possible. I've never done it myself. Nope. Never. Honest.
Some leaps of logic in there, just to sound omniscient:
> You leave an old server running (which may not get security updates anymore)
The "old" material was the contents of the website, served from a certain IP; why would you think that the OS etc wasn't being kept as up to date as everything else, including the particular beast that just happens to contain the latest website data? Which may well have been the same physical machine, OS and software as was providing the new? As you said, shared infrastructure.
> set it behind IP/cookie/whatever rules ... don't want to turn it on
Well, you can start by changing its name, so Gerald can make the comparison by opening oldwww.gerald.com alongside www.gerald.com "Happy with the changes now, Gerald?". Now, if you put it behind more layers of "protection", at what point does Gerald start berating you for "making it hard to compare with the old site"? Such as, calling ahead to get it switched on. By the time of the story, Gerald *may* have stopped looking back at the old site, a nostalgic tear in his eye, but we have not geen given any information about that.
> and don't even check that the traffic has stopped (checking logs is not hard)?
Nobody was complaining, the machine wasn't complaining, both sites were being accessed and that activity was being logged; maybe Neil could have wondered about how much activity Gerald thought there *ought* to be, but on what basis? The machines inside Gerald's office would cache the old site, so not generating actual traffic to be logged, Gerald and *every* accessor outside that office would see the updated site and that was logged.
What exactly was Neil supposed to spot in his logs?
I think we've found one of Gerald's IT men!
Keeping an old site backup is fine.
Keeping it live, online, and accessible over the Internet is not.
That's just a compromise waiting to happen.
Hardcoding www.gerald.com to an IP address on the office network means that when they did the switchover, if he did what they asked and deleted the old site the office would not have been able to access it.
I suppose then someone would have complained, and they might have fixed it.
But that it was unnoticed for *two years*, is an indication that no one in the office really looked at the site.
Did no one in the office look at any updates after they went live, to ensure they look right?
It's always DNS...