123 Bork? Six-day DNS record-edit outage at domain name flinger 123 Reg enrages users
- Reference: 1605097212
- News link: https://www.theregister.co.uk/2020/11/11/123_reg/
- Source link:
The issue is a nasty one as the vanishing of the records prevents users from assigning domain names to IP addresses or making edits. This is quite unfortunate when one considers that registering domain names is what 123 Reg is all about.
Things began to go off the rails as long ago as last Thursday, 5 November, when the company [1]admitted that some customers "may experience issues with DNS management." Not to worry though, the hardworking 123 Reg team was on the case and a fix would be deployed ASAP.
Except, alas, it wasn't soon enough. The problem continued until the afternoon of 6 November, when there was some good news: the cause of the DNS record problem had been identified. However, rolling out the fix would need a bit of weekend working, according to the [2]company's status page , so the new ETA for resolution was set to 9 November. After all, nobody needs to tinker with DNS records over the weekend, right?
A Register reader got in touch to share his own experience of attempting a customer's migration to Microsoft 365 which, sensibly, was planned over Saturday and Sunday. His tale of woe included repeated attempts to coax a response out of the company and a lost weekend while he fruitlessly refreshed his control panel in search of the needed DNS records.
Another told us the inability to edit zones was holding up deployments for a number of clients, and even the 123 Reg support team appeared unable to make some of the more urgent changes manually. "Outbound domain transfers do seem to be working however," he told us.
It took until after 17:00 UTC on Monday for an update to be released by the 123 Reg team, but the fix only partially resolved things. Some users found themselves still unable to make updates to their DNS records as of Tuesday, days after problems were first reported.
The company told those still wondering just where in blue blazes the functionality they were paying for had gone that the next deployment was set for Tuesday and would resolve the issue once and for all.
Hello Chris and thank you for reaching you. Our team deployed a fix yesterday which resolved the problem for some users. The next deployment is set for today, which should fix it for everyone else. Please keep an eye on our status page for more info: [3]https://t.co/7BvXqz8TtH . — 123 Reg Help (@123regHelp) [4]November 10, 2020
At time of publication, however, [5]123 Reg's status page continued to admit the company was "aware" of problems, but affected customers hoping for an ETA after a nearly a week of borkery weren't given one.
The Register has contacted 123 Reg to find out what befell its systems and the company has now sent us a statement:
"Following a recent domain platform update, some customers were prevented from viewing or editing their DNS records. We have identified and corrected this for the majority of customers affected. We encourage any customers who are still experiencing errors to contact us at www.123-reg.co.uk/contact for assistance. We understand the importance to our customers of the management of DNS records and we are sorry for any inconvenience caused."
We'll leave the last word to one of the company's frustrated customers:
Well [6]@123reg DNS management has been down for 4 days now so they just lost a customer. Domain transferred to [7]@GoDaddy . [8]https://t.co/ShZmyryscR — Paul Beebe (@paulbeebe) [9]November 9, 2020
Paul remained sanguine when informed that 123 Reg was actually owned by GoDaddy. ®
Ah that's a bit annoying but at least Godaddy has working DNS! — Paul Beebe (@paulbeebe) [10]November 10, 2020
* Tottering Infrastructure Thwarts Service User Plans
Get our [11]Tech Resources
[1] https://twitter.com/123regHelp/status/1324650848690016257
[2] https://www.123-status.co.uk/
[3] https://t.co/7BvXqz8TtH
[4] https://twitter.com/123regHelp/status/1326070034557509632?ref_src=twsrc%5Etfw
[5] https://www.123-status.co.uk/
[6] https://twitter.com/123reg?ref_src=twsrc%5Etfw
[7] https://twitter.com/GoDaddy?ref_src=twsrc%5Etfw
[8] https://t.co/ShZmyryscR
[9] https://twitter.com/paulbeebe/status/1325905975950323718?ref_src=twsrc%5Etfw
[10] https://twitter.com/paulbeebe/status/1326069072929443840?ref_src=twsrc%5Etfw
[11] https://whitepapers.theregister.com/
Re: Cloudflare FTW
he.net also have a free (and easy) DNS offering which has served me well for the last few years.
How is 123reg still a thing? At least once a year they have some sort of major incident and yet keep going.
Been using Namesco myself for a while, haven't had any issues and they didn't try and pull a fast one with .uk domains that I never asked for appearing in my renewal unlike some registrars.
It's at this point...
I find myself nervously looking at fasthosts who have been * touch rigid, organic cellulose, lignin and hemicellulose based structure * pretty good for the 10+ years I've been using them in various capacities.
It does always feel like money for nothing though until they fall over like this.
Gimp icon because I'm probably going to suffer for my comments (in one form or another).
Re: pretty good for the 10+ years I've been using them
Hmm, in that case I suggest you absolutely refrain from doing a search for Fasthosts here on El Reg as you might find things are not quite as rosy as you assume:
[1]https://search.theregister.com/?q=fasthosts
[1] https://search.theregister.com/?q=fasthosts
Pay peanuts, get monkeys - 123Reg, GoDaddy, 1and1 (Ionos), Fasthosts - when things go wrong all these are shite.
Famous last words - never had a problem with Express Hosting
OMAAD*
I'm guessing "the hardworking 123 Reg team" is still recovering from that last piss-up before lockdown.
*OMAAD : the much missed One Man and A Dog (The Story of a Small Systems House) comic strip from the 80's
And then...
And then in a shockingly honest update to the previous press release...
"Following a recent domain platform update that we totally couldn't be bothered to test properly before making live (because we're cheapskates), some customers were prevented from viewing or editing their DNS blah blah blah"
Keep a log file.
Every time you have an issue with $Company record the date & time the issue happened, the exact nature of the issue, & any steps you had to do to (try &) correct it. Update the file to include any email/phone contact attempts you make to resolve the issue, whom you talked to, what they said, etalia. You want to maintain a facts-only paper trail that can be used later in court if need-be.
Once you've recorded ten issues in a short amount of time (the number & span is up to you) then contact $Company with a copy of the log file to that point & ask them what they'll do to keep you as a customer. "I've had ten issues in as many months. That is unacceptable. What will you do to keep me as a customer?" If they make you a deal to discount service for a few months then calculate what the total discount would be spread across all the incidents so far -- is your time worth (more/less) than that amount? If they refuse to make it right then feel free to add that to the log file as well (after the part where you've contacted them to discuss the matter) & then post a PII-stripped copy to social media as a name-&-shame of $Company.
You can then start researching other companies with which to do business & can offer them copies of the log file as a warning example of the crap you won't put up with from them either.
It's amazing what a paper trail can do when it comes time to nailing someone's arse to a wall or defending yourself against someone else trying to nail YOURS to one.
Go on, spill the beans!
Ah, go on, go on, go on. Spill the beans on which companies you've done this to, including all the juicy details! I'd love to see that list!
Cloudflare FTW
Cloudflare's free tier (https://www.cloudflare.com/en-gb/plans/) is probably at least as fully featured as 123's offering.
I've always thought that having Domain Reg, DNS and Hosting all in one place to be a bit "All eggs in the same basket".