News: 1736148910

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Brackets go there? Oops. That’s not where I used them and now things are broken

(2025/01/06)


Who, Me? Have you remembered it’s 2025 yet? The Register asks the question because it’s often the small things that make for the kind of big problems that we share each week in “Who, Me?”, the column that celebrates readers’ escapes from their errors.

This week, meet a reader we’ll Regomise as “Henry” who told us of his early career adventures working for a managed services provider that provided a spam filtering service for its clients.

“Customers would point their MX records to us, and the product we used filtered for nasties and then sent legit emails to right inboxes,” Heny explained.

[1]

There was just one problem: the product that Henry’s company used to run the service had a terrible user interface that relied on some decidedly odd syntax that he wasn’t entirely across.

[2]

[3]

“Creating new rules was painful and required one to pay specific attention to the logic used, namely AND/OR operators,” Henry told Who, Me?

When Henry was asked to implement a new filter rule, he therefore resolved to concentrate fiercely.

[4]

“The rule was supposed to block messages from one email address at a single domain from ever reaching the client,” Henry told Who, Me?

The requirement was therefore to filter any message from the hypothetical address makeitstop@bad.com and make sure it never reached an inbox at henrysclient.com.

[5]Coder wrote a bug so bad security guards wanted a word when he arrived at work

[6]Panic at the Cisco tech, thanks to ancient IOS syntax helper that outsmarted itself

[7]NetAdmin learns that wooden chocks, unlike swipe cards, open doors when networks can't

[8]Network engineer chose humiliation over a night on the datacenter floor

Which seemed simple enough.

But that finnicky UI and syntax meant the rule Henry wrote looked for messages sent from makeitstop@bad.com OR sent to any inbox at henrysclient.com.

“Yes, I effectively told the spam filter to drop any email sent to my client,” Henry confessed, describing this as a “major blunder”.

[9]

That assessment may have been a little harsh, as it was only the next day the client reported zero incoming emails and thought to ask if there was a problem.

Henry was reprimanded for his mistake, then given the job of reviewing the spam filter’s logs to compile a list of everyone who emailed the client during the time his rotten rule was in force.

But he also felt a little vindicated because his employer changed its processes after the incident, so that any new rules in the spam filter could only be implemented after signoff by two people – one of them a senior tech.

Have you caused a fault by flubbing syntax? IF you [10]click here to send Who, Me? your story THEN we can consider it for a future column. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z3u31XKFsntpXb-3spyPgAAAAMk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z3u31XKFsntpXb-3spyPgAAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z3u31XKFsntpXb-3spyPgAAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z3u31XKFsntpXb-3spyPgAAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2024/12/16/who_me/

[6] https://www.theregister.com/2024/12/09/who_me/

[7] https://www.theregister.com/2024/12/02/who_me/

[8] https://www.theregister.com/2024/11/25/who_me/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z3u31XKFsntpXb-3spyPgAAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Any system...

Anonymous Coward

...that require a signoff by two people (one "senior"... like they never make mistakes, right?) for a trivial change and doesn't have a 'TEST' env or a 'test' options is a badly designed system in need of overhaul.

(aks me how I know)...

Re: Any system...

graemep

My thought exactly, but they also should have fixed or replaced a system that has error prone syntax.

Re: Any system...

Gene Cash

> error prone syntax

You mean like sendmail or perl?

Re: Any system...

MJB7

EVERY system has a TEST environment. Some people are lucky enough to have the luxury of a separate production environment too.

Re: Any system...

SVD_NL

What do you mean? Any senior worth their salt would undoubtedly devote their full attention to every single email filter rule they need to approve, and diligently check the syntax of said rules to account for missing or misplaced brackets... /S

You'd literally be better off letting your seniors write a small syntax debug program.

Re: Any system...

that one in the corner

You just need to gamify[1] the process:

"I've made a change somewhere on this line I copied from a rule written by you three years ago; when the lovely Samantha sounds her horn you have one minute to write a matching input for a chance to win one of these fabulous prizes! And don't forget the Booby Prize that Sven is showing the audience now - and for the listeners at home, the Mystery Voice ('Documenting the regex language; documenting the regex language')"

Parp

[1] apparently, *not* pronounced like a side of ham

Not quite a misplaced bracket but definitely misplaced testing...

42656e4d203239

Once upon a time a senior analyst wrote a script to delete the mailbox records from AD for everyone who had an email account on the old exchange server...

You can see where this is going can't you?

I had only just (3 months) joined the organisation, so watched, with a kind of crazy grin, as other user accounts were getting deleted... along with their access to Office365.... My account, being new, was not affected.

That was a few years ago now - analyst has moved on, but the long term effect is that users no longer trust cloud services (which is a win I gess)

Re: Not quite a misplaced bracket but definitely misplaced testing...

K555

The UI is/was a also bit like that in Exchange, so you could do similar even without an errant script!

Because you were in the Exchange Management Console it was very easy to assume the 'delete' button was just for exchange stuff. It deleted the user's AD account too. I'm sure everyone managed to do that once, but at least it's good practice on recovering objects once the user phones up to tell you they can't log in any more.

Michael H.F. Wilkinson

Not sure it was a syntax problem, but our mail system developed a tendency to block everything from the IEEE Computer Society (of which I am a member, so it shouldn't count as spam), and, worse still, anything from NWO, which is the main Dutch funding body. Not only was I in one of their assessment committees, I was also preparing a very large funding proposal (to the tune of 1.5 million euro), which I did not want to go AWOL due to some email cock-up. It got so bad that NWO would call me to see if the emails they sent me had arrived. Fortunately, I could put two rules in place, one preventing the spam filter from touching the IEEE emails, and one for NWO. Of course I gave these rules the highest priority (5), higher than the regular filter (3), and all was well.

For a time.

I then noticed things went missing. I inspected the filter settings, and somehow the priorities had been reset to the default (3). I set the priorities back to 5. This happened twice. I then sent an angry email to the system administration that if this happened again, and I missed a deadline for responding to reviews due to this, causing me to miss out on the funding, I would send them bill to the tune of 1.5 million euro.

That seemed to have sorted the issue.

Crashed an IBM 360 mainframe

Andy Non

I once accidentally missed a full stop at the end of a line of COBOL and the compiler didn't handle it gracefully, crashing the computer.

Usually such syntax errors would be picked up but there must have been something about that particular line and what followed that wasn't catered for.

Re: Crashed an IBM 360 mainframe

David Harper 1

You, sir, are a Real Programmer.

Re: Crashed an IBM 360 mainframe

UCAP

Ahh, yes - COBOL. An abomination unto Nuggan, and must therefore be shunned.

Many decades ago, when I was a research student at university, I made the serious mistake of admitting that I had done some COBOL programming in the Real World. This unfortunately meant that my Comp Sci department had precisely two people in it who new COBOL - a lecturer who was providing courses in the language to under- and postgrad students, and myself. Guess who got to take most of the tutorials!

The biggest problem I had during this time was trying to teach the students how to work out which error message was actually the meaningful one, and how to identify and discard the many errors generated because the compiler had developed a headache and wanted to lie down. I think the record was 1 meaningful message embedded in about 200 spurious ones - again caused by a missing full stop at a critical point.

Re: Crashed an IBM 360 mainframe

MiguelC

I remember once in my first work in a bank's IT department a room filled with 90+ developers, typing silently (as silently as those wonderful keyboards allowed us to be) when suddenly we hear a really loud and prolonged "Fuuuuck!" Everyone turns to the new guy, who's turning bright red by then. His compilation had just produced over 4000 errors and he thought he'd have to rewrite his program from scratch... I think he was as embarrassed about his loud expletive as for the public explanation he got from our manager about the error count.

From then on we would hear, day in day off, someone muttering the prolonged fuuuuck whenever something was awkward, had really gone awry, or just because.

Re: Crashed an IBM 360 mainframe

Flocke Kroes

C permits leaving parameter names in function prototypes. Only good taste prevents you from putting different names in the definition. This used to crash some compilers but not others, but only when compiling the file with the definition if that file included the prototype. Until you discovered the cause you could easily discount the possibility that the problem was in the header file because it did not cause problems compiling other files.

Re: Crashed an IBM 360 mainframe

Anonymous Coward

Brings back a memory from the early 1980's where I was the lead engineer on a project to design and implement a material traceability system across our entire site five large workshops with machining, fabrication, assembly and test facilities for a wide range of oilfield equipment. I was the interface between the IT team and the rest of the organisation, looking at the real world issues whilst the IT team got something working on the mainframe (which, incidentally, was backed up overnight, by a satellite link, I believe, to the USA).

The workshop interface with the system was through punch card readers and numeric keypads, which was quite limiting in what would be entered. It was complicated by having to be compatible with parts and equipment from our parent organisation based in the USA (and who had no interest in material traceability - we were transitioning from a "it's OK if it looks right and doesn't fall apart" mentality to one of ensuring it actually is right. The system had to cope with a variable traceability standard ranging from not traceable, through traceability to a batch, to one where every component had to be traceable back to raw material. I set out four trace codes"

N - Not traceable (self-explanatory, no action - for non-critical parts and material).

U - Unique traceability required (for components and small sub-assemblies needing traceably back to raw material certificates, or an approved equivalent from the manufacturer).

T - Traceable (the default for most things, traceable back to our supplier or production batch).

S - Serialised (for major equipment, whereby I could enter a serial number into the system and receive a full breakdown printout).

Naturally, the system was totally NUTS - something I used to gain buy-in from the workforce.

But, to the point (sic). The computer ran its main processing as an overnight batch and I received a morning printout of any errors (such as an incorrect trace number entered) - usually quite short and all cleared in an hour. One morning, however, everything had been rejected - and it was clear there was a glitch in the program. The code (COBOL) was printed out and I dived into it with the programmers. I didn't know COBOL that well (I'd learned FORTRAN IV at university, and then BASIC in my previous job), but I was able to follow the logic. Come lunchtime, we'd discovered a missing full stop at the end of one line. The code was updated and the batch rerun (not popular with other users but, unless we reran the overnight batch, nothing new could be entered). Success and underwear changed!

Opportunities for error

IanRS

About 20 years ago I briefly worked on the Galileo GPS system. Because this was an EU funded system, politics came into play, and all suppliers had to be from the EU. I had to configure some of the firewalls. EU supplier? Firewall? Yes, there was one, from Austria, that I had never heard of before and never used since. Their rules used what looked like CIDR notation, typically with /8 and /24 networks for what we needed, but they had their own special syntax, where the number after the slash gave the number of variable bits in the range, not the fixed bits, possibly copying the idea from the Cisco wildcard masks. Hence a /24 was a large network and a /8 was a small one.

Not Syntax, but

ArrZarr

We used to develop our own in-house tech where I work, and that tech was thrown out the door by competent developers who were never given anywhere near enough time to make something even approximate user-friendly.

Sadly I was the only person anybody trusted to do anything scary in the UI. For about 18 months, this was all well and good but one fateful day, the toggle switch to the Include list was left on so I reduced the size of an important feed file from ~70k rows down to 10. Not 10k, 10.

This has left me with strong opinions (TM) about any tool's ability to check for weird changes in core systems. Needless to say, my feedback on how to prevent this happening again was never implemented until the tech was shut down a few years later.

Korev

This could have been worse, Henry could now be an Exim-ployee

Anonymous Coward

I've emailed you your P45 - phone us when you get it.

Mind your language

MrBanana

I can see the problem with config files that have a strange syntax structure, but usually I find Emacs has an edit mode that can help somewhat with keeping on the correct path. But I struggle to see how some languages can get cross mangled, like one recent example I came across. An install script written for bash was failing due to a syntax error. Fortunately the install package had the script tacked on to the front of the binary, so I could see that some dimwit C++ programmer had commented out part of the script with '//' type syntax - Doh!. Unfortunately the install process was protected by a checksum so that it wasn't possible to just change the script. No idea how it got past any testing or QA process.

Re: Mind your language

Richard 12

As CrowdStrike have demonstrated, a lot of suppliers don't even attempt to install their released product.

I got bored with SEO/APP spam...

Camilla Smythe

... So I blocked outlook.

The mosquito is the state bird of New Jersey.
-- Andy Warhol