News: 1587712507

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Elevating cost-cutting to a whole new level with million-dollar bar bills

(2020/04/24)


On Call While the days may seem to be blurring a little at the moment, we can assure you that it is Friday, which means it is time for another in The Register 's series of reader experiences at the hands of the [1]On Call telephone.

Readers might remember [2]Jeff from last week , and his A-Team-esque paperclip solution for an RS-232 diagnosis. Today's instalment sees him back in action, dealing with charges on a bill that would make even the most avaricious stuffer of the hotel mini-bar say, "Steady on, old chap."

The call came in complaining that hotel guests were seeing multiple million-dollar charges appearing on their bills. This was, of course, back in the 1980s when a million was a lot of money.

Heck, it is still a lot of money unless, of course, you're trying to [3]fit wheels to an Apple computer .

A paper clip, a spool of phone wire and a recalcitrant RS-232 line: Going MacGyver in the wonderful world of hotel IT [4]READ MORE

The customers, recalled Jeff, "ate in the restaurant and when the restaurant management system sent the charge, the charge that was received was wrong by millions of dollars."

It was then followed by the correct charge.

As with last week's telephone shenanigans, the restaurant system was on one end of the hotel while the computer was on the other, as well as being on an entirely different floor.

This time the paperclip trick wasn't going to cut it and so Jeff, along with the vendor of the restaurant system, flew out to the hotel to look into the problem.

"The interface," he recalled, "was a simple 'send and pray' format of 'data<cr>' just like a printer or teletype machine would use."

Those were indeed simpler days, before every IoT edge device needed to be a fully fledged server running Kubernetes in order to itemise a customer's bill and point the finger of guilt at who ordered the wine and which greedy so-and-so had three starters.

"We ran wire down the halls and through stairways, so we could tap the data connection and sit at one location and watch what was happening," he told us.

The pair watched and saw many requests get corrupted (hence the million) but the system self corrected. It was a mystery – a short in the wire somewhere? Something in the software? Computers weren't cheap in those days, and a hardware fault would be less than ideal.

As the duo scratched their heads, the elevator could be heard down the hallway. As it passed, a corruption event happened. Another elevator passed. Another corruption.

"It dawned on us that the RS-232 connection was run through the elevator shaft..."

The power used by the elevator was being picked up by the cable, which was acting as an antenna and causing corruption on the line. So if someone called an elevator while figures were being transmitted, the glitch would occur.

"The fix," he laughed, "was $600 [5]line drivers ."

Jeff had another story to tell, about the time he got a call reporting "a computer crash," which turned out to be the result of a car hitting a kerb and smashing through the wall of a hotel's computer room, but that will have to wait for another day.

Ever been called out to the scene of a computer foul-up that involved four wheels and some careless driving? Or seen interference on the line do the most unexpected things? You have? Send an email to [6]On Call and tell us all about it. ®

Sponsored: [7]Choosing A Low-Code Vendor



[1] https://www.theregister.co.uk/Tag/on-call

[2] https://www.theregister.co.uk/2020/04/17/on_call/

[3] https://www.theregister.co.uk/2020/04/16/apple_mac_pro_699_dollar_wheels/

[4] https://www.theregister.co.uk/2020/04/17/on_call/

[5] https://www.wut.de/e-80201-ww-daus-000.php

[6] mailto:oncall@theregister.co.uk

[7] https://go.theregister.co.uk/tl/1936/-8579/choosing-a-low-code-vendor?td=wptl1936

Elevator interface

Phil O'Sophical

Not a call-out situation, but years ago the guys at CERN used to see a strange wave pattern superimposed on test data from time to time. Then for a while it stopped. After some headscratching they realized that the stop coincided with a French train strike. Turns out that running a TGV drawing ~9MW of power from overhead wires creates a significant earth current, enough to deviate the beam in the nearby underground accelerator.

Re: Elevator interface

richardcox13

Reminds we of a story boardcast on (maybe RI Christmas Lecture when I was the right age... so a while ago!)

Some academic (so really a post-Grad or post-doc) had developed a really sensitive inclinometer in Aberdeen.And were confused by a not-quite 12 hour cycle in the readings when left sitting on a (very stable) surface.

Aberdeen.... built on lots of granite was moving with the tide.

Will this still work?

Caver_Dave

I had a call to go down to Silverstone one day from an F1 team who was using some of my lap timing kit; they wondered if the timing beam (a little metal box on a normal camera tripod) would still work after a "little prang".

I found the box and parts of the tripod embedded quite deeply into the side pod of the car and some of the data cable dangling on the floor.

The box was fine, once I had replaced the data connector, but the tripod and cable had to be replaced.

[icon : I designed and sold all sorts of electronics, computing and telecoms kit to most of the F1 teams, some NASCAR (via Nigel Mansell) and other high-end formula around the world, and did the first in car telemetry in the late 1980's and early 1990's]

The fix was $600 line drivers.

jake

And thus Gandalf worked a little magic to save the day yet again :-)

Re: The fix was $600 line drivers.

PM from Hell

I was desperately trying to remember the name of the line drivers. They were a bit pricey at the time but after a while we stopped trying to make connections work on the other side of the building without them as trying without then having to go back and ask for cash for the ,ine drivers always provided a fight with the bean counters. A beer for the memory jogger

It didn't affect us

Roger Kynaston

Though we were entertained once.

There was a furniture shop on the other side of the road to our office - back when you had computers under your office. Anyway, one day some of the local ne'er do wells had 'jacked' a car and were being pursued by the local plod. They crossed the main road at speed and parked it in the doors of the furniture shop.

I don't remember if plod picked them up straight away or whether they managed to reach the posh back streets.

My experiences with RS232 were rather mundane and I won't bore the commentariat with them.

Beer because Friday

Re: It didn't affect us

juice

Back in t'day, I lived in a victorian house which was in the heart of a large town - less than half a mile from the shopping high street, and on one of the main commuting routes into said town.

(Which handily meant that there was around half a dozen pubs in the immediate vicinity, of which at least one would have live music or a jam session going on a given night. Which was nice, especially when I ended up being paid to not work for a few months. But that's another story...)

And as this was an Victorian townhouse, the front garden was basically a small gravel pit, stuck behind a short and chunky brick wall surmounted by cast-iron railings that had somehow managed to survive the great WW2 melt-all-the-things purge.

Plus a bus-stop pole sat directly in front of said brick wall. Which was nice (for a very small value of nice), as the bus engine vibrations tended to hit the resonance frequency of the house's windows. Still, it played a part in what was to come...

In the early hours one morning, I was awoken by some strange screeching noises, followed by some rather crunchy noises.

Staggering out of bed, I went to the window, opened the curtains and blearily peered out.

And then I turned and yelled to my housemate "Oi! We've got a car parked in our front garden!"

Some muppet in a sporty little number (A Toyota MR2 or somesuch) had come barrelling down the slight incline of the road, lost control and put his car into a spin, which only ended when they hit the aforementioned chunky wall.

It was actually quite a neat - if accidental - bit of maneouvering - the townhouses were the standard narrow "single room" design, but he somehow managed to completely avoid the houses to either side of mine, and plant his car squarely in our gravel pit. Atop the remnants of said wall and railings, facing backwards, and with the rear bumper of his car barely a foot from the front room's bay window.

This is where the aforementioned bus stop came in handy, as when the car hit it's steel pole, it helped to spin it into the brick wall!

Still, I can think of nicer ways to wake up...

Shafted!

Anonymous IV

Nothing like something groan-inducing on an otherwise boring Friday?

Never saw a car crash into a computer

Pascal Monett

But one day I did see a Fenwick crash an entire rack of servers.

I was parking at a customer site when I saw a Fenwick barreling down the loading dock with a rack of servers on its enormous tongs. I remember thinking "this guy is going a bit fast with that", then I saw two guys running behind the Fenwick, shouting. I stopped the car and got out when I heard an enormous crash. When I turned around, the Fenwick was stopped, the rack was lying on the ground, and there were three guys arguing.

Turns out the two guys who were shouting made the driver think there was someone in front he couldn't see, so he slammed the brakes, which had the inevitable result.

The rack was fine, if a bit scratched here and there, and it turns out that only a few hard disks had bought the farm. The kit was new, so I guess insurance paid for the disk replacements. They had one hell of a time getting the rack back up again, though. I don't know what happened after that.

I'll never forget the almost cartoon-like quality of those two guys running behind that Fenwick.

Re: Never saw a car crash into a computer

jake

For the Yanks in the audience, a "Fenwick" in this context is an approximation of a Yale Forklift (as built by the French), not a fishing rod.

This cross-pond translation service brought to you by the number e and the letter O.

Re: Never saw a car crash into a computer

Korev

This Brit didn't know what a Fenwick was either! Thanks for the explanation

Me -->

Re: Never saw a car crash into a computer

tip pc

"For the Yanks in the audience, a "Fenwick" in this context is an approximation of a Yale Forklift (as built by the French), not a fishing rod."

Explanation not only needed for the Yanks, the only Fenwick i had heard of before was the department store.

https://en.wikipedia.org/wiki/Fenwick_(department_store)

I've never heard anyone call a forklift a Fenwick.

i guess its like us calling a digger a JCB or a vacuum cleaner a Hoover, but Fenwick is hardly a household name in the UK.

Another elevator anecdote.

jake

Picture a data center in the basement of a tall building in San Francisco's financial district. Card punch up against a wall, near the ancient Otis heavy goods lift. Every now and again, at seemingly random times, the punch generated errors for a couple characters. Nobody could figure out why, not even IBM's field circus dudes.

Until IBM was traipsing in and out one fine weekend, upgrading who knows what hardware, as only IBM could. Someone (ahem) noticed that the gibberish was being generated about ten seconds before the elevator doors opened.

Turned out that the motor for the lift was drawing so much current when it first started that it was inducing errors in the punch on the other side of the wall. Nobody put two and two together prior to this because the lift rarely went into the basement (that level was key-protected) ... until IBM was in and out that morning.

Interference ...

Blofeld's Cat

Many years ago I got called out to a mysterious issue that was bringing down a workshop's servers every Monday morning at about 08:30.

Systems were checked; cables replaced; even one of the power supplies changed, all to no effect. The environment was checked and it was established that nothing out of the ordinary was happening at that time.

Eventually it was decided that I had to go and spend a Monday morning on site to try and catch it red handed. For the first hour everything worked normally and then silence. After a few moments the servers came back to life and started rebooting.

It turned out that the workshop used a lot of air tools and had a huge compressor tucked away in a shed outside. Usually this cut in and out throughout the day with no difficulty to keep the air reservoirs topped up.

Over the weekend the ancient grease in the compressor's bearings essentially set, stalling its electric motor until it overcame this resistance, and stirred the grease back to a normal thickness - which it maintained until the next weekend.

During this period the supply voltage on that one, somewhat flaky, circuit dropped to about 30V, and needless to say, the servers were on the same electrical circuit.

My boss quoted for a complex solution involving UPS and the like, but the workshop simply oiled the compressor instead ...

Climate control could only be set in the morning

Jens Goerke

before the big printing press was turned on, as it disrupted the RS232 signals between the control panel and the actual climate control unit.

Replacing the cable with a length of fibre and a converter on each end was the most effective solution, though a bit pricey, but also reduced the amount of environment-related problems of the printing press.

Re: Climate control could only be set in the morning

Martin an gof

What is all this talk of "line drivers" and fibre converters, when RS485 was designed for this task, has been around just about as long as RS232 and can actually [1]be quite cheap ? Neater solutions are available, but this would work as a quick fix, even on standard serial cable; proper RS485 cable is different, but if a system "nearly works" on RS232, converting to RS485 would probably do the job just fine.

M.

[1] https://www.amazon.co.uk/Port-Powered-Industrial-Communication-Anti-surge-Protection-RS232-RS485/dp/B01LVVI1XI/

Sounds familiar

Doctor Syntax

First server in the building (Z8000/Unix v7). TPTB decided that it could be located in a cleaner's cupboard small room next to the lift shaft. Experience gained led to its being rapidly located to the library.

The other computer and lift shaft episode was much, much earlier - the computer was probably an Elliot - from the company where my Dad worked. It had to be located on an upper floor of the building so it was decided to haul it up the lift shaft. The rope failed. According to Dad the man who'd tied it on* fainted which might have run in the family; his daughter was in my class at school and fell off a biology lab stool when "blood" was mentioned.

* For added irony his surname was Crane.

Been here before!

Will Godfrey

Our people fitted a new central process control unit in a plastic film factory. This had just about every test imaginable before being fitted including being hit by brownouts and static discharges. However, once installed at the location demanded by their engineers all sorts of random errors were occurring. This was in the middle of the factory on a partition attached to the back wall and at the top of a flight of steps. It was months before we identified the problem, and only then because one of our engineers took a laptop there to make notes of readings and it kept crashing. One of their production workers commiserated and said that their mobile phones would sometimes freeze if they went there. In a moment of inspiration our guy asked what the partition was there for, and was told the 3 phase supply for the entire factory complex was behind it.

Something Smillar

OGShakes

I did massive amount of Cable and switch replacements in a small school to try and fix an intermittent drop in the network. It happened mostly at lunch times but at random through out the day, it was only luck I was in the staff room one day when it happened. Some one used the ancient microwave in the kitchen area, seriously it must have come off the arc, and that was when it all dropped. As I said this was a small school and the server room was off the staff room, most of the cables were trunked behind the microwave in question. Thankfully due to its closeness to the server room, I was able to pull though enough slack to move the trunking up about a foot and fix the issue. We also added tinfoil to the inside of the trunking and added a wire to earth it as a little extra protection. To be honest it would have been better to replace the faulty microwave, as it must have been slowly cooking their staff too, but they were happy we had fixed the issue...

Lazlo Woodbine

Had a similar head scratch situation whereby a customer would be in the process of buying something when the stock would suddenly disappear from the inventory mid transaction.

This was back in the early nineties, we were a catalogue store not unlike Argos, the customer would select an item, the sales person would enter it into the till, process the order and the item would be picked from the stock room, but occasionally the item would disappear from the inventory between the software finding the item in stock and then processing the sale.

We had the IT people in several times, they couldn't locate the glitch, and the disappearing item would always reappear the next day after the End of Day routines were run.

Then, one day the software guys were on site with all kinds of gear when they at last spotted the stock disappear from inventory, and they saw just why it happened, it was when the store manager walked past the server room when on a call on her analogue cordless phone.

They swapped her phone for one of the new fangled DECT phones and the problem never reoccured...

...in order to itemise a customer's bill...

Kane

... and point the finger of guilt at who ordered the wine and which greedy so-and-so had three starters.

[1]Bistromathics at work.

[1] https://hitchhikers.fandom.com/wiki/Bistromathics

Anonymous Coward

Not quite computer equipment but when I spent a few years in silicon valley at end of the 90s I used to look at the USGS web pages showing all recorded eartquakes and noticed that there seemed to be small tremors on a regular basis near where we lived in Cupertino on the map. After a bit I went to the more detailed pages to find the records of this "quakes" and discovered that happened on almost every weekday at 12 noon with the USGS comment saying "probably quarry blast" ... and there was a quarry just outside Cupertino. N.b. these "quakes" were 2.0 magnitude and one of these on a daily basis from a quarry seemed to cause no qualms which, for me, has always put the alarmism over occasional 1.0 tremors (10x les powerfull) being sufficient to close down fracking into perspective!

radio interferences

Anonymous Coward

Years ago, I was checking a cabling infra in Paris, only to hear a perfect radio signal in the controller.

Turned out the whole cabling was acting as a gigantic radio antenna !

Matter will be damaged in direct proportion to its value.