News: 1597389314

that can't be solved by a sizable application of high explosives.

"Work or I'll pack you full of c4 and turn you into atomic dust... Oh look, it's working now. Good. Pub time!"

I prefer the slightly more tactile approach.

That is to say, approaching a recalcitrant machine with a well selected hammer and giving it a a touch of percussive maintenance.

It's important to have a tool kit that will permit incremental levels of maintenance with the largest hammer fully visible to the offending machine.

We used to have a 20 pdr sledge hammer in the server room just for such a reason.

So that's where my sledgehammer disappeared to. And all this time I thought it was one of the removal men that had nicked it.

I've always followed the following simple principal: "If it does not work, hit it with a hammer. If it still does not work, fetch a bigger hammer. Repeat the second step until it works or is scrap".

Sorts out problems almost all of the time, one way or the other.

... Or user...

That's what my walking stick is for.

Hummm, Are you sure C4 is opaque enough to block the light detector?

A primary tool in problem solving! Well done to the PFY

Cold beverage earned.

I remember a customer who had a problem with the software I supported going on for months. Every morning without fail, at Market Open, a link would stop and restart. We said it looked like "application code". We could see nothing wrong with our code. It got political and I was sent on site. I turned up Monday morning, and we all stood around in the ops centre waiting for it to fail - and it worked without any problems. There were comments like "the code knows you are here".

I spent the day doing a health check, and said I would come back the next day.

I went back to the hotel and saw one of my colleagues, and so had a beer and catch up. He had spent the weekend assisting in a major ugprade to another product at the same site, putting on a year's worth of fixes and restarting all 100 transaction servers.

Next morning in the ops centre it worked perfectly. Then the light dawned on me. The problem was with the "transaction server" which they had fixed with the big upgrade. The problem they were having was the application would have a problem, close the connection to my product and restart. My product reported the break in the connection and so got blamed. It was a case of "shoot the messenger".

The customer took his team and us out to an expensive dinner to say thank you.

Once had an interactive screen with sensors around the bezel, and during the fall months, the sun would come in the window just right, and "blind" the sensor and all kinds of interesting things happened to the interactivity. Small cardboard triangle taped over the corner of the bezel was the quick and dirty fix.

Used to be a problem when repairing VCRs (remember them?) as well. The tape beginning/end sensors were triggered by some clear leader tape at each end of the magnetic stuff. I've heard of bench technicians opening up a unit to repair a minor fault, and having it stubbornly refuse to operate at all, eventually traced to sunlight on a bright corner of the bench falling on the tape sensor once the cover was off.

the cap from a black bic pen used to be the standard "make this machine think it has a tape" tool

I had a primary school where mice kept acting up, but the teachers couldn't work out why it worked for some people (them) and not others (pupils).

I traced it down to the mice casing allowing IR to pass though it when the sun was in the right (wrong?) place. The kids hands were too small to cover the entire mouse body, but the teacher's did.

Coating the inside of the shell with paint fixed the issue till the mice were eventually replaced with the new fangled optical mice.

bring a hammer on any IT repair job.... handy for 'repairing' the pranksters.......

Back in the day I worked on a lot of T-carrier stuff. I can't tell you how many times an owner/client ranted about a shiny new (fractional) T1/E1 link being down, how the equipment was shit, the field guys were incompetent, and how pretty much everybody involved with the installation should be taken out behind the barn & horsewhipped. Most of the time, it was an incorrectly set loopback switch on the new node. Seems bosses in general can't resist flipping switches ... and can't read blinkenlights.

Sometimes I'd casually reached out and toggle the loopback switch, thus fixing the link and painting the boss's face an interesting shade of red when I presented him with the bill reading nothing more than[0] "Call out. Flipped loopback switch. $1,000" on an official invoice.

But once in a while, after inspecting the node, I'd stand aside & motion the boss through the door before me. While he had his back to me, I'd flip the switch ... and we'd go off to his office for a chat about fixing the obviously broken machine. I'd let him rant on for several minutes, around 20 was the record, but always ending up with "so what are you going to do about it, then?". To which I would quietly reply "Oh, I've already fixed it. We'll invoice you for the call out". Sometimes the resulting sputtering reached epic proportions ...

[0] The rest of the time it was a cable that had fallen out of the CSU/DSU because it hadn't been screwed down properly. We always took the blame for that, even if it was their guys bolting stuff together. We've all done it, we're only human, I'll take the blame, no charge ... sometimes it's handy to have a friendly couple of faces in a client's datacenter who probably won't ever try to throw you under a bus.

Our story takes place as the 1980s rolled into the 1990s

I had VAXstations & X-terminals on my desk in the 80's.

Nowadays just being in 'the office' is a dim and distant dream...

From the time.... The time before....

*stares wistfully out of my study window onto the suburban street below and remembers the time before the apocalypse*

I was running Ventura Publisher under Windows 2/386 GUI in 1988 (PS/2 Model 80, with a whole 6MB of RAM)

I had a costumer in the late 90's that had more or less the same problem with a mouse. If the sunlight went directly on the mouse, it would start to freak out. Took a good while to figure that out. Fortunately, that was their previous IT partner.

I had a costumer in the late 90's that ...

What... suppliers of theatrical or fancy-dress costumes and the like?

http://www.catb.org/~esr/jargon/html/magic-story.html

Though in that case they never got a completely satisfactory root cause for why flipping the switch crashed the machine and could only speculate as to why a switch that all logic and reason says shouldn't be able to do anything would crash the machine whenever it was flipped.

Reminds me of an AI koan ...

A novice was trying to fix a broken Lisp machine by turning the power off and on.

Knight, seeing what the student was doing, spoke sternly: “You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.”

Knight turned the machine off and on.

The machine worked.

Our office was nicely redone and equipped with fancy sensors for the lights except in winter the low sun in the afternoon comes through the window hits the sensor and turns the lights off in the open office.

I got talking to one of the phone guys about weird faults and he recounted the tale of the company that kept reporting that random phone lines were going down in the afternoon but by the time he'd been called out and arrived the next morning all was fine. He tried swapping everything he could think of but the faults were random, so things might be fine for a while and he though he had fixed it, but he would receive a call a few days later to say the problem was back.

In the end he decided to camp out so he would get to see the problem as it started to occur. After a week or so he figured it out. It only happened late on sunny afternoons, just as the sun blazed on to some industrial-type shelving that had been attached to the back wall of the switch room and OVER THE TOP of the phone cabling... the large metal sheets expanded in the heat and were crushing the cables

when i was a child, (early 90s) my dad used to sell computers, and give support to his customers.

One day a tower came in that had "random freezes". We're talking a full tower case (about 1m tall) fully loaded with a P1 166MHz.

My dad startet running the usual series of troubleshooting tests, and the bloody machine seemed to work flawlessly.

Being the before-pf-youngster that i was, i crawled under the table and peeked inside the beast - we had removed the side panel.

EUREKA: the tiny cpu fan was not spinning!

So when the customer hit the machine with a serious load, the cpu would just thermal throttle and halt everything, and then when we tried to troubleshoot, we didn't do much more than open a couple of programs and look at settings.

Now, 25 years after that, i work at the IT helpdesk of a medium-large company........ Coincidence?

My son saw the spoiled family meals, lost weekends and the midnight call outs. He works in the film industry as an electrician. Coincidence?

Ah, the days before a machine needed an active fan even for idling.

Intel were worried about cooling a CPU that drew 15W at full chat. These days a desktop PC will draw anything between about 65W and 135W and will pretty much fall over instantly if the heat sink is dislodged (stopped fans will take them a little longer).

I like the alternative version...

Any technology that is distinguishable from magic is insufficiently advanced

Mine's the one with all the straps!

maybe 10 years'ish ago I was working for a lab, our WAN connection was via a radio link to a building at a university, as the crow flies maybe 1km away. We suffered a very unreliable connection during a summer period, the link could go down for hours. Never got to the bottom of what was going on but the uni was building a new large tall copper clad building at the time that was in the Los from our site to the building where the B end was, was its omething to do with the hot sun and the copper cladding? or maybe tower cranes jobs etc getting in the way? All very odd and annoying at the time, both A and B end dishes were swapped out etc

That reminds me of the story of a guy who had a problem with his car. He apparently reported the problem that when he went to his local corner shop to buy ice cream, if he bought vanilla his car wouldn't start when he went back to it, but if he bought chocolate it started fine. It turned out that vanilla was at the front of the shop by the till, and other flavours were at the back, and the car would flood something if it was started within a short time from switching off, where it would be fine if given a minute or two longer to drain.

That reminds me of a Kia car that I ended up with for a while that would refuse to start (seemingly at random).

weather made no difference, time of day... no difference.

if it started it would be perfect, if it didn't start you could spend the rest of the day trying and failing to get it started, only to have it fire up perfectly the next morning.

Eventually it was worked out that if you even touched the throttle before turning the key it would flood and any attempt after that would only make it worse. Leaving it alone for around 3 hours let the excess fuel evaporate off and then it would start.

(Posting anonymously because of implications)

A friend of a friend worked at the local [Car_Maker_Which_Is_THGTG's_Prefect's_Fist_Name] manufacturing plant as a technician. One day, a customer comes in with an issue on his almost new Ec* Sp*rt which any of the service centers he visited was able to diagnose/solve. Whenever he passed 120 km/h, the stereo would up the volume by itself until it was maxed out. Nobody at the service centers believed the customer. Then the customer begged this friend of a friend to take a ride with him down the highway, and upon hitting 120 km/h voilà! the stereo went all out.

The friend-of-a-friend technician asked the customer to leave the car at the plant and promised to call back once they had a solution. After a month of diagnosing, conference calls with the Mothership at the US, and general candle lighting and praying with no results, the friend-of-a-friend tech decided to seach at the lowest level. This led to the customer getting really agry about not getting a call back and visiting the plant. Then he entered the storage deposit where security escorted him to, only to find his car completely torn down to every single piece. All was stripped from the body: transmission, engine, suspension, brakes, control arms, upholstery, even the cables.

Naturally the customer had a nervous breakdown. The company ended up reimbursing him the full price, and then they grabbed all the pieces of the car, put it in a container and shipped it back to the Mothership for further investigation.

Source: I got the story from the friend-of-a-friend technician's mouth maybe 8 years ago, when exchanging stories about tech pieces and spooky action at distance over a few beers.

Had a co-worker tell me the tale of a rogue temperature sensor. Said sensor controlled the temperature for an office space. Early in the morning, everything would be fine. Midmorning, employees would start calling complaining that the room was cold. By mid-afternoon, it would return to normal. Sensor reported the temperature was right on setpoint the whole time. A tech dispatched to the office (in the afternoon) verified that the sensor was reading accurately. This went on for quite some time, until my co-worker got fed up with it and went to investigate.

He sat and watched the area one morning. Nobody got anywhere near the sensor (thoughtfully mounted in front of a window), but when the sun got high enough to clear the trees, it shone directly on the sensor, heating it up, which turned on the AC so that the sensor reading stayed on setpoint. Once the sun was high enough, the building itself shaded the sensor, so it went back to normal.

One small piece of cardboard taped to the window later, and all was well.

Back in the early '80s I worked at a Polytechnic, and was responsible for spending a chunk of the contingency fund at the end of one financial year, with the aim of producing a teaching room for what was called "Computer Appreciation". The purpose was to have lots of different types of hardware available so people who did not know anything about computers could see what they could do. The systems were BBC micros, of course.

One of the devices we found was a robot arm with 6 axis's of movement that was very functional for the price. To keep the costs down, they used normal rotary motors, with shaft encoders made up of IR transceiver devices, with a 4 quadrant rotating reflector on each shaft to 'bounce' the IR back from the transmitter to the receiver. It was all very elegant, and worked really well.

That is, until I tried using one in direct sunlight one day. Unfortunately the design did not have light covers on the encoders. As soon as I issued the return to home command to the controller to calibrated the position. It started all the motors (something quite interesting, because all the motor channels could run simultaneously, unlike most of the other non-industrial teaching robots), and proceeded to pull itself off the desk as all of the moving parts moved to the physical end-stops, causing the arm to contort in a way it was never really designed for.

We worked out that the bright sunlight swamped the IR detectors, so even though the motors were running, it could not detect the movement of the rotary shafts. I suspect that there was also a bug in the controller software that needed to see the shaft rotate before it would look for them to stop to detect the end stops, but I never knew that for certain.

The fall damaged some parts, so we went back to the manufacturer, and asked whether they could provide spares. We explained how the problem happened, and they said that they would see about designing some light tight covers, but although we got the parts to repair the robot (they came in kit form anyway, so replacing some of the parts was not a problem), we never saw any covers from them.

Shame really. They were the best educational robot arms I ever saw, so long as you used them out of direct sunlight. It's a shame that some of the HND students never saw fit to do their project with the kit available in the lab, as we had hoped. I often wonder what happened to all of the 'toys' when the lab. was replaced ( I left before the lab was dismantled), but I guess that some of the lecturers with their own BBC micros gave them a home.

Way back when, the project I was on had a support team over in Wales. Nice bunch of people, too.

There was one server which occasionally needed to be stopped and restarted. And this was all documented and (IIRC) fairly simple.

But for some reason, this process never worked for the Welsh lads. They'd try it, it'd fail, and then they'd call me out. And when I did it, it worked perfectly.

This happened a few times, until eventually they discovered that the restart would work perfectly if they were talking to me on the phone when they pressed the button...

I'm sure there's some boringly scientific reason for this, but I just took it as evidence of my Alpha-BOFH status. And charged accordingly for the call outs :D

I remember a chemical balance that changed its reading as you leaned over it. It was one of these https://oertling.com/balances/top-pan/tp-series.php used to weigh out small amounts of expensive reagents. The reading would change as you leaned over it which made using it a tad tricky.

It was on a bench on the first floor of a building put together by some 'orrible '60s steel-framed pre-fab technique. The concrete floor slabs were supported on a triangulated mesh of steel tubes sufficiently flexible to move very slightly in response to the movements of whoever was standing on them.

We didn't have any problems with the replacement. It was replaced after the building was destroyed in a fire and the replacement building was a nice solid reinforced concrete job.