If the pins weren't contacting the motherboard, what got soldered to what?

I'm just trying to imagine the mess.

I am too, it's making me laugh and cry at the same time (with a dash of anger).

Good question! I'm wondering the same. Although, I did come across the odd SMD that were mounted and partially(!) soldered upside down - directly out of the soldering oven. And pre-SMT chips with their pins bent upside and then mounted upside-down on purpose (it wasn't me!), for some prototype.

Surely we've all designed a circuit board and forgotten to mirror it before printing the lithographic sheet?

I only realised when I noticed my signature line backwards on the resulting board but was so invested by then that I made it work regardless.

I'm guessing that the idiot who soldered had zero notion of electricity and just basically soldered the back of the chip to the motherboard, regardless of the fact that there were pins and emplacements on the motherboard.

So, not only an idiot, but a myopic one as well.

I can't imagine any other solution to this unbelievable situation.

And, if Tex was fired because he pointed out the obvious, then it was obviously not a company worth working for.

A lot of processors have thermal pads in the middle of the underside as well as on the top. I can imagine one of those being placed upside-down and going through the oven.

Not saying that's the only way it could be messed up, but one I can picture.

I duess that an IC with a very low profile enclosure being pressed into solder paste upside down could result in the solder reaching up to the "wrong" side of the pins. If anything can be done wrong, then it will.

Just this spring we had a landscaping firm plant plants upside down (roots pointing upwards), so the stupidity of someone "just doing their job" should not be underestimated.

Some "chips" are capped with metal cans[0]. I have seen a badly adjusted pick & place machine flipping a chip over just before running the motherboard through the wave solderer. Robotic manufacturing, especially in the early days, can cause all kinds of havoc ... but Ive never seen a complete run of prototypes shipped to a customer with this kind of fuck-up. It would be caught by the operator after the first one or two units.

[0] For instance a couple of bare ICs and a few surface-mount devices and perhaps a few printed & etched resistors wired together with something like a gold-ball bonder, all on a substrate, which is then capped with a metal can, sometimes in a vacuum, sometimes in nitrogen or other non-reactive gas. The cap can be epoxied on, or welded, depending on application. The wires for this little circuit are bonded to the external pins before the cap is installed, the pins, in turn, can be soldered to a standard socket. This was sometimes called a hybrid integrated circuit, or hybrid for short.

I'm not a huge believer in conspiracy theories and talk of brown envelopes changing hands* but I've also worked in manufacturing for a great many years and I'd be very suspicious of why the manager didn't want someone knowledgeable on the inspection team. The phrase, "What's in it for him?" springs to mind.

Full disclosure, my Dad ran an inspection department (largely before it was renamed to QC or QA) for many years and yes, there were inducements on offer to be lenient, in at least one case negotiated by the chief buyer without his knowledge. He would never compromise principles however and would treat all suppliers equally regardless.

*It's rarely that overt, it's donations to an election fund or holidays disguised as a business trips or similar.

Report goes up the chain to the decision makers. What they do with that information is somebody else's problem.

buyer wants less inspections??

"What's in it for him?" definitely springs to mind.

Getting fired for that? That Tex the piss

Several years ago I used to work production line jobs, and spent a while making implantable medical devices with built-in electronics at a client manufacturer. We used to receive the PCB's from the supplier upstream already encased in their little plastic resin-filled tubes. Each one had a internal cost of approximately £1000 US dollars.

Unfortunately the US based client who owned the product had only contracted one upstream supplier who could be best described as "cottage industry" who had no room or interest in scaling things up.

The client contracted a large manufacturer to set up a line and boost production. $2 million later they shipped their first batch of 150 to us for processing. Client insisted we skip incoming inspection steps and concessions were drafted to remove all visual inspection stages up until end of line QC, just prior to packing for the sterilisation house.

It was at the end of line QC we discovered that this entire batch of 150 had been encapsulated into their tubes upside down, and although the electronics communicated as normal their essential functions didn't. Cue much screaming from the client about "why hadn't this been spotted sooner" despite them having signed off concessions and demanded we not check anything visually....

It isn't clear from the description exactly where in the chain he fit, but I recall working for an engineering company in my first "corporate" job that a new line of products rolling out from an overseas division (I think in Italy...memory is hazy) Only problem was, they didn't work right - some software problems that made them error prone. They had committed to customers, and stockholders, for a launch by the end of the year, so they dutifully shipped thousands of products in the weeks leading up to Christmas. Not one of them functioned.

I don't remember the details, but they insured a component was installed wrong so that it would power up, but not operate. Customers returned those products, and they relied the calendar working out so two work weeks were burned due to holidays. They had engineers working overtime during the holidays to fix the real problem, and since they'd "broken" the products in a simple way it was easy to rework them and ship them back out in early January apologizing for the manufacturing error claiming a bad lot or something.

They would have got away with it but one of their executives got in a bit of legal trouble so he spilled the beans hoping for a deal. No idea what happened to him, but it caused a lot of higher ups in that division to be fired and maybe prosecuted (no clue there either) and the SEC got involved in the US since it turned out the top people here knew about it. I don't think they could prove anything but there was a shareholder lawsuit I got mail about for years because I'd held a few shares from a company stock purchase plan during that time.

The (extremely large IT) company I used to work for did something similar back in the day. Distributed thousands of corporate-branded USB sticks as part of a large scale marketing promotion; the sticks were huge plastic monstrosities (shark shaped, for some unfathomable reason) with 32MB capacity. MB not GB, in the days where 4GB sticks were the norm.

And they didn't work. Not a single one.

Somebody in Marketing had clearly scored a bulk deal for next to no money on some defective swag that was going to be junked, "saved" the company a fortune, probably got an award for it, and the result was that thousands of potential customers decided to steer well clear of us as we were clearly incompetent. Not a good look.

Clearly they werent the only sharks in on the purchase.

Our new boss decided that the product could be delivered much faster if we could eliminate some of the antiquated procedures we had adopted. Like inspection and pressure-testing before assembly. He placed the order for the materials from an overseas supplier for 'finished product' i.e. inspected, tested, certified, documented and fully compliant with our requirements. A big saving in price. And early to the schedule. An early adoption of 'Move fast and break things'. What could possibly go wrong?

As instructed but against our instincts, we carried out no validation of our own, assembled these major components into our machines and prepared them for customer-inspection and performance test. My instrumentation immediately started raising alarms which were initially brushed off as operator incompetence; a label I never dispute, at least at first. We had all the supplier-documentation which confirmed compliance. However, as the puddles on the floor suggested, these containers were not pressure-tight and never had been, despite the certification, and certainly not suitable to prevent water reaching the high voltage insides..... To his credit, the boss didn't ask me to 'review' the indicators.

It took ages to rectify the problem involving a lot of dismantling, assembling, retesting on multi-repeat. The schedule went away on a handcart. Rework and late delivery meant extra costs....

Most likely, Tex was not so much fired for pointing out the error per se, but for pointing out a bleeding obvious error the manager has managed to miss. It reeks of an organization suffering from [1]injelititis or palsied paralysis , first studied by C. Northcote Parkinson, where high levels of incompetence and jealousy in management mean that anyone showing any degree of competence is removed, as they are perceived as a threat.

[1] https://eden.one/31232

Thanks for putting a name to that which I have experienced with at least one company in my career. I would rather said company actually accepted their incompetence and asked for help, but no it seems that wasn’t allowed.

During the Y2K project I was in, my team was correcting an application consisting of almost a 100 programs chained together (just as COBOL likes) and we were made aware that that particular batch chain would break every single night, meaning an hefty on-call bill. And is had been going like that for years.

So I analysed the failure point and it was seemingly simple to correct - IIRC there were numeric conversions that would fail under certain conditions, conditions that would almost certainly be met once or twice per run.

That correction was deemed out-of-scope by our client rep, who, curiously, was also that application's IT boss.

My previous employer decided that they wanted to bring in an MSP.

We were INCREDIBLY understaffed and this was their solution to "Well, actually, you're just too busy to spend time babysitting the boss when he can't open Teams, so we need more" and rather than hire, they decided to try to get an MSP. With an unstated and repeatedly-denied purpose of replacing in-house IT, I'd like to add. It was so blatantly obvious.

Anyway, they brought on an MSP, despite my objections, and that MSP proceeded to lecture me on my job, and claim to know better than me about everything, and tried to take over everything (it was clear that their brief was "take over the IT" while that was always denied in meetings, etc.).

So... I let them. Not without objections, and clarifications, and pre-warnings and I-told-you-so's, but I let them do it.

It turned out hilarious.

One of their "network team" (they had a "team" for everything, but those teams were always busy with whatever they did, which meant that the slightest query always went back to the MSP, lingered for days, then came back half-hearted with no time spent on it, a bill for doing that properly, and "no, our other guy you're paying can't do that, it has to be the network team") literally lectured me on NTP servers. One of the most trivial and relatively unimportant things ever... we had no need of time sync beyond basic domain operations. But they decided we were "wrong" with our "non-standard" deployment, because we were using a local NTP server and one of our remote NTP servers was one they hadn't heard of.

What they failed to take account of? The local NTP was a literal Bodet NTP radio clock sync device, the kind used in stock exchanges and railway operations. It was designed for site-wide sync and someone (*cough* me *cough*) had bought it as part of an all-site tannoy-like system because it synced time for free to all the units, used GPS and radio for timesync, and provided a local network NTP server that was certified to some ridiculous accuracy. We never needed it, but it was already there and cost us nothing... why not use it?

No, apparently, we had to use time.windows.com.

Then they argued about the remote service and demanded we replace it with NTP Pool servers. They gave me some huge bluff piece on it, and there was a LOT of time wasted on this, especially for something that we absolutely did not need. That's why they tried to lecture me on how NTP works and why the pool was better, and how to configure our NTP. In the middle of which they specified settings which were both insecure but also... that included a particular NTP pool server (in an incorrect way to address it, I'd like to point out). I let them argue with me some more. Then I told them. That's my server. It's literally mine. I operate it. I joined it to the pool. It's been there for over a decades. It's one of the more reliable in the pool. It handles more NTP traffic than the entire commercial network we were using for that employer. Every day. It's literally my personal server that I operate outside of work. You're telling me to use MY server. Then you're telling me how MY server is configured and how it works and that I should use it.

And the fact of the matter is... we already were, via the use of NTP pool. They just didn't understand how it worked.

I had similar run-ins with them on all kinds of issues. They replaced our perfectly-functional intra-site VPN with one that literally didn't work. I know why it didn't work. I told them. I had even pre-warned them, and dropped hints at every opportunity and told them explicitly half a dozen times. But their "network team" never understood that they had to route additional subnets over the VPN or those subnets wouldn't work on the remote site. They were trying to pretend they could interpret Wireshark traces. They were trying to pretend our networks were undocumented (I literally pointed them at the existing, working configuration in plain text). They were trying to pretend that we were doing something completely impossible (It's bloody working already!!!!!). Etc. Etc. But if you don't route those additional subnets over the VPN... then the VPN isn't going to bother to route that traffic.

Everything on the main subnet... fine. Everything on another VLAN/subnet... never transited the VPN. Access control stopped working. Telephones. Printers. Digital signage. Anything on another subnet didn't work on the remote site from the second they put in the totally-unnecssary VPN of theirs. The irony was... their VPN box was literally just a VPN box. To operate, it had to sit behind the routers at each site that... had been running the VPN between those sites. So we had had to turn off the VPN functionality on the router, install two expensive boxes behind them, configure port-forwarding etc. for VPN ports on both ends, and then have the boxes route the VPN traffic... badly and incompletely.

After six months, my employer got tired of the constant arguments and people complaining to me about stuff not working at the remote site (and I just filed tickets with the MSP... not my problem!) and told me to back it out. Ten minutes later (I had saved the config), the VPN worked and all traffic routed properly and we threw away the VPN boxes they'd made us buy. I mean... I had literally told them what was necessary and told them that we routed several subnets over there... not once did they ever put in any additional config to route those other subnets.

Similarly, they installed an new high-availability router device. Massive, expensive rack-mount thing. I asked how they intended to deploy it. They said it had to sit directly behind the main gateway. Okay. Well... we have two gateways, you know. Because we have two leased lines to the Internet for redundancy. And we use both in failover. They said it had to sit directly behind the main one. But what happens when we're in failover? We get no internet, that's what. Again, "the network team know better than peons like you and they've spent months designing this and you're just a guy we intend to put out of a job".

(shrug). Okay.

Because of power and other problems, that site would failover about once a week. There was a reason we had two leased line, two routers, at either end of the site, on independent power supplies. So within weeks, it failed over to the other device. I asked why it wasn't working was still in place. But obviously they couldn't contact their device. Everything else was still working (because we designed the network to work like that, and they were well aware of that) but the router was now entirely out of the loop, sitting on a dead gateway. Worthless.

There were MONTHS of that. Literally MONTHS of accusations flying around about how we must have turned it off deliberately (I honestly didn't need to sabotage the idiots, they were doing quite well by themselves!). But ultimately, they realised... this wasn't going to work. Not only had they spent months putting in a device in a terrible configuration, but even the "HA" portion of it literally never worked. Not in a single demo. Not once. Never. They even made us run 100Gbps fibre between the routers SPECIFICALLY for HA heartbeat. Fibre was fine. HA never worked.

They did something else similar. Bought an IDS/IPS. Attached it to one gateway. It never detected anything. Literally nothing. I kept complaining. And complaining. And complaining. And it kep getting escalated but they assured me it was all working. I got it in writing. MONTHS this went on.

It was at that point that I pointed out that the device they were supposed to install was still sitting in the rack, uncable and unpowered. They'd racked it. And that was it. Then they tried to blame me, but I had not only a trail of evidence, but I'd deliberately pointed it out to my boss who had - sensibly - not said a word and let them drop themselves in it.

When they did cable it and turn it on, it didn't work. Why? It was only monitoring one gateway. Fine while that was the active one. Useless when it wasn't. Our network was unprotected 50% of the time. They claimed that it was fine and they'd checked with the manufacturer and it was a supported config. Strange that. Because I had a written statement from the manufacturer (thanks BlueDog!) who had agreed to talk to me when they realised I was the customer, that they had SPECIFICALLY warned that MSP (the reseller) that they would need at least two such devices, and it would be worthless without. They sent me the email chain. Where the MSP dismissed that and just ignored it, repeatedly, against BlueDog's advice.

Whoops.

But apparently, I "don't know what I'm doing" and their "expert network team" were geniuses. The same geniuses who trashed our SAN in the middle of the day by stomping over IP addresses on a reserved subnet, the same geniuses who couldn't deploy a webfilter onto a Chromebook over WEEKS and said - quite literally - that it was impossible to do with the kit that THEY THEMSELVES had bought, and implied it wasn't possible with any kit. They said they'd never seen a project "so set up to fail" and basically accused me of sabotage (again!). (Presented with this fact by my boss, I took out a fresh Chromebook out and configured it in five minutes in front of the senior management and showed them that it was working... they took it away and confirmed it themselves and presented it to the MSP who were forced to admit they were wrong). The same geniuses, in fact, who remotely booted our in-house team out of remote desktop while we were working on the servers, logged in themselves as the user we'd given them, and for some insane reason decided to APPLY all checkpoints on the VM cluster. Not delete... not housekeeping... not tidying... they APPLIED them. Without warning, reason or permission. Rolling the entire network back months in the middle of the working day. Then denied it. By which point I was already on the phone to my boss telling them what I was watching happening as we spoke and they came and witnessed it. Restore from backup was required. Of the 2-node S2D cluster that they'd forced us to migrate to. Oh, with 1Gbit networking that was run off the motherboard network port, which then failed.

But apparently... little old me, with my strange ways, and my lack of certifications (just a degree and 25+ years experience) was the dumb one who just "didn't understand anything" and their "expert dedicated teams" for security, networking, servers, implementation, etc. did. I wasn't "trained" like their teams of experts. No.. I just built and ran the exact servers they were lecturing me on using, I just designed the network and told them exactly how and why it worked and watched them fail repeatedly. I just realised that if you have two paths to the Internet, then you need to cover both. And so on.

So after we'd dropped some £200k+ on these idiots, they were finally let go after an absolute screaming match where they decided it was a good idea to yell at our only senior management who understood IT and had been brought in specifically to mediate and clear up the confusion (i.e. me telling and demonstrating that these people were idiots, and the MSP playing their roles as idiots perfectly). He calmly replied things like "No, that's absolutely not true, though, is it? That's not how it works, or could ever work." (on technical matters) which basically convinced the entire senior team that actually I'd been right all along.

I asked for a reasonable raise. Was denied. Got a 20% raise elsewhere within HOURS. Went back to them. Nothing. Quit.

I hear they're now employing another MSP, under the charge of the senior manager who had a clue about IT. At significantly more cost than ever before. And they have had to seriously dial down their expectations of the network that were given them when I was running it. You think that a tiny team of in-house staff running EVERYTHING with response times in the minutes was your problem? Okay, see how you deal with an MSP with 40 staff that won't even respond to a ticket same-day most of the time, and who just pass stuff off or back saying it's not to do with them all the time.

Little ol' untrained me. Now working at a bigger place, earning more, actually working less (hours, holidays, systems that were budgeted for appropriately, etc.) and under comparatively zero pressure, with no sign of an MSP (and no intention to get one).

Wow. Just, wow!

You must have some very interesting stories to tell.

;)

that would be gibbering lunacy just behind blithering idiocy for line honours ?

... don't shoot the messenger.

If they're that clueless, then probably time to move on anyway before the whole thing collapses.

Never underestimate the ingenuity of fools.

About 4 years ago a took a car for an MOT test with a brake pad inserted into the caliper (by my own hand) much the same way this chip was mounted .

oh how me and the boys at the test station laughed ...