Dodgy procedures doomed Arianespace's Vega before it even left the launchpad
- Reference: 1608294613
- News link: https://www.theregister.co.uk/2020/12/18/dodgy_procedures_doomed_arianespaces_vega/
- Source link:
Which, frankly, sounds a lot like someone plugged something in the wrong way round.
The rocket blasted off on schedule at 01:52 UTC on 17 November from the French Guiana spaceport, carrying a pair of European-built satellites: Spain’s SEOSAT-Ingenio and France's TARANIS. All went well for the first eight minutes until the AVUM upper stage was ignited. "At that time, a degraded trajectory was detected, followed by a loss of control of the vehicle and the subsequent loss of the mission," [2]according to Arianespace at the time .
[3]Suspicion fell on the integration of the fourth-stage AVUM Thrust Vector Control (TVC) system.
[4]
Vega’s dual-satellite payload being encapsulated in the payload fairing for flight VV17 back in November Pic: Arianespace
An Independent Enquiry Commission (IEC) set up by ESA and Arianespace has found that, yep, it was the TVC wot dunnit and no, the design itself wasn't to blame. Instead, it was that "inversion of electrical connections" during integration that inverted TVC steering commands and, as the report delicately put it, caused a "trajectory degradation."
The sequence of events makes for grim reading. The integration procedure was "misleading" and the resulting incorrect routing and connection of the control lanes was not picked up in testing.
It was only the second Vega launch since [5]2019's failure and the 17th overall. While the earlier failure was a design flaw, managers are doubtless relieved that the recommendations this time around involve some additional inspections of existing hardware and changes to the manufacturing, integration and acceptance processes on Vega assembly lines in Italy and its final acceptance in French Guiana.
The Vega had a busy 2021 planned, and the next launch was scheduled for the first quarter. Stéphane Israël, CEO of Arianespace, praised the clarity of the report's conclusions and said: "It may allow a return to flight by end Q1 2021, in full confidence to Vega's quality and competitiveness on the market."
Still, Arianespace is in good company. One can't help but remember 2013's Russian Proton-M mishap, reportedly the result of an engineer slotting in a critical component [6]the wrong way around . ®
Get our [7]Tech Resources
[1] https://www.esa.int/Newsroom/Press_Releases/Loss_of_Vega_flight_VV17_Independent_Enquiry_Commission_announces_conclusions
[2] https://www.arianespace.com/mission-update/vv17-anomaly-inquiry/
[3] https://www.theregister.com/2020/11/19/esa_vega_failure/
[4] https://regmedia.co.uk/2020/12/18/vega_vv17_payload_prep.jpg
[5] https://www.theregister.com/2019/07/11/arianespace_vega_failure/
[6] https://www.theregister.com/2013/07/10/human_error_blamed_for_toxic_russian_rocket_explosion/
[7] https://whitepapers.theregister.com/
Re: Are they really saying there were no design issues?
Only if the wires are put into the connector in the right positions.
Re: Are they really saying there were no design issues?
"The integration procedure was "misleading" and the resulting incorrect routing and connection of the control lanes was not picked up in testing."
That says it is a design issue, in the actual design as well as the testing process.
Re: Are they really saying there were no design issues?
Or have different lengths of the spurs from the loom so the short spur cant reach the connector at the end. But never underestimate the ingenuity of the installer to make it fit!
Re: Are they really saying there were no design issues?
Red plug, red socket. Blue plug, blue socket. It is all about decreasing the chance of error. Color coded connectors will not stop mistakes, but it helps find them.
Color blind? Well....
The best wiring guy i have is color blind. In terminating a cable, he looks at the twist on the cat6 wires to determine which wire goes where.
Re: Are they really saying there were no design issues?
"Wouldn't the use of keyed connectors and/or better testing have caught this?"
Keyed connectors -- Maybe. I was once tangentially involved in an incident investigation of a missile that crashed a few hundred meters downrange during a routine test while tumbling rapidly end over end. The cause was determined to be a manufacturing defect. As it was explained to me, there were steering vanes in the rocket exhaust that were driven by a motor. The motor had tabs to which the control wires were connected. In this case, backwards. Using tabs of incompatible sizes would (probably) prevent that. But if your Mil-Std qualified motor comes as a lot of (most?) motors do with identical connectors for both wires, you're sort of stuck with it.
Better testing? For sure. At least in that case. The test for the wired motor was that powering it moved the steering vane. Nothing about which direction. The test spec was changed forthwith.
Note to self
...Must remember to design in polarised connectors on billion-dollar missions.
Re: Note to self
The problem is that if you have a hundred connectors, you now need a hundred different connectors, all space-qualified. You could maybe solve this by having one kind of connector which could, after manufacture, be keyed to fit only its partner, but then you're relying on people keying them properly.
Wrong photo for this story
I remember when El Reg would have pulled the Airplane still of the guy holding cables and that look on his face...
I would hate to be the guy who mixed up these connectors. I'm no rocket scientist, but wouldn't it be prudent to make different shaped connectors for stuff like this? Making it impossible to cross lines?
Re: Wrong photo for this story
Without trying to shift blame here.. There will be a limited amount of connector types available that are certified (in house) for the use in the product. Typical mil-spec round connectors are not keyed by a polarisation pin, but raher by keying in the enclosure. So per shell size and pin count, the designer may be left with only 4 different key options. If these, one may be available on the market, with lead times of 10-20 weeks for the alternative key options.
Now add to this, a push to use as many as possible identical modules in order to get the part count up (and the number of non-interchangeable parts down), you may end up with Port A of Gizmo X always having the same keyed connector. Now you are left with cable length, cable routing and cable labeling as the only things preventing you from plugging cable A into the correct Port A in the wrong Gizmo A.. And you must have routines in place, that will pick it up if you DID plug it in the wrong way. This may also be solved in software if the signal sent over cable A contains ID information. However, if the signal is a (semi) analogue controlling signal, then this may not be possible.
Generall speaking Aerospace deals with very expensive hardware. However, the very low production numbers mean that component manufacturers will not bend over backwards to provide a large range of infividualized parts at shortest possible notice even if a single connector may cost $100. It is easier to make money by switching on a machine that makes 100.000 identical pars for a car- or a smartphone-manufacturer.
As the saying goes, the quickest way to make a small fortune in aviation is by starting with a big one.
Re: Wrong photo for this story
Add in the possibility that you might have to use 2 of the same items - then they'll have the same connector. (Normally sub-D like old serial ports)
Then you are down to cable lengths.
Think about what happens in a comms cabinet when someone unplugged a few ethernet cables from a core switch, and puts them back in a different order, thinking that one network cable is the same as any other....
It really isn't a trivial matter making flight compatible keying to add to this connector type
A matter of deduction....
When faced with all the evidence, and you have eliminated the most likely causes, one has to conclude that what is left, is actually right....
Re: A matter of deduction....
The day of the failure, the TMs made it obvious.
By the start of the next day, they had found photos of the assembly showing the incorrect connections.
Causes
" One can't help but remember 2013's Russian Proton-M mishap, reportedly the result of an engineer slotting in a critical component the wrong way around. "
The component was an attitude sensor, and in addition to having an arrow on it, it was mechanically keyed to fit the right way round. The original Russian report noted that "considerable force had been used" to insert the component incorrectly - i.e. someone probably hammered it in.
The cause of the Vega failure seems rather different - an error due to poor communication and the high complexity of the system.
"not picked up in testing"
Then the testing is not sufficient.
I really can't imagine any scenario where plugging something in the wrong way around won't create merry mayhem with the electrical signals. Electricity is not something that adapts itself to the configuration - either you plug it in right or you're going to get a melted component.
Seems like Vega needs some physical modification in the connections to ensure that you plug it in right - like computer PSU connectors on motherboards. We've had those since forever, why does rocket science not have them ?
Re: "not picked up in testing"
Yes, its not rocket science is it.
Re: plug it in right or you're going to get a melted component
Depends on what is sent over the connection. If it's just a relative voltage that is perfectly acceptable as positive or negative, your component will be fine but it will be outputing inverted results.
Testing should still have picked this up though. An expected result suddenly inverting should have raised some eyebrows.
Also there's the possibility a connector is pinned-up incorrectly. So it's possible to plug a connector in with the proper orientation, keyed or unkeyed, but still the connections are wrong.
Re: plug it in right or you're going to get a melted component
I am reminded of the Hawkwind 'PXR5' album cover. Fairly sure that it either had to be changed or it was covered by sticker when it was sold.
Re: plug it in right or you're going to get a melted component
Yep, just googled that: https://en.wikipedia.org/wiki/PXR5 . Never seen it before. Given that PAT is part of my job these days, I would surely notice it :)
Link to image in question (don't do it at home folks!):
https://www.vinyliciously.com/images/uploads/HAWKWIND---PXR5-2015-UK-Grey-2LP-2.jpg
Re: plug it in right or you're going to get a melted component
Well, it does contain the tracks "Death Trap" and "We like to be frightened"...
From an interim report a while ago I think it wasn't so much a plug put in upside down as the plug that was supposed to go to unit 1 of something was plugged into identical unit 2 and vice versa. This meant that when the flight computer said "Left hand down a bit" the wrong unit responded and it screaming "No! Your OTHER left!" just made things worse. Splitting the routing of control and sensor feedback would be a sensible solution if it isn't done already, a test of "Left hand down a bit" with the feedback going through a different route would need at least two wiring errors to not give a result of "Left hand moving up a bit".
Are they really saying there were no design issues?
Wouldn't the use of keyed connectors and/or better testing have caught this?