Rivian bricks infotainment systems in 'fat finger' fiasco
- Reference: 1700046014
- News link: https://www.theregister.co.uk/2023/11/15/rivian_bricked_infotainment_systems/
- Source link:
The manufacturer took to [1]Reddit to update customers on what went wrong – the hyper-technical condition prone to cause IT errors known as a "fat finger."
"We made an error with the 2023.42 OTA update – a fat finger where the wrong build with the wrong security certificates was sent out," explained Rivian.
[2]
The infotainment system and in some cases the instruments were affected on a number of R1T and R1S vehicles. The cars were still driveable, just potentially less enjoyable without climate control, music, and the assurance of adherence to speed limits.
[3]
[4]
"A vehicle reset or sleep cycle will not solve the issue," said the vehicle maker.
The manufacturer explained that it had cancelled the OTA campaign and would restart with proper software. Rivian announced that the team had developed a solution, which was expected to be ready by Wednesday.
[5]
Unfortunately, for some customers affected by the issue, updates alone may not suffice, meaning a trip to receive physical repairs.
Some Redditors offered empathy. "I once deleted shark week. True story. They had to get the site from backups," said one.
"At least they didn't blame an intern," commented another less generous observer.
[6]
A few asked for heads to roll, such as one owner questioning why there was not A/B firmware architecture, who didn't seem appeased even after Rivian booked a rental car on their behalf.
[7]Thousands of Teslas recalled over brake fluid bug
[8]Ford SYNC 3 infotainment vulnerable to drive-by Wi-Fi hijacking
[9]EV truck maker Nikola stalls in 2023, pulls out of Europe, hits brakes on production
[10]Ford in reverse gear over AM radio removal after Congress threatens action
"This kind of stuff is unacceptable on a 100k car," wrote another owner.
But mostly the Redditors just wanted their vehicles fixed or came to offer advice and assist in troubleshooting.
"For those in cold/hot weather, you can at least pre-set and pre-heat/cool your vehicle even without the infotainment – at least if yours ended up in the same state that mine did," said one.
Rivian's decision to post the information to Reddit rather than on their website is curious. The company said in the post that it was contacting affected owners via email and text. The EV maker's website does include a chat feature, but in The Reg's experience, to get any details on infotainment system outages the inquirer will need to have a user account.
The Register has contacted Rivian to better understand these decisions and the corrective rollout.
Rivian is far from the only manufacturer to suffer infotainment system woes. In August, Ford [11]warned that its SYNC 3 infotainment was susceptible to drive-by Wi-Fi hijacking. That scenario, however, was corrected by an OTA update. ®
Get our [12]Tech Resources
[1] https://www.reddit.com/r/Rivian/comments/17usikn/202342_ota_update_issue/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZVT5OJU8tnPi0j62i9XSjwAAAFM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZVT5OJU8tnPi0j62i9XSjwAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZVT5OJU8tnPi0j62i9XSjwAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZVT5OJU8tnPi0j62i9XSjwAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZVT5OJU8tnPi0j62i9XSjwAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/10/17/55000_teslas_recalled_over_brake/
[8] https://www.theregister.com/2023/08/14/ford_sync_vulnerability/
[9] https://www.theregister.com/2023/05/09/nikola_results_europe/
[10] https://www.theregister.com/2023/05/24/ford_backtracks_on_am_radio/
[11] https://www.theregister.com/2023/08/14/ford_sync_vulnerability/
[12] https://whitepapers.theregister.com/
And this is why a bought a car with physical heater controls and an infotainment system that is separate to the vehicle operation, and that isn't connected to the 'net!
Sadly by the time I replace it I suspect there won't be any option.
It's possible that manufacturers will have remembered that drivers like not having to look at the controls by then.
Unfortunately, there's no reason a firmware update can't brick physical controls if they are still tied into the same system.
100K for a Nevaherdov?
As Musk is still finding out, designing a prototype is easy. Selling and distribution is more complex than it looks. Manufacturing is painful and difficult, Warranty, spares and service even more so. Evidently software is also a challenge. And whilst you keep those plates spinning you need to build the high volumes to survive, and be working on a model replacement cycle that extends out to the replacement for the replacement of your current models.
And Rivian investors are discovering that. A $100 purchase of stock when the floated means you now own stock worth $13. And despite that, markets place Rivian's value at roughly a third of Ford. I'm pretty sure I know who will still be around in thirty years time, making and selling cars profitably, and it doesn't begin with R.
What, no speedo?
Illegal to operate in the UK, then...
Re: What, no speedo?
... and a bit tricky to take into the dealer for a fix. Made worse, of course, by the fact that an upmarket product for a self-elected elite of Very Special People will have a sparse dealer network.
Re: What, no speedo?
"an upmarket product for a self-elected elite of Very Special People will have a sparse dealer network"
Ahhh, that's the one thing that doesn't matter. When you're a VSP, you delegate to your flunkies. Some years ago I got delegated with a consulting colleague to go and do a car swap for the CEO. My cost to the company was north of £100k a year, the consultant about double that, but regardless we set off together to do the round trip of 80 miles, spending most of the afternoon doing so.
Re: What, no speedo?
By my reading, I think that it was just the cruise control or maybe speed excess warning that was borked,* not the speedometer itself. But the 'and instrument systems' is rather vague; it includes speedometer, of course, but it's not ' all instrument systems'.
* hence the "no assurance of adherence to speed limits."
Re: What, no speedo?
About 20 years ago, I temporarily borked my OH's car's dashboard by discovering a hidden test mode. Holding down the trip computer reset button whilst turning on the ignition sent it into a diagnostics mode with all the dash lights flashing and all the dials flicking up and down.
The car still drove ok so we got home, but had no instruments. This persisted through an ignition cycle and even removing the battery. Eventually found that simply pressing the trip reset button again reset it back to normal. in hindsight, that was fairly obvious but I didn't think of it at the time.
Re: What, no speedo?
"Illegal to operate in the UK, then..."
Not true.
A vehicle's speedometer must be kept free from any obstruction that may prevent it from being read and shall at all times it is used on a road be maintained in good working order. The only exceptions to this are when:
- The speedometer became defective during the journey being undertaken, or
- Steps have been taken to have the defect remedied by replacement or repair as soon as possible.
- The vehicle has an approved tachograph.
(1) and/or (2) apply here so Rivian owners would be fine.
Simple solution - a phased rollout system that hits the CEO's device, whatever sort of device that may be, a couple of days before those of paying customers.
or even better, a lot of internal QA...
This isn't a game they're updating, it's a frigging car, it should be treated the same as an entirely new model due to go on mass sale and tested for safety accordingly
"tested for safety accordingly"
Tested for safety?
I suppose that's why an Aimotive Inc 'disengagement report' dated 2022/06/15 states " Our software has a limitation in that we do not (yet) cancel lane changes once it has been initiated; however, we do cancel lane changes if the software deems it unsafe prior to the actual maneuver. In this case we had already started the maneuver, therefore the driver had to take over. This isn’t a software bug, it’s a known issue that is being worked on " (my emphasis).
>>a phased rollout system that hits the CEO's device, whatever sort of device that may be
I imagine that a Rivian targeted OTA update will have little effect on the CEO's Ferarri... I would bet 50p that he doesn't drive is own brand!
The Register has contacted Rivian to better understand these decisions
I suspect that understanding them might only make them seem worse.
Wrong security certificate?
"the wrong build with the wrong security certificates was sent out".
IT security is not my forte, but can someone explain how something with the wrong security certificate was advertised and installed by the car? I presume it didn't really on users clicking ok on a "incorrect security certificate" type of message.
Re: Wrong security certificate?
Assuming Rivian aren't so bad at this stuff that their OTA mechanism will blindly accept incorrectly-signed packages, then could it be that the OTA package itself was signed correctly, and the problem is that the contents of the package was *both* the firmware image AND the corresponding certificate, such that the update installed both the new image and the new cert at the same time without first checking one against the other due to a presumption that, as the package they arrived in WAS signed properly, the contents of said package MUST be correct. Right. Riiiiight???
Re: Wrong security certificate?
"Verify this certificate/fingerprint" really means "look at these random numbers --- aren't they pretty? --- and press 'ok'".
Users are lucky if they even get the random numbers.
> I presume it didn't really on users clicking ok on a "incorrect security certificate" type of message.
Very possible and even likely but it could also have been at the manufacturer.
100k for a car that basically requires an always-on internet connection if you want to know what FREAKING SPEED you're driving at!
or to listen to radio 1
Holy crap, and I thought Diablo 4 was bad.
So basically Revian cars CANNOT be used for cross-UK driving as you'll lose cellphone connection at some point and the dashboard will crash!
So basically, wrong...
Provided the onboard firmware is running correctly at the start of the journey (and this is the crux of the issue - the borked OTA update prevented THIS condition from being true), then losing network connectivity at any point once you've started to roll should be at most a minor inconvenience if it means not being able to play livestreamed audio, receive live traffic updates etc, and in many cases will go entirely unnoticed by the driver.
"A vehicle reset or sleep cycle will not solve the issue"
Vehicle reset?
In the old joke, that used to mean "stop, take the engine out, put it back in, start". But that was a joke .
Sorry, I forgot, this ain't a car, it's a smartphone on wheels.