News: 1646746032

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Rate of autonomous vehicle safety improvement slowing – research

(2022/03/08)


Autonomous vehicles in the US are showing a declining rate of improvement in one key safety metric, according to research outfit IDTechEx. Despite this, the industry is making progress and may be nearing early commercialisation of technology such as robotic taxis.

IDTechEx has cast its eye over the [1]2021 autonomous testing and disengagement data from the California Department of Motor Vehicles (DMV), figures that can be used to gauge how safe autonomous vehicles (AVs) are.

Disengagement figures refer to the number of times an AV operator had to disengage the vehicle's autonomous driving function and take manual control while testing the vehicle.

[2]

Miles per disengagement. Source: IDTechEx

According to IDTechEx, the progress seen in the miles per disengagement metric has slowed between 2020 and 2021 for the biggest three firms developing so-called robotaxi vehicles. Prior to this, performance typically more than doubled year-on-year, but the most recent figures show an increase of just 1.5x.

Waymo is singled out for putting in by far the largest test mileage, at 2.3 million miles (3.8 million km) in 2021, a quadrupling its previous effort. However, Waymo's disengagement metrics have fallen from 29,900 miles (∼48,120km) per disengagement to just 8,000 miles (∼12,875km).

[3]

IDTechEx notes that Waymo has recently increased its testing within the San Francisco area, and speculates that the more crowded urban environment with more traffic, cyclists and pedestrians has proven a much more difficult challenge for its AV systems compared with the locations where the company had previously been testing.

[4]

[5]

Perhaps this is one reason why Waymo was recently reported to be [6]suing California's Department of Motor Vehicles in an attempt to keep information about its AV crashes and other operational details a secret.

According to IDTechEx, this change sees Waymo drop to seventh place in the disengagement metrics, to be replaced in third place by China's DiDi, which has increased its performance from approximately 5,000 miles per disengagement in 2020 to about 41,000 miles per disengagement in 2021.

[7]

The second place goes to Cruise, which reportedly achieved 41,700 miles per disengagement, 46 per cent up on its 2020 results. Top of the leaderboard is AutoX with 50,100 miles per disengagement, which is apparently a 2.5 times improvement over their 2020 performance.

However, IDTechEx notes that AutoX covered a total testing distance of just 50,100 miles and reported just a single disengagement, so this achievement must be taken on that context. If they had experienced just one more disengagement, their performance would have been cut in half, whereas Waymo and Cruise racked up much more testing miles in total and so another disengagement would have less effect on their score.

[8]

IDTechEx analysis of California DMV, Disengagement Report 2019/2020. Source: IDTechEx

Despite this slowing rate of progress, as measured by miles per disengagement, IDTechEx claims that the outlook is optimistic for autonomous vehicles, and predicts that we shall see some early (and very limited) [9]commercialisation of robotaxis this year.

While California and China are the key areas for AV development, the research firm also points to other places where trials are planned. One is Moovit, powered by Mobileye, expected to begin testing in Tel Aviv, Israel and Munich in Germany by the end of this year.

[10]Waymo, Cruise get green light from California's DMV for self-driving taxi services

[11]Will this Mobileye IPO fund Intel's comeback plan?

[12]Indian services giants target emerging technologies with PaaS plays

[13]Baidu plans 100-city robot taxi rollout by 2030

IDTechEx claims that Germany is now a growing hotspot for AV, with recent law changes paving the way for commercialisation of autonomous services on German roads. It predicts that Germany will be the second place after Japan to permit level 3 autonomous vehicles for public use, with an updated Mercedes S-Class set to be Germany's first level 3 vehicle later this year.

The research firm's report "Autonomous Cars, Robotaxis & Sensors 2022–2042" has more details [14]here . ®

Get our [15]Tech Resources



[1] https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/disengagement-reports/

[2] https://regmedia.co.uk/2022/03/08/milesperdisengagement.png

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YieLuWbt8eToHE68cACeOAAAAIE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YieLuWbt8eToHE68cACeOAAAAIE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YieLuWbt8eToHE68cACeOAAAAIE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2022/01/31/in_brief_al/

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YieLuWbt8eToHE68cACeOAAAAIE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[8] https://regmedia.co.uk/2022/03/08/testingmiles.png

[9] https://www.theregister.com/2021/10/03/amazons_astro_home_ai_bot/

[10] https://www.theregister.com/2021/10/03/amazons_astro_home_ai_bot/

[11] https://www.theregister.com/2022/03/07/intel_files_for_mobileye_ipo/

[12] https://www.theregister.com/2022/03/03/indian_services_giants_target_emerging/

[13] https://www.theregister.com/2022/03/02/baidu_q4_2021/

[14] https://www.idtechex.com/en/research-report/autonomous-cars-robotaxis-and-sensors-2022-2042/832

[15] https://whitepapers.theregister.com/



msobkow

If you want real safety improvements, the vehicles need to be able to network and coordinate their movements to optimize the traffic flow and safety gaps. But that can't happen as long as bags-of-mostly-water are allowed to control vehicles. :)

Filippo

I'm going to try to make a prediction here. I don't claim certainty. Maybe in ten years' time I'll fish it out and either shake my head or nod sagely.

The prediction is: for the general case, we are not going to get a self-driving car that's so reliable that you can stop paying attention. We might get it for special cases - motorways, some kind of special lanes, very slow speeds, things like that. Not for regular urban roads, not any time soon.

Much like so-called "artificial intelligence", current self-driving technology will improve a bit more, and then reach a plateau where further improvement will be very difficult, and it still doesn't work anywhere close to well enough. Getting past that point will require some kind of largely unpredictable major breakthrough, not incremental progress.

I'd still love to have a car that can self-drive on the motorway, though, and that might be feasible, especially with some kind of smart motorway infrastructure.

@Filippo

codejunky

"I'd still love to have a car that can self-drive on the motorway, though, and that might be feasible, especially with some kind of smart motorway infrastructure."

I think this will be the winner because of all the self driving technology with obvious benefits this is it. I expect it makes sense for haulage to get the truck and goods over long distances with only local driving being needed and it would improve the driving experience for people travelling long distances. It could even increase the speeds on the motorway! Not that the law makers would do that.

Re: @Filippo

AMBxx

The logical conclusion ends up as implementing really big trolley buses.

jmch

One disengagement per 8000 miles in a mixed environment seems quite good performance ... If disengagement simply means human had to take over rather than any accident.

While I agree that general AI suitable for all driving conditions is unattainable, a mixture of better sensors, better processors, improved training sets and specifically designed roads will, I think, bring a 'close to' general case for most roads, not just motorways within 10 years.

It's more a question of the time it takes for infrastructure to be renewed.

Oddlegs

I drive around 1000 miles per month. From a technical point of view one disengagement per 8000 miles is fantastic but if I can effectively ignore my car for 8 months while it drives itself then it's very unlikely I'm going to be paying attention the one time in 8000 miles it needs me to do something.

This is the inherent problem with autonomous cars. Them becoming better means the driver pays them less attention which means when they do need attention they're less likely to get it. Autonomous cars are probably already good enough to fall into this awkward middle ground. Moving past it is going to take an awful lot of work. Any disengagements are probably too many and autonomous cars should concentrate on manoeuvering themselves automatically into a safe position if they're unsure of something rather than expecting a distracted driver to help them out.

AMBxx

But, but they had an exponential chart! Reliability will soon surpass infinity.

Wish you were wrong. But I don't think you are

vtcodger

I'd like to be wrong, but I fear you are pretty much right. Expressway driving is pretty simple most of the time. Except when it isn't. So I think autonomous long distance trucking may be feasible. ... As long as the vehicles reliably pull over and wait for a human to help them out when they come to something -- snow, an accident, construction zone, whatever -- that they can't handle. Maybe a human can shepard them to the expressway on-ramp and to the terminal at the end of the trip? I have no idea whether the economics work. And the potential problems are legion. But just maybe that works. And if that works, you and I will likely be able to put our vehicles into autonomous mode on the expressway with some confidence that it'll wake us up when it needs help rather than assassinating us or innocent third parties.

But the notion that autonomous vehicles can safely drive urban, or suburban, or rural roads now or any time soon seems to me to be crazed. Too bad. I'd feel a lot safer sharing the road with well designed, reliable, autonomous vehicles. If only we knew how to build them.

Today In Solution We Will Never Have To A Problem That Doesn't Exist News

Alan Bourke

from your correspondent Crash Likely.

Will never work

Yet Another Anonymous coward

The autonomous cars will presumably stick rigidly to speed limits

Re: Will never work

Mike 137

" The autonomous cars will presumably stick rigidly to speed limits "

There are reports that Teslas don't always. Sometimes at least they seem to 'keep up with the traffic' and in one case a Tesla accelerated into a crash barrier, hitting it at 70 MPH.

"progress seen in the miles per disengagement metric has slowed"

Mike 137

There are two factors that seldom get mentioned. One (and the most important) is that, in the absence of understanding autonomous systems can only be reactive to a limited range of stimuli, so there's inevitably an upper limit to improvement. The other is that testing has almost certainly been conducted in increasingly complex situations as the technologies have developed. Both factors make apparent progress increasingly hard.

However, overridingly, until cars can genuinely think, accumulate complex memories and draw intuitive judgements from them, the technologies will never reach the competence of the highly experienced and attentive driver. The best we'll probably achieve is replacement of the least attentive and most careless.

As an example of what I mean, I was driving one dark morning along a narrow unlit country road with a 50 MPH limit and many concealed tracks on both sides to plutocrat dwellings. Not far ahead off my near side, I briefly saw a moving red light partially concealed by trees. My brain said "fast car emerging suddenly into the road" so I braked. I was dead right - a low sports car shot into the road from an invisible track and turned across in front of me. If I hadn't braked we'd have collided. I'd be surprised if an autonomous system could work that out as the brief glimpse of the red light was about 45 degrees off my near side only lasted a second or so, and the offending car did emerged a few seconds later with no other indications of its presence in between.

How's the old saying go?

juice

"The last 10% of building anything takes 90% of the effort"

So far, the various autonomous vehicle initiatives have been able to pick off some relatively low-hanging fruit. Motorway driving, where vehicles are generally going at the same speed, lanes are clearly marked and rules are clearly defined. Supermarket car parks, where everything is slow moving and clearly marked. Etc.

But now, we're hitting the complex stuff. Town and city driving, where pedestrians and cyclists can - and do - behave irrationally or unsafely. Road layouts (at least in Europe) inherited from the romans and then mutated through centuries of gradual evolution and hasty modern hacks to deal with population growth and the mass adoption of motorised vehicles. And so on.

And dealing with all these scenarios is exponentially more difficult than what's already been solved. And they all need to be solved to the level where 99.99% of situations can be safely dealt with, because neither the public nor the various government car-safety departments will accept anything less...

Reapply as necessary.