Algorithm used to predict sepsis infections in hundreds of US hospitals isn’t as good as maker claims — study
(2021/06/23)
- Reference: 1624413371
- News link: https://www.theregister.co.uk/2021/06/23/ai_sepsis_mistakes/
- Source link:
An algorithm used by hundreds of US hospitals to predict whether or not patients with infections have contracted sepsis is less accurate than its maker claims, according to a study.
Sepsis is a leading cause of death in American hospitals — more than a third of patients have it when they die. So Epic Systems, a major healthcare software provider whose products are used in the majority of US hospitals, developed a tool to identify whether a patient is at risk of sepsis.
The idea is that if hospitals can foresee cases of sepsis they’ll be able to provide care for patients before their condition gets worse. But not only does the algorithm tend to forecast patients will contract sepsis when they don’t, it’s less accurate at correctly identifying cases when people really do end up getting infected with the life-threatening issue, it is claimed.
The model predicts sepsis that was recognized by clinicians at the time it was recognized by clinicians
Epic reckons its algorithm is accurate up to 83 per cent of the time, yet a paper [1]published in JAMA this week alleges it’s actually only good in about 63 per cent of cases.
The issue lies with how the model comes up with its predictions. “Epic chose to use billing codes to define the outcome of sepsis,” Karandeep Singh, co-author of the study and an associate professor focused on machine learning and healthcare at the University of Michigan, explained to The Register . The model analyses what drugs or medical procedures that a patient is being charged for to determine whether a patient is at risk of sepsis, we’re told.
[2]
It’s often not very helpful, for example, in cases where antibiotics are already being administered to a patient for sepsis. “In essence, they developed the model to predict sepsis that was recognized by clinicians at the time it was recognized by clinicians. However, we know that clinicians miss sepsis,” Singh [3]added .
[4]
[5]
Singh said the research team raised their concerns to the company in April. Epic, however, doesn’t agree with the academics.
“The authors used a hypothetical approach,” a company spokesperson told El Reg . “They did not take into account the analysis and required tuning that needs to occur prior to real-world deployment to get optimal results. In order to predict who might become septic, the model is trained on past patients who had a clinical diagnosis of sepsis.”
[6]That AI scanning your X-ray for signs of COVID-19 may just be looking at your age
[7]Mounties messed up by using Clearview AI, says Canadian Privacy Commissioner
[8]Apple sued in nightmare case involving teen wrongly accused of shoplifting, driver's permit used by impostor, and unreliable facial-rec tech
[9]Insurance startup backtracks on running videos of claimants through AI lie detector
The academics tested the model with 27,697 patients who had been hospitalised 38,455 times. Most did not go on to develop sepsis during their stay. Epic’s algorithm was only able to predict whether they would be affected by sepsis before doctors could only seven per cent of the time, and has a false positive rate of 18 per cent, the researchers said.
A better approach for the software would be to use a model that analyses healthcare symptoms defined by health agencies, such as the US Centers for Disease Control and Prevention, rather than just relying on billing codes, it would seem.
[10]
“This is not typically how sepsis is defined for the purposes of quality measurement or model development. There are several consensus criteria that exist. Either of these would have been preferable to simply using billing codes,” Singh told us. ®
Get our [11]Tech Resources
[1] https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2781307
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://labblog.uofmhealth.org/lab-report/popular-sepsis-prediction-tool-less-accurate-than-claimed
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2021/06/07/in_brief_ai/
[7] https://www.theregister.com/2021/06/11/canadas_privacy_watchdog_said_police/
[8] https://www.theregister.com/2021/05/29/apple_sis_lawsuit/
[9] https://www.theregister.com/2021/05/26/ai_insurance_lemonade/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
Sepsis is a leading cause of death in American hospitals — more than a third of patients have it when they die. So Epic Systems, a major healthcare software provider whose products are used in the majority of US hospitals, developed a tool to identify whether a patient is at risk of sepsis.
The idea is that if hospitals can foresee cases of sepsis they’ll be able to provide care for patients before their condition gets worse. But not only does the algorithm tend to forecast patients will contract sepsis when they don’t, it’s less accurate at correctly identifying cases when people really do end up getting infected with the life-threatening issue, it is claimed.
The model predicts sepsis that was recognized by clinicians at the time it was recognized by clinicians
Epic reckons its algorithm is accurate up to 83 per cent of the time, yet a paper [1]published in JAMA this week alleges it’s actually only good in about 63 per cent of cases.
The issue lies with how the model comes up with its predictions. “Epic chose to use billing codes to define the outcome of sepsis,” Karandeep Singh, co-author of the study and an associate professor focused on machine learning and healthcare at the University of Michigan, explained to The Register . The model analyses what drugs or medical procedures that a patient is being charged for to determine whether a patient is at risk of sepsis, we’re told.
[2]
It’s often not very helpful, for example, in cases where antibiotics are already being administered to a patient for sepsis. “In essence, they developed the model to predict sepsis that was recognized by clinicians at the time it was recognized by clinicians. However, we know that clinicians miss sepsis,” Singh [3]added .
[4]
[5]
Singh said the research team raised their concerns to the company in April. Epic, however, doesn’t agree with the academics.
“The authors used a hypothetical approach,” a company spokesperson told El Reg . “They did not take into account the analysis and required tuning that needs to occur prior to real-world deployment to get optimal results. In order to predict who might become septic, the model is trained on past patients who had a clinical diagnosis of sepsis.”
[6]That AI scanning your X-ray for signs of COVID-19 may just be looking at your age
[7]Mounties messed up by using Clearview AI, says Canadian Privacy Commissioner
[8]Apple sued in nightmare case involving teen wrongly accused of shoplifting, driver's permit used by impostor, and unreliable facial-rec tech
[9]Insurance startup backtracks on running videos of claimants through AI lie detector
The academics tested the model with 27,697 patients who had been hospitalised 38,455 times. Most did not go on to develop sepsis during their stay. Epic’s algorithm was only able to predict whether they would be affected by sepsis before doctors could only seven per cent of the time, and has a false positive rate of 18 per cent, the researchers said.
A better approach for the software would be to use a model that analyses healthcare symptoms defined by health agencies, such as the US Centers for Disease Control and Prevention, rather than just relying on billing codes, it would seem.
[10]
“This is not typically how sepsis is defined for the purposes of quality measurement or model development. There are several consensus criteria that exist. Either of these would have been preferable to simply using billing codes,” Singh told us. ®
Get our [11]Tech Resources
[1] https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2781307
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://labblog.uofmhealth.org/lab-report/popular-sepsis-prediction-tool-less-accurate-than-claimed
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2021/06/07/in_brief_ai/
[7] https://www.theregister.com/2021/06/11/canadas_privacy_watchdog_said_police/
[8] https://www.theregister.com/2021/05/29/apple_sis_lawsuit/
[9] https://www.theregister.com/2021/05/26/ai_insurance_lemonade/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YNKx40QNtORAtxRBaCze7gAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/