Two new studies show that while AI can beat physicians when it comes to patient diagnoses, the technology doesn’t appear to offer improvements in productivity. Here’s what researchers found in reviewing how physicians use AI for diagnosing clinical cases and taking patient notes.
AI and diagnoses: Science
A study in Science found that an AI system built by Microsoft named Diagnostic Orchestrator to diagnose complex medical cases reached a correct diagnosis in about 83.2% of cases. Physicians, by comparison, got the right diagnosis in 36% to 47% of cases.
Researchers used 304 clinical cases from The New England Journal of Medicine and 325 emergency department encounters to test the diagnostic skills of humans and AI. Unlike other studies that present one scenario, the Science research started by providing only a brief clinical presentation. The study forced physicians and the AI tool to work through the cases step by step, giving additional information only when follow-up questions were asked.
In nonemergency cases, a group of physicians that included internists and emergency medicine physicians achieved accurate diagnoses in 22.3% of cases without using reference tools. When physicians used reference tools, their diagnostic accuracy jumped to 36.1%. The AI system, by comparison, was accurate in 83.2% of cases.
A Medical Economics analysis of the study found that the results were similar for emergency cases. AI was accurate in 80.3% of emergency cases compared to 46.8% for physicians.
Researchers noted that AI greatly outperformed even the highest-scoring individual physicians in both emergency and nonemergency cases. The peak performance by physicians on nonemergency cases was 51.8%. For emergency cases, the best physicians were accurate 62.5% of the time.
The study uncovered differences in how physicians and AI approach patient cases. Physicians, for example, asked an average of 5.6 questions before ordering a test. The AI system, by comparison, asked 25.5 questions before ordering a test.
In addition to reviewing diagnostic accuracy, researchers tried to compare the costs of tests ordered by physicians and AI. The AI system spent $5,431 on tests in nonemergency cases compared to $4,725 spent by physicians. In emergency cases, AI spent $4,979 on testing compared to $3,400 for physicians.
Ambient AI and physician productivity: Annals of Internal Medicine
A second study published in Annals of Internal Medicine found that physicians using ambient AI scribes cut the time they spent taking notes but didn’t see other gains in productivity.
Outpatient physicians using an AI system known as Heidi slashed the time they spent on patient notes from 21 to 18.6 minutes. At the same time, median note length jumped from 4,854 to 5,187 characters.
The study found that reducing the time physicians sped on patient notes didn’t lead to other productivity gains. Researchers, for example, found that physicians didn’t close out patient notes any faster either on the day they saw patients or later.
A MedPage Today analysis of the study noted that some physicians have reported spending an average of three hours working on patient notes on their days off. Some experts hope that tools like ambient AI can help reduce time spent on administrative tasks to not only improve physician productivity, but reduce burnout.
An accompanying editorial suggested that one reason physicians didn’t end up saving more time was that they may have to spend more time on the back end reviewing and editing notes produced by AI.
The editorial hypothesized that the combination of physicians spending less time taking notes and AI producing notes that are longer may produce “downstream consequences” for patient care. And because the technology can produce issues including fabrications, omissions and misinterpretations, the editorial said, regulations may be needed to protect patient care.
