Health Editor
A major new study has issued a stark warning about the use of AI chatbots to give medical advice, revealing that ordinary people using the latest “doctor bots” could still fail to spot serious illnesses or make the right decisions — despite the bots themselves scoring top marks in medical exams.
The news comes as the government announces greater use of AI in healthcare as part of its ten year plan.
In a world-first trial, 1,298 members of the public were asked to tackle ten common health scenarios – like chest pain or abdominal problems – using either a leading AI chatbot (such as ChatGPT’s GPT-4o) or traditional sources like Google or the NHS website.
The bots, when tested alone, performed impressively – correctly identifying medical conditions 95% of the time. But when real people used them for help, the results were alarmingly poor. The diagnosis was only correct in just 34.5 percent of cases and most of the time – 56 percent of cases – an incorrect decision about what to do (such as going to A&E or staying home) only was made.
The failure, say the researchers from major academic institutions, lies not with the bots’ medical knowledge but with how they interact with humans. Users often gave the bots incomplete or vague information, while the bots, though technically correct, failed to clearly explain what to do next.
The results raise serious concerns for the government and tech firms promoting AI in healthcare.
Robert Dingwall, a professor in social sciences at Nottingham Trent University said: “This is only one study but it is a useful reminder that what works in the models, simulations and imaginations of tech developers rarely transfers as successfully to real life. The Department of Health and Social Care should take care not to bet the farm on fools’ gold.”
And Professor Carl Heneghan, director of Oxford University’s Centre of Evidence Based Medicine said: “Doctors have to train for ten years to be consultants, developing the experience and expertise to recognise serious and life-threatening illnesses from the mundane. While AI has a role in areas such as interpreting X-rays and ECG, it is no replacement for a thorough history and examination when it comes to diagnosing disease. The widespread rollout of untested AI can waste resources and, as this study shows, harm patients seeking a diagnosis.”
Last week the government published its 10-Year Plan to shift the NHS from treating illness to preventing it — a strategy heavily reliant on digital tools, apps and AI to empower patients.
In its manifesto, published this month it stated: “The plan will bring it (the NHS) into the digital age, making sure staff benefit from the advantages and efficiencies available from new technology….The government will also use digital telephony so all phone calls to GP practices are answered quickly. For those who need it, they will get a digital or telephone consultation the same day they request it.”
But this research shows the gap between AI’s performance in the lab and its use in the real world. The authors warn that current benchmarks are misleading, and that AI tools must be tested not just on knowledge, but on how well they communicate with non-experts.
“Just because a chatbot can pass the doctor’s exam doesn’t mean it can help you when you’re sick,” said one of the study leads. “It’s like giving someone a stethoscope and expecting them to perform heart surgery.”
The researchers say future AI tools must be far more proactive — asking clear, guiding questions and actively managing the conversation, instead of relying on users to know what details are medically important.
The findings echo earlier research showing that even trained doctors didn’t get better at diagnosing patients with AI help. Now we know the same is true for the general public.
Experts are calling for rigorous user trials before deploying AI in healthcare, especially when it comes to direct patient advice. Without this, there’s a real danger that people could be lulled into a false sense of security, putting off seeing a doctor — or rushing to A&E unnecessarily.
A Department of Health and Social Care spokewo said: Through our 10 Year Health Plan, we’re slashing bureaucracy across the health service, reducing burdensome administrative tasks and making use of technology so doctors can spend time on what they do best – caring for patients. This includes rolling out AI scribes to end the need for clinical notetaking, letter drafting, and manual data entry so clinicians can focus on treating patients, saving the same time as adding 2,000 more doctors into general practice.
“We have also already reduced the amount of repetitive mandatory training resident doctors are required to do and alongside delivering the second above inflation pay increase in a row this year, we have been listening to doctors to make their working lives better. There’s more to do, but the NHS has been making good progress on small changes which have an outsize impact.”