The AI classifiers trained on audio recordings cannot accurately predict whether someone has Covid-19 by analysing the sound of their coughs, according to the study led by the UK’s Alan Turing Institute.
As first reported in a paper from researchers led by the Massachusetts Institute of Technology, there were claims that AI could detect the difference in cough sounds between those with and without Covid-19 with up to 98.5 per cent accuracy, reports The Register.
The result led to efforts to build an app powered by the algorithms to provide people with a cheap and easy method to test for the novel coronavirus.
The Department of Health and Social Care in the UK even went so far as to award Fujitsu two contracts totalling more than 1,00,000 pounds to develop the government’s so-called “Cough In A Box” initiative in 2021, according to the report.
The software would collect audio recordings of coughs from users to analyse on its Covid-19 app.
Researchers from The Alan Turing Institute and Royal Statistical Society, commissioned by the UK Health Security Agency, conducted an independent review of audio-based AI tech as a Covid-19 screening tool.
They found that even the most accurate cough-detecting model outperformed a model based on user-reported systems and demographic data like age and gender.
The researchers examined data from over 67,000 people recruited through the National Health Service’s Test and Trace and REACT-1 programmes, which asked participants to send back Covid-19 nose and throat swab test results as well as recordings of them coughing, breathing, and talking.
In an attempt to see whether coughs could be used as a biomarker, the researchers trained an AI model based on audio recordings and test results, said the report.
“But as we continued to analyse the results, it appeared that the accuracy was likely due to an effect in statistics called confounding — where models learn other variables which correlate with the true signal, as opposed to the true signal itself,” Kieran Baker, a statistics PhD student at King’s College London and research assistant at the Alan Turing Institute, was quoted as saying.
The confounding was caused by the Test and Trace system’s recruitment bias, which required participants to have at least one symptom in order to participate. The researchers conducted additional tests, pairing participants of similar ages and genders, with only one having Covid-19, the report mentioned.
“When we evaluated these models on the matched data, the models failed to perform well, and so we conclude that our models cannot detect a Covid-19 bio-acoustic marker from this data,” Baker said.
