Experts such as epidemiologists and statisticians were much more accurate than the public in predicting the number of COVID-19 infections and deaths that would occur in the United Kingdom and worldwide, but both groups significantly underestimated the pandemic's true impact, according to a study today in PLOS One.
In April 2020, University of Cambridge researchers surveyed 140 UK experts and 2,086 members of the general public, asking them to make four quantitative predictions about how the pandemic would unfold by the end of 2020. Specifically, they were asked to forecast the number of infections and deaths in the United Kingdom and the worldwide case-fatality rate.
The aim was to determine the accuracy of expert "off-the-cuff" forecasts given to the media, as they may influence public policy and opinion. The experts included 65 statisticians, 44 mathematical modelers, 35 clinicians, 19 epidemiologists, and 1 virologist.
The researchers also asked respondents to indicate certainty by providing the upper and lower bounds of where they were 75% sure that the outcome would occur (eg, a respondent might indicate that they were 75% certain that the number of infections would range from 300,000 to 800,000).
The expert group's predictions fell within their confidence ranges 44% of the time, compared with only 12% of the non-expert group. Even when the non-expert group was limited to participants who had a good grasp of numbers, as determined by a numeracy test, the predictions were only 16% accurate.
Importance of admitting uncertainty
Although the case-fatality rate as of Dec 31, 2020, was difficult to estimate because the true number of infections is unknown, the researchers estimated that 4.55 of 1,000 people worldwide and 11.8 of 1,000 in the United Kingdom had died of their infections. The experts' median prediction of case-fatality rate was 10 of 1,000 people worldwide and 9.5 of 1,000 people in the United Kingdom, while the non-expert group predicted 50 of 1,000 and 40 of 1,000, respectively.
The official UK death toll at the end of 2020 was 75,346, compared with the median expert prediction of 30,000 and the median non-expert forecast of 25,000.
Lead study author Gabriel Recchia, PhD, said in a University of Cambridge news release that the results highlight the difficulty of forecasting outcomes early in the pandemic. "Predicting the course of a brand-new disease like COVID-19 just a few months after it had first been identified is incredibly difficult, but the important thing is for experts to be able to acknowledge uncertainty and adapt their predictions as more data become available," he said.
After all, as the study points out, if experts minimize uncertainty, their confidence may play well to the public in the short term but may undermine trust if their predictions turn out to be wrong.
Recchia added that, although the experts didn't perform as well as the researchers thought they might, they were much more accurate than the non-experts. His words echo the study's conclusion, which reads, "The ultimate message may be that 'the experts have much to learn, but they also have much to teach.'"
Expertise, situation-specific judgment
Recchia said that it's important to discern different kinds of expertise and the kind of judgment needed in a particular situation. "People mean different things by 'expert': These are not necessarily people working on COVID-19 or developing the models to inform the response," he said. "Many of the people approached to provide comment or make predictions have relevant expertise, but not necessarily the most relevant."
In addition, the researchers said, there is a critical difference between research assessing expert predictions and research evaluating epidemiologic models, even if the models informed the experts' forecasts. Many coronavirus pandemic models have proven reasonably accurate in the short-term but not in the long-term because of the challenge of the nonlinear, rapidly changing nature of the processes influencing viral transmission.
"Given the continued impact of COVID-19 and risks of other future pandemics, further research into improved epidemiological forecasting may prove vital," the authors wrote. "In the meantime, we must all learn to acknowledge and admit that the uncertainties may be greater than we think they are, whether we are experts or not."
