A mathematical model is a simplified description of reality that can be used to solve practical problems, predict future events or explain how different systems work.
During pandemics, mathematical models can be valuable tools for simulating disease transmission and, among other things, forecasting how many hospital beds will be needed. Early in an outbreak, however, there is often considerable uncertainty about how the disease spreads, resulting in uncertain forecasts.
“At the beginning of a pandemic, the amount of data available is very limited. This creates uncertainty in both the models and their assumptions,” says Philip Gerlee, Professor at the Department of Mathematical Sciences at Chalmers University of Technology and the University of Gothenburg.
Combining models produces more reliable results
To investigate how different models perform under different conditions, researchers at the Department of Mathematical Sciences at Chalmers University of Technology and the University of Gothenburg, in collaboration with École Polytechnique in France, generated synthetic data from 324 simulated epidemics.
The epidemics were generated using a complex agent-based model of disease transmission based on Covid-19.
“By varying factors such as the characteristics of the disease and human mobility, we were able to create a wide variety of epidemic scenarios,” says Philip Gerlee.
The tests showed that none of the 14 individual models evaluated performed best in every situation. Their performance varied depending on factors such as how quickly the disease was spreading. The researchers therefore also developed a new method that combines forecasts from different models into a so-called ensemble forecast.
“This allows you to cover a wider range of possibilities rather than relying on a single model,” says Philip Gerlee.
Ensemble methods have previously been shown to produce more reliable forecasts than individual models. What is novel about the new method is that the different models are given different weights depending on how quickly the disease is spreading and how well the models performed under similar conditions in the simulated epidemics.
“With our method, you could say that we listen most closely to the model that has proved to work best under the specific conditions at that point in time, although the other models still have some influence,” says Philip Gerlee.
Of the six different ensemble methods the researchers tested on synthetic data, the new method was the most accurate. It also performed well when tested on real data from the Covid-19 pandemic.
At the same time, the tests showed that a more advanced method does not necessarily produce the most accurate forecast in every situation.
“Although our new method performed best on synthetic data, it was surprising that the simplest possible method for creating an ensemble forecast – taking the median of all the individual model forecasts – performed almost as well, and even better on real data,” says Philip Gerlee.
Differences can indicate uncertainty
The researchers were also able to show that differences between the forecasts produced by different models can provide information about the reliability of the combined forecast.
“By statistically analysing the difference between ensemble forecasts and the actual outcomes, we see that when the individual model forecasts differ substantially, there is, on average, a larger discrepancy between the ensemble forecast and the outcome,” says Philip Gerlee.
The differences between the model forecasts could therefore provide an indication of when uncertainty in the combined forecast is greater – information that could be valuable to decision-makers using the forecast.
“We believe that our results could be useful in future pandemics. Synthetic data can be used to evaluate the suitability of existing models, while the relationship between variability within an ensemble and forecast accuracy can provide important information about the reliability of the combined forecast,” says Philip Gerlee.
Text: Julia Romell
