Modeling a pandemic is really hard. Who would have guessed?
By Sarah Jacoby – www.self.com
Adobe Stock / Angelina Bambina
It didn’t take long for many of us to realize that the coronavirus pandemic was going to be a deadly force that changed our lives in very significant ways, possibly for a very long time. But according to some major projections, the rates of hospitalization and death due to COVID-19 might be slightly better than experts initially thought.
In late March a frequently cited model from the Institute for Health Metrics and Evaluation (IHME) at the University of Washington’s School of Medicine projected that the U.S. would see about 81,000 deaths—but possibly up to 162,000. By early April the model had shifted somewhat. Although it still predicted that we’d endure around 81,000 deaths, the highest estimate came down to about 136,000. Its estimates of the amount of hospital ICU beds and ventilators we would need were reduced as well.
Does that mean we overreacted in shutting down schools, businesses, and nearly entire cities? Well, no. As it turns out, projections like these are exceedingly difficult to put together and most of them end up being wrong in one way or another. But even if a model doesn’t end up fully reflecting reality, that doesn’t mean it can’t be helpful. A change in projections doesn’t necessarily mean you did something wrong—in fact, it might mean you did something right.
What goes into creating a model of infectious disease spread?
Short answer: a lot.
The long answer? Well, get ready. Essentially there are a few major types of models that researchers can create, Jeffrey Shaman, Ph.D., professor of environmental health sciences and director of the Climate and Health Program at Columbia University Mailman School of Public Health, tells SELF.
The first kind is a mathematical model, which describes the complex transmission process within some sort of construct, like how people in a city will become infected with the new coronavirus, says Shaman, who has been leading Columbia University’s work in creating COVID-19 models.
In some types of mathematical models, called agent-based models, researchers are able to take into account many different factors that have effects on each other. That means the model measures different individual “actors” that go to work, go shopping, et cetera, and calculates how their disease status—whether they’re infected or not—will change over time based on who else they come in contact with and what environments they go into.
Mathematical models like this are “computationally expensive,” Shaman says, and they have to make a fair amount of assumptions about people’s behavior and how the virus works that may or may not actually be true.
There are simplified versions of mathematical models, called compartmental models, that researchers might use in a case like this. One very common one is an SI or SIR model, which estimates the number of susceptible, infected, and recovered individuals in a particular situation over time, Shaman explains.
With this type of model, you try to measure “the rate as people move between the different compartments from being susceptible, to being infected, to being removed from the population because they’re recovered or dead,” Amesh A. Adalja, M.D., senior scholar at the Johns Hopkins Center for Health Security, tells SELF.
For example, a model published in the International Journal of Infectious Diseases by scientists in the U.S. and China, takes a SEIR (susceptible, exposed, infected, removed) approach to modeling the outbreak in Wuhan. And, in a study currently in preprint, Sherman and his coauthors used a dynamic metapopulation model, which functions sort of like a network of compartmental models, to examine the role that those with mild or asymptomatic infections had on spreading the outbreak in China.
The other major type of model is a statistical model, which creates a projection of what the situation might look like at some point in the future based on data we’ve already collected about what happened in the past. The oft-cited COVID-19 model created by the IHME is one such statistical model used to predict the need for hospital equipment as well as the rate of deaths due to the new coronavirus in the U.S. and across the world.
All of these models have to take into account different factors about the virus and the people it infects, such as how quickly the disease spreads, how many people each infected person goes on to infect, and how long the incubation period is, says Dr. Adalja, whose work involves assessing pandemic preparedness. But at the beginning, those are just assumptions—and we may not know how accurate those assumptions are for quite a while. “All of these models are based on certain assumptions that need to be refined as the outbreak unfolds,” he says.
It’s really, really hard to create a model for a new virus that’s spreading in real time.
Creating a model for the spread and effects of an infectious disease always takes a lot of time and complicated guesswork. But situations like this coronavirus present some very unique challenges that make it even harder to create accurate projections about what could happen.
Take the seasonal flu, for example. Although this is an infectious disease event happening at the same time researchers are trying to create statistical projections around how severe that particular flu season will be, our treatments and prevention practices don’t change that much from year to year, Shaman says. That makes it easier to create a more accurate model of how the flu season will go.
But in the case of the new coronavirus, “we have to assume what society is going to do,” he says, including when social distancing orders are given, how well people follow them, and when people start to go back to work.
Other major challenges have to do with the testing process, Shaman says. We know that there’s a window of time—up to 14 days in the vast majority of cases—between when someone is infected and when they start experiencing symptoms that lead them to get tested. So when looking at test results, “we’re seeing what happened about two weeks ago,” he says, not the result of any new policy changes implemented in the past few days, for instance, and definitely not what’s actually happening right now.
The availability of tests and when people decide to seek testing plays a role too. For instance, earlier on in the outbreak, someone with a mild cold may not have felt it necessary or even thought to get a test for COVID-19. But later on, with this virus top-of-mind for pretty much everyone, it’s much more likely that someone with even mild symptoms will seek out testing. Also, if the demand for tests is high, but there aren’t enough tests for everyone to get one, that doesn’t necessarily give you a full picture of the rate of positive tests. What’s more, not all states report the amount of negative tests they get.
All of these variables help give researchers a better picture of the true number of cases out there and how they’re spreading—and all of them are changing pretty much constantly. That has real-world effects on how governments, hospitals, and individuals prepare for the pandemic.
For example, down the line, testing issues have made it difficult to accurately interpret the data we have and estimate the hospitalization rate. Initially, data from other countries (like Spain) indicated that a very high number of people who were infected with the new coronavirus would require hospitalization, Dr. Adalja says. And according to CDC data, the hospitalization rate was initially quite high (over 30% for some age groups). But more recently the hospitalization rate in the U.S. has been much lower. So why were the initial projections wrong?
“We know we’re undercounting a number of cases because of testing constraints,” Dr. Adalja says. And if that’s the case, then “that means we’re overestimating the hospitality and fatality ratios.”
Getting these numbers as right as possible is incredibly important if you’re, say, a hospital planner. That number may tell you that you’re “going to need this many hospital beds, this many ICU beds, this many ventilators,” Dr. Adalja says. “And that may be wrong if your hospitalization rate number is overestimated.”
Models like these were designed to change as time goes on.
As we learn more about the disease, as local social-distancing policies are put into place, and as we see how people actually behave, it’s completely normal for the projections to change.
“Remember that models are still not a substitute for real-world data. They’re tools used by policymakers to think about different ranges of scenarios,” Dr. Adalja says. “They’re not ironclad; you should expect models to change as more data comes out.” In fact, most models end up being wrong for one reason or another, he says. It’s also important to remember that all models have an area of uncertainty, or a range of possible outcomes, not just one specific outcome, Shaman says. And the further we look into the future, the more uncertain the projected outcomes are.
Unfortunately, when you come across a news story or just a tweet that talks about a model of the impact of the coronavirus, it’s simply not going to be able to take in the full complexity of the model’s details or the data the researchers were working with, Dr. Adalja says. The nuance and assumptions about the model as well as the many possible outcomes involved are often lost in translation.
But sometimes, like in this case, the reason why models change is actually encouraging. “When people are talking about flattening the curve, that is something that is happening because of human intervention,” Shaman says. It’s not a treatment or a vaccine, but it’s something that people—human beings like you and me!—are doing that has a real impact on the course of the outbreak and the projections researchers make for our future.
Those early models did take social distancing measures into account, but doing so is not as easy as just adding one more number to the equation. You have to think about when the orders are put into place, whether they are a true order or just a suggestion, and how well people will really follow those orders. In a series of simulation charts created by the Washington Post using data from researchers at Johns Hopkins University Center for Systems Science and Engineering, you can see that strict social distancing has a much more significant effect on the curve than a half-assed attempt, so we always knew it would be helpful.
But accounting for social distancing and estimating its true power on the curve has been a bit of a challenge, and even the IHME model saw updates early on to its social-distancing metrics that have made it an ever more complicated measure. For instance, to determine the effects of social distancing in the IHME model, researchers now actually combine the results of several other models based on estimates of three social-distancing measures (school closures, stay-at-home orders, and nonessential business closures). They then use each of those values to create both short- and long-term death rate models.
Some people might see the differences in projections after those changes to the model and interpret them as a sign that our social distancing and closing of nonessential businesses was an overreaction. But that’s the wrong conclusion to draw. If anything it’s a sign that the social distancing has been successful—perhaps even more so than the original models projected. In fact, as Shaman says, that’s what it means to “flatten the curve.”
So what should you take away from these models? Know that researchers across the country and the globe are working hard to find the answers that will keep as many of us as safe as possible. They can use models to project what the future will be like and what preparations we need to make. Once we see those models, the way we act on that information will of course have an effect on the projected outcome. It’s a nice reminder that, even in a pandemic that makes us feel helpless most of the time, many of us can still do something: stay home.
Sarah Jacoby is a health and science journalist and is especially interested in the science of skin care, sexual and reproductive health, drugs and drug policy, mental health, and helping everyone find their personal definition of wellness. She’s a graduate of NYU’s Science, Health, and Environmental Reporting Program and has…