A textbook example of shifting the standards of evidence to suit its authors’ needs.
Did masks work to slow the spread of COVID-19? It all depends on what you accept as “evidence.” Credit: Grace Cary
Recently, Congress’ Select Subcommittee on the Coronavirus Pandemic released its final report. The basic gist is about what you’d expect from a Republican-run committee, in that it trashes a lot of Biden-era policies and state-level responses while praising a number of Trump’s decisions. But what’s perhaps most striking is how it tackles a variety of scientific topics, including many where there’s a large, complicated body of evidence.
Notably, this includes conclusions about the origin of the pandemic, which the report describes as “most likely” emerging from a lab rather than being the product of the zoonotic transfer between an animal species and humans. The latter explanation is favored by many scientists.
The conclusions themselves aren’t especially interesting; they’re expected from a report with partisan aims. But the method used to reach those conclusions is often striking: The Republican majority engages in a process of systematically changing the standard of evidence needed for it to reach a conclusion. For a conclusion the report’s authors favor, they’ll happily accept evidence from computer models or arguments from an editorial in the popular press; for conclusions they disfavor, they demand double-blind controlled clinical trials.
This approach, which I’ll term “shifting the evidentiary baseline,” shows up in many arguments regarding scientific evidence. But it has rarely been employed quite this pervasively. So let’s take a look at it in some detail and examine a few of the other approaches the report uses to muddy the waters regarding science. We’re likely to see many of them put to use in the near future.
What counts as evidence?
If you’ve been following the politics of the pandemic response, you can pretty much predict the sorts of conclusions the committee’s majority wanted to reach: Masks were useless, the vaccines weren’t properly tested for safety, and any restrictions meant to limit the spread of SARS-CoV-2 were ill-informed, etc. At the same time, some efforts pursued during the Trump administration, such as the Operation Warp Speed development of vaccines or the travel restrictions he put in place, are singled out for praise.
Reaching those conclusions, however, can be a bit of a challenge for two reasons. One, which we won’t really go into here, is that some policies that are now disfavored were put in place while Republicans were in charge of the national pandemic response. This leads to a number of awkward juxtapositions in the report: Operation Warp Speed is praised, while the vaccines it produced can’t really be trusted; lockdowns promoted by Trump adviser Deborah Birx were terrible, but Birx’s boss at the time goes unmentioned.
That’s all a bit awkward, but it has little to do with evaluating scientific evidence. Here, the report authors’ desire to reach specific conclusions runs into a minefield of a complicated evidentiary record. For example, the authors want to praise the international travel restrictions that Trump put in place early in the pandemic. But we know almost nothing about their impact because most countries put restrictions in place after the virus was already present, and any effect they had was lost in the pandemic’s rapid spread.
At the same time, we have a lot of evidence that the use of well-fitted, high-quality masks can be effective at limiting the spread of SARS-CoV-2. Unfortunately, that’s the opposite of the conclusion favored by Republican politicians.
So how did they navigate this? By shifting the standard of evidence required between topics. For example, in concluding that “President Trump’s rapidly implemented travel restrictions saved lives,” the report cites a single study as evidence. But that study is primarily based on computer models of the spread of six diseases—none of them COVID-19. As science goes, it’s not nothing, but we’d like to see a lot more before reaching any conclusions.
In contrast, when it comes to mask use, where there’s extensive evidence that they can be effective, the report concludes they’re all worthless: “The US Centers for Disease Control and Prevention relied on flawed studies to support the issuance of mask mandates.” The supposed flaw is that these studies weren’t randomized controlled trials—a standard far more strict than the same report required for travel restrictions. “The CDC provided a list of approximately 15 studies that demonstrated wearing masks reduced new infections,” the report acknowledges. “Yet all 15 of the provided studies are observational studies that were conducted after COVID-19 began and, importantly, none of them were [randomized controlled trials].”
Similarly, in concluding that “the six-foot social distancing requirement was not supported by science,” the report quotes Anthony Fauci as saying, “What I meant by ‘no science behind it’ is that there wasn’t a controlled trial that said, ‘compare six foot with three feet with 10 feet.’ So there wasn’t that scientific evaluation of it.”
Perhaps the most egregious example of shifting the standards of evidence comes when the report discusses the off-label use of drugs such as chloroquine and ivermectin. These were popular among those skeptical of restrictions meant to limit the spread of SARS-CoV-2, but there was never any solid evidence that the drugs worked, and studies quickly made it clear that they were completely ineffective. Yet the report calls them “unjustly demonized” as part of “pervasive misinformation campaigns.” It doesn’t even bother presenting any evidence that they might be effective, just the testimony of one doctor who decided to prescribe them. In terms of scientific evidence, that is, in fact, nothing.
Leaky arguments
One of the report’s centerpieces is its conclusion that “COVID-19 most likely emerged from a laboratory.” And here again, the arguments shift rapidly between different standards of evidence.
While a lab leak cannot be ruled out given what we know, the case in favor largely involves human factors rather than scientific evidence. These include things like the presence of a virology institute in Wuhan, anecdotal reports of flu-like symptoms among its employees, and so on. In contrast, there’s extensive genetic evidence linking the origin of the pandemic to trade in wildlife at a Wuhan seafood market. That evidence, while not decisive, seems to have generated a general consensus among most scientists that a zoonotic origin is the more probable explanation for the emergence of SARS-CoV-2—as had been the case for the coronaviruses that had emerged earlier, SARS and MERS.
So how to handle the disproportionate amount of evidence in favor of a hypothesis that the committee didn’t like? By acting like it doesn’t exist. “By nearly all measures of science, if there was evidence of a natural origin, it would have already surfaced,” the report argues. Instead, it devotes page after page to suggesting that one of the key publications that laid out the evidence for a natural origin was the result of a plot among a handful of researchers who wanted to suppress the idea of a lab leak. Subsequent papers describing more extensive evidence appear to have been ignored.
Meanwhile, since there’s little scientific evidence favoring a lab leak, the committee favorably cites an op-ed published in The New York Times.
An emphasis on different levels of scientific confidence would have been nice, especially when dealing with complicated issues like the pandemic. There are a range of experimental and observational approaches to topics, and they often lead to conclusions that have different degrees of certainty. But this report uses scientific confidence as a rhetorical tool to let its authors reach their preferred conclusions. High standards of evidence are used when its authors want to denigrate a conclusion that they don’t like, while standards can be lowered to non-existence for conclusions they prefer.
Put differently, even weak scientific evidence is preferable to a New York Times op-ed, yet the report opts for the latter.
This sort of shifting of the evidentiary baseline has been a feature of some of the more convoluted arguments in favor of creationism or against the science of climate change. But it has mostly been confined to arguments that take place outside the view of the general public. Given its extensive adoption by politicians, however, we can probably expect the public to start seeing a lot more of it.
John is Ars Technica’s science editor. He has a Bachelor of Arts in Biochemistry from Columbia University, and a Ph.D. in Molecular and Cell Biology from the University of California, Berkeley. When physically separated from his keyboard, he tends to seek out a bicycle, or a scenic location for communing with his hiking boots.