:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/72001330/1239913973.0.jpg)
Here’s a seemingly simple question with a deeply unsimple answer: Do the masks that so many of us spent more than two years wearing actually prevent the spread of Covid-19? There have been dozens of studies trying to answer that question, and often they point in different directions, or are too small to find the effect they’re looking for.
With something as complex as masks and respiratory disease, the right tool for the job isn’t one study but many: a meta-analysis of a number of different smaller studies to determine what effect is detectable in aggregate across the body of research.
For medical questions, a major source of high-quality, reputable meta-analyses is Cochrane, a UK-based nonprofit that publishes long and comprehensive meta-analyses of current evidence on medical and therapeutic interventions. It’s a good place to go if you’re wondering if antidepressants work, if blood pressure medications help, if therapy does anything, and lots more. Cochrane reviews are frequently called the “gold standard” for evidence-based medicine.
But there are a lot of methodological decisions that go into a meta-analysis, and Cochrane brings its own set of assumptions to that table: they tend to exclude many studies as low-quality, and tend to be conservative in declaring that an effect exists.
A few weeks ago, a Cochrane team published “Physical interventions to interrupt or reduce the spread of respiratory viruses,” their latest review of the evidence for masking and hand-washing, and whether they reduce flu-like illness (including Covid-19). Thanks to Covid, the debate over how well masks work against respiratory illness has gone from a niche disagreement to a question on which everyone in the country now has an opinion. It’s a question ripe for a good, rigorous meta-analysis.
What we know about masks and Covid
Nearly all of us wore masks at one time or another, and many of us still do. Did that even help?
It doesn’t seem like it, according to the Cochrane review: “Wearing masks in the community probably makes little or no difference to the outcome of laboratory-confirmed influenza/SARS-CoV-2 compared to not wearing masks.”
Some observers have taken this as a final, authoritative conclusion on the matter, an opinion shared by the man behind the review. “There is just no evidence that they make any difference. Full stop,” Tom Jefferson, the study’s lead author, said in an interview. Even fitted N95 masks in health care settings, the interviewer asked? “It makes no difference — none of it.”
I think Jefferson — an Oxford University epidemiologist who has a number of eccentric and flatly nonsensical opinions about Covid-19, including that it didn’t originate in China and may have been circulating in Europe for years before its global emergence — is overstating his case. There is something we can learn from the Cochrane paper, but it’s as much about the process of science as it is about the effectiveness of masks.
First, the reasons I don’t totally buy the Cochrane review’s conclusions:
The review includes 78 studies. Only six were actually conducted during the Covid-19 pandemic, so the bulk of the evidence the Cochrane team took into account wasn’t able to tell us much about what was specifically happening during the worst pandemic in a century.
Instead, most of them looked at flu transmission in normal conditions, and many of them were about other interventions like hand-washing. Only two of the studies are about Covid and masking in particular.
Furthermore, neither of those studies looked directly at whether people wear masks, but instead at whether people were encouraged or told to wear masks by researchers. If telling people to wear masks doesn’t lead to reduced infections, it may be because masks just don’t work, or it could be because people don’t wear masks when they’re told, or aren’t wearing them correctly.
There’s no clear way to distinguish between those possibilities without more original research — which is not what a meta-analysis of existing work can do.
Digging into the research on masks and Covid
Those studies that did take on Covid and masks directly often painted a different picture than the broader conclusions from the meta-analysis.
One of the largest studies of mask-wearing during the Covid pandemic was conducted in Bangladesh, with more than 170,000 people in the intervention group and similar numbers in the control group. The authors studied a series of public announcements and mask distributions which raised the frequency of mask-wearing. In the end, around 40 percent of the experimental group wore masks, compared to around 10 percent in the control group.
The result, the study found, was a substantial reduction in the share of people with Covid-19-like symptoms, and in antibodies that would suggest a Covid-19 infection: “In surgical mask villages, we observe a 35.3% reduction in symptomatic seroprevalence among individuals ≥60 years old ... We see larger reductions in symptoms and symptomatic seropositivity in villages that experienced larger increases in mask use.”
That looks like pretty substantial evidence that mask-wearing reduces Covid-19! And this article is one of only two studies of mask-wearing included in the Cochrane review which happened during the Covid-19 pandemic. The other, a study in Denmark, assigned people to wear masks (though, of course, not all of the people told to wear masks did so consistently or correctly) and had a control group that generally did not wear masks. The group that was told to wear masks had slightly lower infection rates than the group that didn’t wear masks, but the sample was too small for the effect to be significant.
Given that — one study finding very solid evidence for the benefits of masks, and one finding limited but encouraging evidence — how did Cochrane arrive at its conclusion that mask wearing “probably makes little or no difference?” Because their meta-analysis mixes these studies with many more pieces of research that were conducted before Covid-19 and found little effect of masks on the transmission of other illnesses like influenza.
Science is full of judgment calls
The new Cochrane review paper strikes me, and may strike you, as something of a scientifically irresponsible way to represent these findings. It gets at one of the core challenges of science: There is no methodology that can straightforwardly find answers in messy study data without many judgment calls by scientists, who are humans with their own strength, weaknesses, and eccentricities. A meta-analysis, after all, can’t meta-analyze itself.
“I was really surprised that the Cochrane group let this go through,” Jake Eaton, a public policy and global health researcher who was the lead researcher on a Cochrane review of childhood nutrition, told me. “The fact it’s looking at masks across different settings and with different diseases makes it really tricky. Cochrane reviews are very good if you really want to assemble the most rigorous evidence and say, ‘Do we have a conclusive signal that this works?’ This is something of a perverse use of a Cochrane review.”
The big problem is the sheer scope of the question: not “does masking reduce Covid transmission during a pandemic?” but “does masking help against all respiratory illness across contexts?” Because of the questions it chooses to consider and the ways it chooses to aggregate its evidence, it has arrived at a mask-skeptical conclusion with limited applicability to the hot-button question we all care about: whether there’s evidence for masking during high-transmission periods in the Covid-19 pandemic.
We want science to be objective and impartial, for scientists to get the same answers regardless of their starting worldviews. On some questions, like the efficacy of antibiotics against bacterial infections, the evidence really is overwhelming. But on some, like masks, it’s limited.
And I do think the government bears some responsibility for poor communication about masks: this is an intervention where evidence is limited and suggestive, not a sure thing. But I’m also frustrated by the way this quite bad meta-analysis has been seized on as proof that masks don’t work.
How you interpret limited evidence depends substantially on where you start. A meta-analysis is a useful tool, but the results it gets will be substantially informed by how the question is posed and which research is included in answering it. There often aren’t easy answers even to seemingly easy questions, and there sometimes aren’t any answers — even to questions that we care about deeply and that have been studied in some depth.
A version of this story was initially published in the Future Perfect newsletter. Sign up here to subscribe!
