Be very optimistic about the vaccine efficacy...

Statistical inference in the Covid era...

This article was originally published on 12/12/2020 for my informal email list. I’m re-posting some of my earlier newsletters here as a gradual process to shift to Substack.

In light of last night’s news that the FDA approved the Pfizer-BioNTech vaccine, I want to write a quick note about the efficacy rate of Covid vaccines. We’ve been hearing these highly optimistic numbers like 94.5% for Pfizer – is the vaccine really as effective as it appears? The punchline is yes, and you should be extremely optimistic, at least from a statistical inference point of view. 

(I’ve always struggled to balance between being sufficiently technical with my explanations and not overwhelming you with detailed definitions. If you would like to learn more or need clarification, please simply let me know). 

Basic set-up of vaccine trials:

What happens is vaccine trials first give people shots of either the actual vaccine (treatment group) or placebo (no therapeutic value, control group). You then release these people back to live their normal lives and wait for people to contract Covid, exhibit symptoms, and get a positive test. These people then report back, and based on the number of Covid positives and number of people receiving the vaccine, you can arrive at an efficacy rate. 

(Vaccine efficacy is the % reduction in disease incidence in a vaccinated group compared to an unvaccinated group under controlled conditions, whereas effectiveness is its ability to prevent the disease in the real world.) 

Data from the Pfizer trial: 

In the case of Pfizer’s interim results announced early November, the company recruited 43,661 volunteers, 38,955 of whom received two injections of the vaccine. They waited for 94 people to exhibit symptoms and test positive for Covid-19. Out of these 94, only 8 had received the real vaccine, and the rest had received a placebo shot. 

Pfizer essentially says that their vaccine prevented Covid-19 symptoms for over 90% of those who received the vaccine compared with placebo. Or in other words, only fewer than 10% of infections were in participants who had been given the vaccine. 

But now you may ask – the effective sample size used to calculate the efficacy rate is just not that large (8 out of 43,661 total participants is 0.018%), how can we be certain that it’s truly effective? 

Efficacy explained through coin flip: 

That above question is a classic estimation problem in econometrics – namely how do you make sure that your effect is actually statistically significant, especially when you’re running your experiments with such a small sample. What my econometrics professor told me is that even the most pessimistic person should be encouraged by the 94.5% efficacy rate because the estimation error isn't that big, assuming that the experiment was done correctly. Here’s the simplified logic: 

Say you take the sample of 94 Pfizer trial participants who tested positive with Covid, and let’s treat each person as a coin flip: the tail is essentially “Covid positive + vaccinated” and heads is “Covid positive + NOT vaccinated.”

If the vaccination didn't have any effect, after we flip the coin 94 times, we should theoretically have a 50-50 divide between heads and tails. In other words, if the vaccine is useless (meaning the game is not interfered with), then a Covid-positive person (our coin) should have an equal likelihood showing up as vaccinated (tail) or not vaccinated (head). 

However, after flipping the coin 94 times, we observe 8 tails and 86 heads. Because the probability of getting 86 heads in a row after 94 flips is so extremely low, any reasonable person would shout that this game was not a fair, independent game, so there must indeed be some interference and external effect that made only 8 tails appear. This interference, my friends, is the efficacy of the vaccine. 

The Pfizer trial, after all, only had 8 people as the effective sample size. But what I showed above is that the vaccine has been so effective such that you wouldn’t need to care about the small sample size anymore. In other words, the result is just so statistically significantly strong such that even the small sample shouldn’t worry you, at least theoretically. 

Challenges with testing small samples:

Unfortunately, this Pfizer trial really doesn’t represent the grave challenges confronted by most research projects in the age of Covid – which is that the sample size is too small. 

Suppose I want to know how testing positive for Covid affected Asian men in Michigan (and say Asian men take up 0.5% of all positive cases there), that is a really small effective sample for a much larger data set! How can we zoom in and isolate the actual effect on the small group of people that were actually affected? You can add controls and interaction terms in your regressions and do other adjustments, and maybe you’ll get some statistically significant results, but it’s really hard to say that you’ve truly isolated out the effect, right? Does that regression coefficient tell you the exact effect of that outcome of your interest?

The Pfizer vaccine trial is an entirely different set up, but for most social science research today that relies on these big public data sets with small effective samples, researchers are in general having a fairly difficult time asking narrow questions, and that is why you should take most of the working papers and studies reported in the news with a grain of salt. 

I can go into more detail about this topic on another occasion. If you’re interested in learning more about why I am so skeptical, let me know and I can explain more. 

I could be wrong with some of the details above, so if anyone has more information to add, please let me know and I’ll send out corrections.