Table 2

Some measures to consider for improving the reliability of evidence from non-randomised studies of COVID-19 vaccine effectiveness

Measures to considerRationaleChallenges
Overarching measures
Registration of studies and analysis plansAllows to know what studies and analysis plans were preconceived and adhered to original plans and reduces degrees of freedom for data dredgingMost observational studies are non-registered or are registered after the analyses are done; there is debate on whether retrospective designs should/could be meaningfully registered; analytical plans are rarely registered in sufficient detail
Sharing of raw data and codeAllows independent validation of analyses and optimises the use of the data in overarching syntheses of data from multiple studiesSharing has been limited for various reasons (privacy, consent and legal issues, as well as reluctance of primary investigators)
Better data collection
Background collection of reliable information on seroprevalence, exposures, testing, disease risk factors, risk profiles on hospital admission and use of treatmentsAllows for better adjustments and exploration of effect modificationSome of this information may be biased or very difficult to collect reliably
Blinded assessment of outcomes, for example, death causesAllows removing some outcome misclassification biasesBlinding records requires time and resources and a committed effort
Better designs
Use of maximal/best information in properly matched studies, multivariable analyses, propensity analyses and other modelsDesigns that consider and hopefully address more biases are betterObservational studies are unlikely to ever eliminate all possible biases
Performing randomised trials, whenever possible, for suitable questions (eg, use of booster doses, comparative effectiveness of different vaccination strategies)Removes many of the biasesReluctance to perform randomised trials when data suggest large efficacy (but this may be less of a concern for comparative effectiveness), randomised trials also have biases
Systematic review
Living reviews and meta-analysesProvide bird’s eye view of evolving evidenceMeta-analyses have their own, long list of biases
Better communication
Use of both relative and absolute metrics of risk reduction and presentation of uncertaintyAllows better comprehension of the magnitude of the benefitPoor ability of many/most people to understand risks and other quantitative metrics
Avoidance of exaggeration in communicating results to the general publicMinimises misconceptions, confusion, panic (eg, from misleading claims of loss of vaccine effectiveness) or dangerous behaviour changes (eg, from misleading claims of retaining high effectiveness even with high exposures)There is an avid market seeking immediate information on what is new on the pandemic and vaccines and sensationalism is prominent; the anti-vax movement makes confusion worse by adding extra misinformation