Statistics from Altmetric.com
Meta-analysis using individual participant data (IPD) is becoming increasingly popular, despite being a laborious and resource-intensive method of evidence synthesis compared with a standard review using study-level data.1 It has the potential to overcome the limitations of meta-analyses based on published data through access to raw trial data,1–4 such as standardisation of analysis methods and data across trials1 5 (table 1). Access to IPD can facilitate integrity checks and intention to treat analysis by imputing for missing data. Collation of rarely reported variables for the key outcomes can result in greater precision of the intervention effect and address the problem of selective reporting.1
Existing methodological literature focuses mainly on cost, team’s expertise and management of the collaboration.3 Yet, not much is available on practical challenges associated with data harmonisation and their consequences for IPD meta-analyses. The aim of this article is to discuss some of the opportunities created by access to IPD and the limitations of meta-analysis using IPD as indicated in table 1. We use the i-WIP IPD meta-analysis of 36 trials (12 526 participants from 16 countries; 50 investigators) on the effect of diet and physical activity-based interventions in pregnancy6 as an example (online supplementary appendix 1).
Standardisation of data across trials
Access to IPD should create a unique opportunity to unify all essential data. This is true assuming that collected data can be brought to the same format without losing their value. Routinely collected data such as age, weight or height tend to be captured as real values making them relatively easy to harmonise. Participant characteristics recorded in other formats or those less routinely collected can be much more challenging to standardise. One of the subgroups of interest in the project was maternal ethnic origin.7 The characteristic was available for 47% (17/36) trials of which one differentiated only between indigenous and non-indigenous women, four classified women only as Caucasian or non-Caucasian and eight declared to include only Caucasians or not recognise ‘ethnicity’ in their country. The characteristic was grouped into six categories (Caucasian, Asian, Afro-Caribbean, Central and South American, Middle Eastern and other and unknown) but due to a low proportion of women from groups other than Caucasian (>80% of included women) in the analysis of differential effects of intervention by ethnic origin, the characteristic was used in the binary format (Caucasian/non-Caucasian).6
Harmonisation of outcome definitions faced similar challenges. While some definitions are relatively easy to bring to a common format across the trials, for example, preterm birth, standardisation of others was simply not feasible. The task can be even more daunting when there is no consensus on classification methods, or the definitions changed over the years. Despite access to IPD, direct communication with the research teams and the idea endorsement by the members of the i-WIP collaborative group, standardisation of outcomes such as gestational diabetes (GDM) or caesarean section turned out to be unachievable within the study funding time. Diagnosis of GDM was based on a broad range of guidelines that followed algorithms that did not always overlap with each other. We have made an attempt to standardise the definitions of GDM and collected the blood test measurements used to diagnose the condition. However, the variability in glucose loads (50, 75 or 100 g) and tests’ timing (fasting, 1 hour or 2 hours) leads us to abandon this task and acknowledge the variability in the outcome definition as a limitation. The variety of GDM definitions and the blood test measures, as well as the coding of participants’ ethnic origin in the trials with diet and/or physical activity in pregnancy, is presented in online supplementary appendix 2.
Selective reporting of intervention effects depending on statistical significance is one of the most important sources of bias affecting clinical trials.8–11 Despite clear guidance on reporting of outcomes in the trial reports,12 the problem persists, having a serious impact on the meta-analysis. In combination with variation in choice of trial outcomes,13 they are contributing to the serious waste of research efforts. More frequent reporting of statistically significant results can lead to a potential overestimation of underlying treatment effects in a meta-analysis when using data extracted from trial publications. IPD meta-analysis has the potential to address this problem through facilitating analysis of core outcome sets,14 if available in trial datasets but not reported in publications.
Access to individual records should increase the number of trials included in the analysis and enhance the quality of outcome data. However, the benefits may not always be substantial. In the i-WIP project, the number of trials with the outcomes of interest was higher through access to IPD in comparison with data extracted from publications (online supplementary appendix 3). In addition, use of the raw data to generate outcomes not considered in original trials (eg, use of gestational age at delivery to define the occurrence of prematurity) may lead to a substantial increase in the number of the trial that can be incorporated into the meta-analysis (table 2). Nevertheless, the presence of data in the dataset did not always allow to incorporate a given dataset in the statistical analysis. Too few events (eg, stillbirths) and lack of all measures (baseline and final for weight gain) prevented trial inclusion. Still, in the example, incorporation of trials with previously unavailable outcome data changed the value of the effect estimate by more than 10% in three outcomes and its statistical significance in one (table 2).
The addition of unreported data may or may not lead to a change in funnel plot asymmetry. In the example, incorporation of unpublished outcomes in the meta-analysis for admission to neonatal intensive care unit has not revealed any potential bias. Similarly, for small for gestational age infant where outcome data were generated using raw data, if the outcome was not considered in original trials (table 2). For continuous outcomes (gestational weight gain in the example), the change in the plot asymmetry might also occur due to the standardisation of the analysis methods rather than incorporation of unreported data (figure 1).
Role of IPD meta-analysis in dynamic research areas
The authors of guidance on the appraisal of IPD meta-analyses of randomised trials advocate checking for the proportion of trials from which IPD was obtained.5 A recent study showed that only 25% of evaluated IPD meta-analyses obtained 100% of identified trial data.15 Acquisition of all eligible trials can be challenging for numerous reasons, with uncooperative trial investigators mentioned most commonly.5 IPD meta-analysis is a lengthy and resource-intensive process which can also decrease the chance of complying with the above-mentioned recommendation.
Since the publication of the systematic review that laid the grounds for the IPD meta-analysis we used as an example,16 there has been a significant increase in the number of trials evaluating the effects of diet and/or physical activity-based interventions in pregnancy. Between the end of data acquisition in June 2015 to February 2017, findings from additional 45 trials have been published (figure 2) making achieving the goal of being up-to-date and obtaining the majority of IPD virtually impossible.17 In combination with the trials for which IPD was sought but not obtained, the number of trials outside the IPD meta-analysis (non-IPD studies) constituted 65% of trials (67/103 trials) and 51% of women randomised to all eligible trials (12 960/25 486). The meta-analysis combining IPD with non-IPD studies showed a stronger overall effect of interventions in reduction of gestational weight gain and a significant reduction of odds for GDM than one using only IPD.17
Despite the advantages of meta-analysis using IPD, the method encounters problems faced by other research methods such as uncooperative investigators or incompleteness of records. The IPD meta-analysis is a resource-demanding approach to evidence synthesis and requires a thorough evaluation of what is achievable. It might be that we will need to accept that some primary research is not usable for evidence synthesis. Mapping of definitions and additional data that could help to standardise the outcome across the trials may not tackle all the issues but will facilitate the smoother conduct of IPD meta-analyses. The efforts associated with obtaining IPD and its harmonisation need to be balanced by the potential gains achievable through a complex and profound statistical analysis. Prospectively designed IPD meta-analyses have the potential to overcome some of the challenges described in this article as they tend to collect data in a preagreed format.18 Promotion of consensus on the research standards with regard to outcome definitions, capturing of participants’ characteristics and effective ways of implementing them in the trials should help to reduce the potential research waste. Finally, putting the findings of IPD meta-analysis into a context of the totality of evidence is paramount for the validity of results.5 Currently, guidelines recommend adding non-IPD studies to IPD meta-analysis when a substantial proportion of trials IPD was not obtained at the beginning of the project. In addition, in some areas of medical research, the amount of evidence generated annually makes it difficult to stay up-to-date while conducting IPD meta-analysis. Therefore, adding newly published trials is as important as incorporating the not shared ones.
We wouldlike to acknowledge all members of Collaborative Group who contributed theirtrial data to the i-WIP IPD meta-analysis funded by The National Institute forHealth Research (NIHR) Health Technology Assessment (HTA) programme (No.12/01/50).
Twitter Follow Ewelina Rogozinska @EaRogozinska
Contributors ER wrote the initial draft of the manuscript and all subsequent drafts after critical review by JZ, NM and KSK. ER is a guarantor for the manuscript.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.