Standardisation of data across trials

Access to IPD should create a unique opportunity to unify all essential data. This is true assuming that collected data can be brought to the same format without losing their value. Routinely collected data such as age, weight or height tend to be captured as real values making them relatively easy to harmonise. Participant characteristics recorded in other formats or those less routinely collected can be much more challenging to standardise. One of the subgroups of interest in the project was maternal ethnic origin.7 The characteristic was available for 47% (17/36) trials of which one differentiated only between indigenous and non-indigenous women, four classified women only as Caucasian or non-Caucasian and eight declared to include only Caucasians or not recognise ‘ethnicity’ in their country. The characteristic was grouped into six categories (Caucasian, Asian, Afro-Caribbean, Central and South American, Middle Eastern and other and unknown) but due to a low proportion of women from groups other than Caucasian (>80% of included women) in the analysis of differential effects of intervention by ethnic origin, the characteristic was used in the binary format (Caucasian/non-Caucasian).6

Harmonisation of outcome definitions faced similar challenges. While some definitions are relatively easy to bring to a common format across the trials, for example, preterm birth, standardisation of others was simply not feasible. The task can be even more daunting when there is no consensus on classification methods, or the definitions changed over the years. Despite access to IPD, direct communication with the research teams and the idea endorsement by the members of the i-WIP collaborative group, standardisation of outcomes such as gestational diabetes (GDM) or caesarean section turned out to be unachievable within the study funding time. Diagnosis of GDM was based on a broad range of guidelines that followed algorithms that did not always overlap with each other. We have made an attempt to standardise the definitions of GDM and collected the blood test measurements used to diagnose the condition. However, the variability in glucose loads (50, 75 or 100 g) and tests’ timing (fasting, 1 hour or 2 hours) leads us to abandon this task and acknowledge the variability in the outcome definition as a limitation. The variety of GDM definitions and the blood test measures, as well as the coding of participants’ ethnic origin in the trials with diet and/or physical activity in pregnancy, is presented in online supplementary appendix 2.

Unreported outcomes

Selective reporting of intervention effects depending on statistical significance is one of the most important sources of bias affecting clinical trials.8–11 Despite clear guidance on reporting of outcomes in the trial reports,12 the problem persists, having a serious impact on the meta-analysis. In combination with variation in choice of trial outcomes,13 they are contributing to the serious waste of research efforts. More frequent reporting of statistically significant results can lead to a potential overestimation of underlying treatment effects in a meta-analysis when using data extracted from trial publications. IPD meta-analysis has the potential to address this problem through facilitating analysis of core outcome sets,14 if available in trial datasets but not reported in publications.

Access to individual records should increase the number of trials included in the analysis and enhance the quality of outcome data. However, the benefits may not always be substantial. In the i-WIP project, the number of trials with the outcomes of interest was higher through access to IPD in comparison with data extracted from publications (online supplementary appendix 3). In addition, use of the raw data to generate outcomes not considered in original trials (eg, use of gestational age at delivery to define the occurrence of prematurity) may lead to a substantial increase in the number of the trial that can be incorporated into the meta-analysis (table 2). Nevertheless, the presence of data in the dataset did not always allow to incorporate a given dataset in the statistical analysis. Too few events (eg, stillbirths) and lack of all measures (baseline and final for weight gain) prevented trial inclusion. Still, in the example, incorporation of trials with previously unavailable outcome data changed the value of the effect estimate by more than 10% in three outcomes and its statistical significance in one (table 2).

The addition of unreported data may or may not lead to a change in funnel plot asymmetry. In the example, incorporation of unpublished outcomes in the meta-analysis for admission to neonatal intensive care unit has not revealed any potential bias. Similarly, for small for gestational age infant where outcome data were generated using raw data, if the outcome was not considered in original trials (table 2). For continuous outcomes (gestational weight gain in the example), the change in the plot asymmetry might also occur due to the standardisation of the analysis methods rather than incorporation of unreported data (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Comparison of funnel plots between meta-analyses using published and Individual Participant Data (IPD). 

Role of IPD meta-analysis in dynamic research areas

The authors of guidance on the appraisal of IPD meta-analyses of randomised trials advocate checking for the proportion of trials from which IPD was obtained.5 A recent study showed that only 25% of evaluated IPD meta-analyses obtained 100% of identified trial data.15 Acquisition of all eligible trials can be challenging for numerous reasons, with uncooperative trial investigators mentioned most commonly.5 IPD meta-analysis is a lengthy and resource-intensive process which can also decrease the chance of complying with the above-mentioned recommendation.

Since the publication of the systematic review that laid the grounds for the IPD meta-analysis we used as an example,16 there has been a significant increase in the number of trials evaluating the effects of diet and/or physical activity-based interventions in pregnancy. Between the end of data acquisition in June 2015 to February 2017, findings from additional 45 trials have been published (figure 2) making achieving the goal of being up-to-date and obtaining the majority of IPD virtually impossible.17 In combination with the trials for which IPD was sought but not obtained, the number of trials outside the IPD meta-analysis (non-IPD studies) constituted 65% of trials (67/103 trials) and 51% of women randomised to all eligible trials (12 960/25 486). The meta-analysis combining IPD with non-IPD studies showed a stronger overall effect of interventions in reduction of gestational weight gain and a significant reduction of odds for GDM than one using only IPD.17

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Number of randomised controlled trials with diet and/or physical activity-based interventions provided antenatally.

Summary

Despite the advantages of meta-analysis using IPD, the method encounters problems faced by other research methods such as uncooperative investigators or incompleteness of records. The IPD meta-analysis is a resource-demanding approach to evidence synthesis and requires a thorough evaluation of what is achievable. It might be that we will need to accept that some primary research is not usable for evidence synthesis. Mapping of definitions and additional data that could help to standardise the outcome across the trials may not tackle all the issues but will facilitate the smoother conduct of IPD meta-analyses. The efforts associated with obtaining IPD and its harmonisation need to be balanced by the potential gains achievable through a complex and profound statistical analysis. Prospectively designed IPD meta-analyses have the potential to overcome some of the challenges described in this article as they tend to collect data in a preagreed format.18 Promotion of consensus on the research standards with regard to outcome definitions, capturing of participants’ characteristics and effective ways of implementing them in the trials should help to reduce the potential research waste. Finally, putting the findings of IPD meta-analysis into a context of the totality of evidence is paramount for the validity of results.5  Currently, guidelines recommend adding non-IPD studies to IPD meta-analysis when a substantial proportion of trials IPD was not obtained at the beginning of the project. In addition, in some areas of medical research, the amount of evidence generated annually makes it difficult to stay up-to-date while conducting IPD meta-analysis. Therefore, adding newly published trials is as important as incorporating the not shared ones.