Article Text
PDF
Statistics from Altmetric.com
Published research should be reported to evidence users with clarity and transparency to facilitate optimal appraisal and use of evidence and allow replication by other researchers. However, for a variety of reasons, the literature is commonly viewed as poorly reported with critical information missing from published reports. This necessitated the development of reporting guidelines. Currently, there are guidelines for reporting randomised trials (CONSORT), observational studies (STROBE), systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)), case reports (CARE), qualitative research (SRQR), diagnostic and prognostic studies (STARD), quality improvement studies (SQUIRE), economic evaluations (CHEERS), animal preclinical studies (ARRIVE), study protocols (SPIRIT) and clinical practice guidelines (AGREE). These guidelines are catalogued on the website of the EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research).1 The EQUATOR Network is an international initiative that seeks to improve the reliability and value of health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. Some of these reporting guidelines have extensions for subtypes of research (eg, PRISMA-Equity is used for equity-focused systematic reviews). Some function as a tool for reporting and a tool for assessing the methodological quality (eg, AGREE-II is used to evaluate the rigour and the reporting of clinical practice guidelines).
Meta-epidemiological studies are a fairly new type of study that has been increasingly published in the last two decades,2–4 with no clear reporting standards. Sometimes called methodology research, meta-epidemiological research adopts a systematic review or meta-analysis to provide data for methodological analysis.5 The goal is generally broad but often focuses on examining the impact of certain characteristics of clinical studies on the observed effect, describing the distribution of research evidence in a specific setting, examining heterogeneity and exploring its causes, identifying and describing plausible biases and providing empirical evidence for hypothesised associations. Unlike classic epidemiology, the unit of analysis for meta-epidemiological studies is a study, not a patient. The outcomes of meta-epidemiological studies are usually not clinical outcomes.6–8
In this guide, we adapt the items used in the PRISMA statement9 for reporting systematic reviews and meta-analysis to fit the setting of meta- epidemiological studies. The rationale for starting with PRISMA is that meta- epidemiological research requires a systematic approach (ie, a systematic review) to select data (the input for methodological analysis). The overarching goal of this guideline is to improve the transparency and clarity of healthcare research.
Examples of meta-epidemiological studies
Analysis of Cochrane systematic reviews showed that trials with inadequate or unclear allocation concealment yielded larger estimates of effect relative to controlled trials with adequate allocation concealment.10 In a second meta-epidemiological study, non-blinded trials of complementary medicine exaggerated the effect size by an average of 0.56 SD (95% CI –0.71 to –0.41) compared with blinded trials.11 A third example is a study that explored the effect of trials stopped early for benefit on future research.12 In a fourth example, it was shown that in almost 30% of endocrinology studies, the largest treatment effect was observed in the first or second published studies (ie, earlier studies, when studies were ordered chronologically using the date of publication).13
Proposed reporting guidelines
We have adapted items of the PRISMA statement to fit a meta-epidemiological study. The items are categorised according to the title, abstract, introduction, methods, results, conclusion and funding and are supported by rationale and examples.
Title and abstract
Both the title and abstract should identify the report as a meta-epidemiological study, for example, ‘Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology’.13 The rationale for this identification is to improve indexing and facilitate retrieval. It also helps clinicians searching for systematic reviews about clinical topics to quickly find relevant evidence and avoid reviewing additional non-clinical meta-epidemiological studies. The abstract should also include a structured summary with subheadings that provide the context, hypothesis, methods, data sources, results, limitations and conclusions of meta-epidemiological studies. Variations of these subheadings are acceptable. The rationale for having this information in the abstract relates to the fact that many readers only read the abstract. These requirements are similar to PRISMA.
Introduction
The introduction should include the rationale and objectives of the study. This requirement is similar to PRISMA. However, it is more important to explicitly state the hypothesis being tested in meta-epidemiological studies compared with a systematic review summarising evidence about the effectiveness of an intervention (which may or may not have a clear hypothesis and primarily aim at empirically evaluating a clinical outcome).
Methods
The methods section should specify if the meta-epidemiological study follows a priori established protocol and clarify the eligibility criteria, information sources, search methods, study selection and data collection process, data items collected, summary measures and synthesis methods (including assumptions and computational approximations and methods of additional analyses such as sensitivity, subgroup or meta-regression analyses). These requirements are similar to PRISMA. However, registration of protocols may not be needed, particularly since systematic reviews registration sites are geared more towards clinical questions. Search strategy in a meta-epidemiological study is different because it is not driven by a clinical question. Evaluating and reporting the risk of bias in individual studies may or may not be relevant to the question at hand. The type of bias can also be different. Data extraction is often more complex in meta-epidemiological studies and should be described in details sufficient to allow replication. The methods section of a meta-epidemiological study should describe in details the effect measure used in statistical analysis because it is likely that a clinical audience will not be familiar with such measure (eg, ratio of OR, standardised measures and regression coefficients). Such measures are not commonly used in studies that report on treatment effects (ie, clinical audience would be familiar with OR but not familiar with ratio of OR). Quantifying interagreement between investigators is more critical in a meta-epidemiological study than in a clinical systematic review because of the judgements commonly made to classify data in a meta-epidemiological study. For example, a study evaluated the association between industry affiliation and researchers’ position on the cardiovascular risk of rosiglitazone. Reviewers had to assess how favourable was the researchers’ opinion about rosiglitazone using an ordinal scale.14 This task is fairly subjective and requires judgement. In this case, agreement between reviewers is critical to report and affects the reliability of this meta-epidemiological study (arguably more critical than in a systematic review of clinical outcomes).
Results
The results section should include the results of the study selection process (how many studies, how many patients, etc). Different from PRISMA, study and patient characteristics need to be tailored to the meta-epidemiological question at hand. Therefore, it may not be important to know patients' age, comorbidities and settings in some questions (although such characteristics may be important in other questions). For example, the meta-epidemiological study showing that trials with inadequate allocation concealment yielded larger estimates of effects (relative to trials with adequate allocation concealment) does not necessarily have to describe patients’ characteristics or comorbidities in each individual trial.10 Publication bias should be evaluated and reported similar to PRISMA recommendations.
Discussion and conclusions
As with PRISMA, the discussion section should include summary of the main findings and comparison with existing knowledge and should also include the limitations and strengths of the meta-epidemiological study. Limitations at a study level may not be important (as in clinical systematic reviews); however, limitations at the level of the meta-epidemiological study are important. The quality of evidence is a highly important step in a systematic review reporting on a treatment effect15 and is critical for decision making but also may not be relevant to a meta-epidemiological study. Investigators should, however, inform readers of their certainty in the results, even if this was determined narratively16 because formal approaches15 are intended to rate the certainty of evidence about interventions. Lastly, systematic reviews of clinical questions should include in their discussion the implication of their findings on future research and current clinical practice. Meta-epidemiological studies will have profound implications for future research and should report such implications. Meta-epidemiological studies will likely have minimal direct impact on clinical practice. Nevertheless, indirect impact on clinical practice is possible and should be reported when present. For example, a meta-epidemiological study showed that in endocrinology, the largest treatment effect was observed in the first or second published studies.13 Clinically, this should make endocrinologists sceptical about new drugs, laboratory tests or genetic markers and not rush into recommending them to their patients until more studies confirm findings observed in these earlier studies.13
In table 1, we present PRISMA items with adaptations to fit the setting of meta-epidemiological studies.
Proposed items to be used for reporting methodology research, adapted from the PRISMA Checklist (http://prisma-statement.org/PRISMAStatement/Checklist.aspx)
Discussion and limitations
Reporting guidelines have three goals: to enhance the applicability and use of research results, to allow readers to critically appraise the results and to permit reproducibility and replication of the work, which is essential for accountability and transparency purposes. Meta-epidemiological studies should have similar standards and require reporting guidelines that achieve these three goals.
In this guideline, we propose adaptation to PRISMA items to fit meta-epidemiological studies. Meta-epidemiological studies are inherently heterogeneous in terms of their approach and execution, which is a challenge for current reporting guidelines of systematic reviews (such as PRISMA). For example, a meta-epidemiological study used trial registries as a source of its data.17 Registries are not searched using standard search terms and methods and may not fit the reporting requirement needed for systematic reviews. Entries in trial registries are also dynamic, just like Internet searches. In this case, authors of such study should make every possible effort to describe their work in a reproducible fashion but acknowledge these limitations.17 In addition, reporting guidelines are frequently underused and are not always emphasised by journal editors. A descriptive analysis of published randomised trials in the four highest impact general medical journals shows inconsistencies and non-adherence to the CONSORT statement (eg, allocation concealment and details of blinding were reported only in 12% and 21%, respectively).18 Adherence across all reporting guidelines was estimated to be 86% in another study.19 Evaluation of the instructions given to authors shows that PRISMA guidance was referenced only in 13% of gastroenterology and hepatology journals.20
Here, we have proposed a reporting guideline to be used when publishing meta-epidemiological research. We hope that these guidelines will be used, modified and enhanced and lead to more clarity and transparency of healthcare research.
References
Footnotes
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.