Confusion regarding the many evidence synthesis approaches

Within the family of systematic reviews there are different approaches, including reviews addressing interventions, prognosis, test accuracy, values and other overarching types of systematic reviews.14 Despite the usefulness of these systematic reviews to synthesise research, circumstances persist where they are unwieldy or not the tool of choice to meet the requirements of knowledge users.14–16 This recognition has led to many alternative approaches to evidence synthesis, including realist reviews, scoping reviews, umbrella reviews, concept analyses and others.7–10 17 In addition, numerous methods have been proposed for the synthesis of qualitative research,18 19 including thematic synthesis, realist synthesis, content analysis, meta-ethnography and meta-aggregation.18–20 These developments have expanded the array of methods to conduct evidence syntheses as described in the literature.

Although the differentiation of evidence synthesis types has made it possible to ensure that methods are tailored to particular questions, one pitfall has been the emergence of other (perhaps) less useful and/or rigorous approaches. These include (among others) the numerous sets of products collectively referred to as ‘rapid’ reviews and integrative reviews. Rapid reviews can be critiqued for their lack of uniformity or agreement on an ideal approach, and often appear haphazard in their conduct. However, guidance on the conduct of rapid reviews is being produced21 and they have played a key role in providing evidence in emergency situations such as during the COVID-19 pandemic,22 and when done appropriately (and transparently) rapid reviews do have a place in the evidence synthesis ecosystem when requested by decision-makers. Integrative reviews, which emerged as a reaction to the early practice of only including randomised controlled trials in systematic reviews, represent a diffuse methodology which now appear redundant given the appearance of more rigorous approaches to incorporating different study designs (such as non-randomised and randomised studies in intervention reviews23 or quantitative and qualitative studies in a single review24 and synthesising different types of evidence (such as qualitative evidence synthesis), in addition to statistical methods to meta-analyse non-randomised studies.

Other evidence synthesis types possess clearly defined methods but are yet to realise their full potential within the context of evidence based practice. For example, concept analyses,25 26 ‘a process whereby concepts are logically and systematically investigated to form clear and rigorously constructed conceptual definitions,’26 share at least one similarity with scoping reviews (which are more robust in their approaches) in their intention to clarify concepts in the literature.27 However, concept analyses lack the immediately apparent utility of scoping reviews. A case can be made, given that concept analyses have been critiqued for having no impact on practice,25 to revisit and reconsider whether a separate role exists outside of the scoping review.

It is now possible that almost any question can be answered by some application of an appropriate evidence synthesis process. However, it can be daunting and difficult for researchers to determine what type of synthesis approach is required, how to conceptualise and phrase the synthesis question, inclusion criteria and select the appropriate methods for analysis and synthesis. Typologies, taxonomies and other evidence synthesis classification systems may play a role here.

Typologies and taxonomies of evidence synthesis

Typologies of evidence synthesis types offer a starting reference point for researchers, policy makers and funders when deciding on a review approach.8–10 These individual typologies provide useful guidance on different review types but remain static—largely as traditional publications in a journal format that are not up-to-date and with minimal links to supporting materials. Furthermore, the development processes for many of these taxonomies are largely unclear with non-transparent methods. In a discussion paper by Gough et al,7 the authors distinguished between various review designs and methods but stopped short of providing a taxonomy of review types. The rationale for this was that in the field of evidence synthesis ‘the rate of development of new approaches to reviewing is too fast and the overlap of approaches too great for that to be helpful.’7 Instead, the authors provide a useful description of how reviews may differ and more importantly, an explanation why this may be the case. We concur that evidence synthesis methodology is a rapidly developing field, and that within different systematic review types there may be many different subsets and complexities that need to be addressed. These classifications can be considered to lie at the roots of a much larger family tree.

The evidence synthesis taxonomy initiative

Previous taxonomies and typologies are useful to categorise and direct researchers to the ideal synthesis approach, however limitations regarding their comprehensiveness, development methods and their currency exist. As the research and policy communities become increasingly aware of the need for evidence-informed choices, the breadth of evidence synthesis approaches also continues to expand. As this enthusiasm for evidence synthesis continues to propagate, we need measures in place to ensure: (1) we are making informed choices in the design and conduct of evidence syntheses; (2) we are producing these reviews as efficiently and accurately as possible, using reproducible and transparent methods and (3) all results and implications from these syntheses are trustworthy. As such, the value of a resource that guides and drives appropriate synthesis approaches in response to relevant clinical and policy needs, such as a research-informed evidence synthesis taxonomy, becomes essential. The authors of this paper are now collaborating with like-minded researchers on such an initiative, including the shared commitment to keep this a living taxonomy, continuously updated (such as via a living wiki platform) and refined alongside advances in the field. This work complements a decision tool that currently steers researchers and decision-makers towards the right choice when conducting a review—aptly named ‘Right Review’.28 In addition, this work will align neatly with the position statement from Evidence Synthesis International, in recognising the need to develop and share standards, terminology and methodology consistently across the field while acknowledging the need for ‘fit for purpose’ approaches for diverse evidence requirements.29

We envisage a number of challenges when attempting this work. First, we will need to ensure we commit to a truly living approach given the rate of development of new methods and approaches. Second, given our collaborative approach, it is likely there will be strong views among the various contributors and organisations, which will require the use of formal consensus development methods. Third, we want to ensure all stakeholders are included, which will result in challenges organising a diverse group of people and opinions, but we see this as critical to ensuring the utility of the final taxonomy. The inclusion of these diverse stakeholders will be critical to ensure the taxonomy addresses research waste and overproduction of reviews. As an example, by ensuring the inclusion of editors, we hope that journals and publishers may be able to better identify redundant reviews or reviews applying inappropriate methodology prior to publication, in addition to updating author guidelines and requirements as a means to prevent these problems.