Article Text

Download PDFPDF

What is a ‘complex systematic review’? Criteria, definition, and examples
  1. Kamal R Mahtani,
  2. Tom Jefferson,
  3. Carl Heneghan,
  4. David Nunan,
  5. Jeffrey K Aronson
  1. Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  1. Correspondence to Dr Kamal R Mahtani, Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX1 2JD, UK; kamal.mahtani{at}

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Systematic reviews involve systematically searching for all available evidence, appraising the quality of the included studies, and synthesising the evidence into a usable form. They contribute to the pool of best available evidence, translating research into practice, and are powerful tools for clinicians, policymakers, and patients.1

To be useful for decision making, systematic reviews need to include high-quality evidence. However, there are systemic failings with the publishing, reporting, and interpretation of much of the evidence base, which undermine the findings of systematic reviews.2 3 In some cases, the evidence remains hidden from view or, when published, important outcomes are selectively reported, further hindering interpretation of reviews.4 Also, multiple interventions often have not been compared head to head, requiring more complex and indirect methods of evaluation. Finally, best practice guidance needs to be adaptable to real-world practice scenarios, which often requires combined information from multiple sources of evidence.5

To plug the evidence-to-practice translational gap, review methods are evolving beyond conventional ‘what works’ systematic reviews. Broader forms of evidence synthesis have emerged, such as network meta-analysis, scoping reviews, realist reviews, umbrella and meta-narrative reviews, meta-synthesis, and several others.6 7 Cochrane groups have expanded, with the creation of specific methods groups to reflect the growing number of review types.

The recognition that complex clinical and policy questions require more advanced methods of evidence synthesis to answer them has led to the term ‘complex review’, which is used to cover a wide range of evolving methods. However, there is some uncertainty as to what a complex systematic review is.

Defining ‘complex systematic reviews’

There is a tried and tested method of defining scientific terms, by exploring their etymology, usages, and previous definitions, and considering the elements of which the relevant processes consist.

Etymology and usage

The word ‘complex’ derives from an ancient linguistic root, PLEK, which implies weaving, giving words such as plait, pleat, pliant, and plicate. Adding the prefix ‘com’ (Latin cum, meaning ‘with’) to the basic root implies interweaving of several elements, making things complicated, literally woven together. Reflecting this, ‘complex’ is defined in the Oxford English Dictionary as ‘consisting of parts or elements not simply co-ordinated, but some of them involved in various degrees of subordination; complicated, involved, intricate; not easily analysed or disentangled’. No other usages of the word have emerged since it entered English in this form in the 17th century.

Previous definitions

Seeking previous definitions of complex systematic reviews, we did a rapid scope of the literature, with the following results:

  • First, we identified a need for distinction between a systematic review of complex interventions and a complex systematic review. Petticrew and colleagues have argued ‘that reviews of complex interventions can themselves be simple or complex, depending on the question to be answered.’8 As outlined below, we argue that a complex systematic review may be designated as such for a variety of reasons, regardless of how complex the unit of analysis is (eg, the intervention(s), procedures, and tests).

  • We looked for definitions on the National Institute for Health Research Complex Reviews Unit website, but could not find any, although we did find a list of reasons for the increasing complexity of reviews.9

  • Gough and colleagues described ‘multi-component reviews' as a strategy for dealing with complex review questions, but we were still unable to identify an explicit definition of a complex review.10

  • We found a definition of a ‘comprehensive review’ in the Joanna Briggs Methods Manual: ‘A systematic review is considered to be a comprehensive systematic review when it includes two or more types of evidence, such as both qualitative and quantitative, in order to address a particular review objective.’11

  • We identified an excellent series of articles under the banner ‘Considering Complexity in Systematic Reviews of Interventions’ in The Journal of Clinical Epidemiology, although we were unable to identify an explicit definition of a complex review.12

  • We identified a paper by Whitlock and colleagues, who described a complex review as one that ‘evaluate(s) a number of linked clinical questions, multiple interventions or diagnostic tests, different or distinct population groups, and/or many outcomes.'13

Elements of the process

Considering how complex reviews are carried out, we propose that for a systematic review to be designated as ‘complex’, it should demonstrate exceptional aspects in two broad domains: the nature and sources of the data included in the review and the processes of synthesis involved in their interpretation. We have identified eight aspects of complex reviews that underpin these domains, described in the text below and illustrated in figure 1.

Figure 1

Proposed elements of a complex systematic review. 

Several of these aspects reflect the interweaving of disparate elements implied by the origin of the word ‘complex’. We suggest that a complex systematic review should primarily fulfil the following criterion:

  • The need for a highly skilled and multidisciplinary team to complete the review. Although the Cochrane Handbook alludes to this,14 we extend this absolute requirement by stating that: the skill mix needed to complete a complex review should include a breadth of content expertise, including end-users of the research, to shape and contextualise the relevance and implications of the review findings. The success of a complex review should depend on highly skilled methodological expertise, without which it would be impossible to complete the review.

This is the only absolute criterion. The other elements (below) are optional, but we suggest that a requirement of any complex systematic review is that any two should be included.

  • The use of more than one type of study design (eg, qualitative and quantitative) and more than one type of synthesis approach. Gough and colleagues describe a continuum in synthesis approaches that may, at one end, use a predefined method to aggregate existing data and test a theory, or at the other, use a more iterative synthesis approach and configure existing data to generate a new theory.10 A complex review may do both.

  • The inclusion of a large quantity of data. This could include, for example, clinical study reports (CSRs), large and highly detailed documents written for regulators as part of licensing applications for medicinal products.15 However, we are not proposing an arbitrary cut-off for what would constitute a large quantity of data. A systematic review of over 100 published randomised controlled trials might provide as much relevant data as two CSRs, each over 4000 pages long. Context is important. Instead, we propose that researchers should justify whether their review qualifies under this heading and why.

  • Assessment of a complete evidence development programme. This cumulative collection of data may be presented chronologically, providing greater insights into the origins and purpose of a technology. For example, combined assessment of preclinical, phase I–IV studies of a technology or evidence of the development of a complex instrument or a large diagnostic tool from the horizon-scanning phase to the postmarketing phase, perhaps starting with a definition of its predicate (ie, the index molecule, instrument, or device from which all other versions and medicinal products have evolved).

  • Systematic inclusion of data from several different sources. Most systematic reviews source data from the existing published literature and sometimes from the grey (ie, unpublished) literature as well. However, the validity of a systematic review is threatened by the scope of the data it includes. Problems with publication bias, particularly in relation to pharmaceuticals, are well documented.4 These problems have led to calls for inclusion of data from sources other than just the aggregative summaries in the published literature.2 Including individual participant or patient data may mitigate this to some degree.16 Other sources could include regulatory or registry data, where additional information, for example, on adverse events, may be found.17

  • An evaluation of an intervention in relation to the contexts in which it is set, for example, information to describe the process through which the intervention is being implemented or to explain the rationale for the choice of study design, in the light of current knowledge. Both of these types could also include timelines.

  • Reviews that simultaneously answer questions from different perspectives and viewpoints: societal, healthcare, funders, patients and carers, healthcare workers, and manufacturers. These could include multiple evaluations and comparisons using different evaluation methods consistent with the different perspectives.

  • The use of a particular evidence synthesis method for the first time. A review of evidence synthesis methods used across the fields of nursing, healthcare science and services, and health policy found that many methods were contemporary and probably emerging, implying that more detailed descriptions and validation studies were likely to appear in future.6

Here we provide examples of systematic reviews that can justifiably be termed complex (table 1). All involved a multidisciplinary team.

Table 1

Examples of complex systematic reviews that fit our criteria

Thus, based on etymology, usage, previous definitions, and considerations of the elements of which a complex systematic review consists, we propose the following definition:

Complex systematic review n. A systematic review, performed by a multidisciplinary team, consisting of multiple components, large amounts of data from different sources or different perspectives, collectively contributing more than would be expected from their individual contributions, the individual components not being easily coordinated, analysed or disentangled.


Systematic reviews play a pivotal role in the translation of research findings into practice and policy.18 However, the field is moving beyond traditional ‘what works’ reviews to more complex reviews, which may make a greater contribution to healthcare decision making. In this commentary, we seek to advance the field by presenting a definition of a complex systematic review to provide clarity in the field and to stimulate further thought and discussion.


KRM and CH are both co-investigators of The Evidence Synthesis Working Group, which is funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR) (Project Number 390).


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.


  • Contributors All authors contributed to discussions of this topic. KRM and TJ drafted the first version of this manuscript with contributions from the remaining authors.

  • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

  • Disclaimer The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the National Health Service, the National Institute for Health Research, or the Department of Health.

  • Competing interests KRM receives funding from the National Health Service National Institute for Health Research (NIHR) Health Technology Assessment programme as Primary Care Panel Chair. He is also the director of an MSc in systematic reviews. TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. In addition, TJ receives royalties from his books published by Il Pensiero Scientifico Editore, Rome, and Blackwells. TJ is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011–2013, TJ acted as an expert witness in litigation related to the antiviral oseltamivir, in two litigation cases on potential vaccine-related damage, and in a labour case on influenza vaccines in healthcare workers in Canada. In 2014, he was retained as a scientific adviser to a legal team acting on oseltamivir. TJ has a potential financial conflict of interest in the drug oseltamivir. In 2014– 2016, TJ was a member of three advisory boards for Boehringer Ingelheim. He is holder of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial on an influenza vaccine. TJ is a co-signatory of the Nordic Cochrane Centre complaint to the European Medicines Agency (EMA) over maladministration at the EMA in relation to the investigation of alleged harms of HPV vaccines and consequent complaints to the European Ombudsman. TJ is co-holder of a John and Laura Arnold Foundation grant for development of a RIAT support centre (2017–2020) and Jean Monnet Network Grant, 2017–2020 for The Jean Monnet Health Law and Policy Network. CH has received expenses and fees for his media work, including BBC Inside Health. He holds grant funding from the NIHR, the NIHR School of Primary Care Research, The Wellcome Trust, and the WHO. He has also received income from the publication of a series of toolkit books published by Blackwells. CEBM jointly runs the EvidenceLive Conference with the BMJ and the Overdiagnosis Conference with some international partners which are based on a non-profit model. DN has received expenses and fees for his media work. He holds grant funding from the NIHR School of Primary Care Research and the Royal College of General Practitioners. JKA is a member of the CEBM, with similar potential conflicts to those outlined above.

  • Patient consent Not required.

  • Provenance and peer review Commissioned; internally peer reviewed.

  • Data sharing statement This is an extended adaptation of a preliminary piece that was posted as a blog at BMJ Opinion.