You are here

  • Home
  • Online First
  • Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment

Article Text

Article menu
Download PDFPDF
Research methods and reporting
Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment
  1. Bashar Hasan1,2,
  2. Samer Saadi1,2,
  3. Noora S Rajjoub1,
  4. Moustafa Hegazi1,2,
  5. Mohammad Al-Kordi1,2,
  6. Farah Fleti1,2,
  7. Magdoleen Farah1,2,
  8. Irbaz B Riaz3,
  9. Imon Banerjee4,5,
  10. Zhen Wang1,6,
  11. Mohammad Hassan Murad1,2
  1. 1Kern Center for the Science of Healthcare Delivery, Mayo Clinic, Rochester, Minnesota, USA
  2. 2Public Health, Infectious Diseases and Occupational Medicine, Mayo Clinic, Rochester, Minnesota, USA
  3. 3Division of Hematology-Oncology Department of Medicine, Mayo Clinic, Rochester, Minnesota, USA
  4. 4Department of Radiology, Mayo Clinic Arizona, Scottsdale, Arizona, USA
  5. 5School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA
  6. 6Health Care Policy and Research, Mayo Clinic Minnesota, Rochester, Minnesota, USA
  1. Correspondence to Dr Bashar Hasan, Mayo Clinic, Rochester, MN 55905, USA; Hasan.Bashar{at}mayo.edu

Abstract

Large language models (LLMs) may facilitate and expedite systematic reviews, although the approach to integrate LLMs in the review process is unclear. This study evaluates GPT-4 agreement with human reviewers in assessing the risk of bias using the Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool and proposes a framework for integrating LLMs into systematic reviews. The case study demonstrated that raw per cent agreement was the highest for the ROBINS-I domain of ‘Classification of Intervention’. Kendall agreement coefficient was highest for the domains of ‘Participant Selection’, ‘Missing Data’ and ‘Measurement of Outcomes’, suggesting moderate agreement in these domains. Raw agreement about the overall risk of bias across domains was 61% (Kendall coefficient=0.35). The proposed framework for integrating LLMs into systematic reviews consists of four domains: rationale for LLM use, protocol (task definition, model selection, prompt engineering, data entry methods, human role and success metrics), execution (iterative revisions to the protocol) and reporting. We identify five basic task types relevant to systematic reviews: selection, extraction, judgement, analysis and narration. Considering the agreement level with a human reviewer in the case study, pairing artificial intelligence with an independent human reviewer remains required.

  • Evidence-Based Practice
  • Methods
  • Systematic Reviews as Topic

Data availability statement

Data are available upon reasonable request. Search strategy, selection process flowchart, prompts and boxes containing included SRs and studies are available in the appendix. Analysed datasheet is available upon request.

https://doi.org/10.1136/bmjebm-2023-112597

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Data availability statement

    Data are available upon reasonable request. Search strategy, selection process flowchart, prompts and boxes containing included SRs and studies are available in the appendix. Analysed datasheet is available upon request.

    View Full Text

    Footnotes

    • Twitter @BasharHasanMD, @M_Hassan_Murad

    • Contributors MHM and BH conceived this study. BH, SS, MH, MA-K, FF, MF, ZW, IBR, IB and NSR participated in data identification, extraction and analysis. MHM, SS, IBR and IB wrote the first draft. All authors critically revised the manuscript and approved the final version. BH is the guarantor.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.