Original ArticleThe GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses
Section snippets
Background
The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) working group includes guideline developers, systematic reviewers, clinicians, public health officers, researchers, methodologists, and other health professionals from around the world [1]. The group has spent over a decade developing and refining a systematic, transparent, and explicit process for summarizing, grading, and presenting evidence, and for moving from evidence to health care recommendations. More than
Design
Participants initially worked independently as individual raters assessing the QoE. Once individual raters submitted their judgments about QoE, we randomly paired them with another rater. We asked each pair to discuss their independent ratings and resolve discrepancies by discussion before submitting their final consensus judgment. Both individual raters and pairs of raters worked independently from other raters who evaluated the same evidence. This design allowed us to test the effect of
Results
Twenty-seven members of the GRADE working group and 10 students from the McMaster University HRM graduate program agreed to participate in the study. Fifteen of the 27 GRADE working group members and the 10 students served as raters and completed the assessment of all 16 outcomes.
Table 1 summarizes the baseline characteristics and previous experiences of the raters. Eight of 15 members in the GRADE working group were involved in developing aspects of the GRADE approach. Thirteen raters (nine
Discussion
We found substantial IRR when two individual raters assessed the QoE of 16 outcomes from four systematic reviews. The IRR of the GRADE approach using a GRADE-naive group (students) improved significantly after the calibration exercises from slight agreement (0.11) to substantial agreement (0.66). IRR using the GRADE approach and raters that were familiar with the approach was already high initially (0.62) and improved only slightly (0.72) after the calibration exercises. IRR was similar among
Acknowledgments
Competing interests: No financial competing interest.
Some authors are involved in the development and dissemination of GRADE, and GRADE's success has a positive influence on their academic career.
Authors' contributions: H.J.S. conceived of the study. R.A.M. and H.J.S. designed the study. J.B., E.A.A., S.D.W., R.C., and G.H.G. contributed to the conception and design. R.A.M., G.N., and M.K. performed the statistical analysis. R.A.M. and H.J.S. drafted the manuscript. All the coauthors critically
References (23)
- et al.
GRADE guidelines: 2. Framing the question and deciding on important outcomes
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 8. Rating the quality of evidence—indirectness
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
J Clin Epidemiol
(2011) - et al.
GRADE guidelines 6. Rating the quality of evidence—imprecision
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 5. Rating the quality of evidence—publication bias
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 9. Rating up the quality of evidence
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
(2011) - Available at http://www.gradeworkinggroup.org/. Accessed June 1,...
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
BMJ
Cited by (266)
Perioperative Gabapentin May Reduce Opioid Requirement for Early Postoperative Pain in Patients Undergoing Anterior Cruciate Ligament Reconstruction: A Systematic Review of Randomized Controlled Trials
2023, Arthroscopy - Journal of Arthroscopic and Related SurgeryA Systematic Review of the Efficacy and Safety of Ketamine in Total Joint Arthroplasty
2023, Journal of Arthroplasty