Article Text
PDF
Statistics from Altmetric.com
Introduction
Evidence-based practice forms an integral part of modern-day practice. It is so indispensable that modern healthcare cannot be imagined if evidence and its quality is ignored.1 However, evidence-based medicine is not as perfect as it is thought. Time and again, researchers have questioned the quality of evidence (QOE) in present era’s researches. The evidence in this regard can be found in the article published in The Lancet by the editor himself. Horton argued that much of the scientific literature, nearly 50%, might be false or untrue because of inclusion of small sample size studies, researches with statistically insignificant effects, unfitting analyses, atrocious conflicts of interest, together with an unrestricted fascination to pursue fashionable trends of dubious importance.2
Camouflaged application of evidence-based medicine can be precarious to the patients and may hamper the idea of promoting healthy society.3 According to National Institute for Clinical Excellence, approximately 62% of publications used to formulate the guidelines and recommendations of primary care were based on dubious researches and were judged of uncertain relevance to patients.4
The most recent incident which has raised global concerns on the QOE and how evidence can be tampered can be found from the news printed in The Guardian, where two of the highest ranking international medical journals—New England Journal of Medicine and The Lancet had to retract two papers on COVID-19 pandemic. The Lancet paper, ‘A multinational registry analysis of hydroxychloroquine on COVID-19’, halted global trials on hydroxychloroquine for COVID-19 because of fears of increased deaths from hydroxychloroquine. A thorough introspection later confirmed inconsistencies in the data provision and analysis. Such was the impact of this study that WHO and other countries worldwide had to suspend their randomised control trials (RCT) on the usefulness of hydroxychloroquine on COVID-19.5 Future trials may account for the number of patients put to risk in this global pandemic by such a flawed research. Good quality of research plays a pivotal role in providing evidence in medical practice; hence, every effort should be taken to ensure highest quality in evidence.
QOE is defined according to GRADE (Grading of Recommendations Assessment, Development and Evaluation Working Group) framework, as the confidence one can place that the reported estimates of effect described is real and is competent enough to support a specific recommendation.6 GRADE system has emerged as one of the dominant tools for critically appraising controlled studies and making recommendations for systematic reviews. Understanding the intricacies for critically appraising a systematic review or guidelines requires a sound knowledge of GRADE tool or other critical appraisal tools.7
How to ensure the QOE and its applicability in suggesting guidelines and recommendations?
QOE is dependent on the research methodology and reporting quality.8 Pursuing a high-quality research involves drafting a flawless design of the study and holds prime importance in the level of confidence in the outcome of the study. The Oxford Centre for Evidence-Based Medicine provides an evidence pyramid which is based on diagnostic, prognostic, screening and therapeutic studies.9 According to this, the strongest evidence is provided by the systematic reviews followed by (in descending order of strength of evidence) RCT, cohort, case–control studies, case series and finally, expert opinions. Therefore, whenever feasible well conducted systematic reviews, meta-analysis and RCTs should be used to clinical decision-making.
Factors determining the QOE
QOE depends on the overall quality of the research which in turn is dependent on methodological quality. According to GRADE approach, the QOE is dependent on eight parameters which can affect the QOE. Five of these parameters downgrade the quality while the remaining three are known to upgrade the QOE. Factors having negative impact on QOE includes limitations of the study conducted, inconsistency or variability of the results, imprecision, indirectness of evidence and bias related to publication. Study design plays a pivotal role in deciding the QOE.10
Estimate of the treatment effect loses its credibility in the presence of bias like lack of allocation concealment and blinding, large attrition bias, failure to adhere to the principles of intention to treat and selective or preferential reporting of outcomes. These biases form the major limitation of the study and jeopardise the QOE. Results may become inconsistent in the presence of heterogeneity or variability because of differences in populations, interventions and outcomes. When researcher fails to identify a plausible explanation for heterogeneity the QOE decreases.
Indirectness of evidence becomes apparent when two active interventions exist for similar problem and research is conducted separately for both interventions comparing the intervention/drug with placebo. Such evidence will be of lower quality than would be provided by head to head comparisons of the two interventions/drugs. Imprecision occurs when a study is conducted with a very small sample size with very few variables resulting in large confidence interval (CI). Clinical trials sponsored by the pharmaceutical industry may be negatively stirred by business interests. Industry funded trials particularly small studies or a novel therapeutic RCT, may be limited by bias including methodological, reporting and publication bias. With-holding the publication of adverse results in such studies is quite common, and this can potentially cause serious issues especially when a novel therapy or drug is approved for marketing without taking into consideration the failure of sponsors to disclose all the potential benefits or risks of the therapy or drug in consideration. Hence, publication of such industry funded trials in high peer-reviewed journals may jeopardise the effect estimates by introducing bias, and in turn lead to down gradation of QOE.11 12 The body of evidence is dependent on these limitations, and the greater the limitations the lower the quality of the evidence.6 Additionally, the QOE can also be degraded by imprecise or sparse data and high risk of reporting bias.13
Factors that result in upgradation of evidence includes large magnitude of effect, conceivable confounding and dose–response gradient.6 One tends to give high rating to QOE because of profound or very large magnitude of an effect, when the effect tends to be consistent across all subjects or the effect results in reversal of previous trial of disease. It is pertinent to note that when outcomes are subjective it is important as a researcher to become more vigilant when considering upgrading because of observed large effects. This is especially true when outcome investigators are not blinded and are aware of allocation group.10 The dose–response gradient may reflect significant cause–effect relationship. Hence, its presence might increase our confidence in the findings of studies and thereby increase the QOE. Occasionally all probable biases from observational studies may underestimate the true treatment effect, if such is the condition then the evidence can be upgraded.
Grading of evidence
The QOE not only depends on the study design, quality, consistency and directness but also on limitations of the study, inconsistencies in the results, biases and uncertainty of the evidence. The GRADE criteria analysed these factors and suggested a grading system for the QOE. According to which the QOE can be subdivided into four grades, high, moderate, low and very low.
Highest grade is awarded when further research is unlikely to change the estimate of effect. Grade of evidence is rated as moderate when there is scope of further research and it is likely to impact or change the estimate of effect while if further research in all probabilities will change the estimate, the evidence is graded as low. If it becomes difficult to ascertain any estimate of effect, the evidence is graded as very low. These grades are taken into account when we formulate the recommendations and guidelines based on the QOE and has a bearing on the delivery of evidence-based practice to the patients.10
Other factors affecting recommendations
Having a high QOE does not signify a strong recommendation. Recommendations and guidelines should consider factors other than mere QOE. They should take into account prudent and imprudent effects of an intervention, patient values and their preference, and costs.7 Hence, the overall QOE is an amalgamation of QOE across all outcomes which are considered crucial for answering a healthcare question, deciding or formulating recommendation and guidelines.
A simple and easy way to understand, how GRADE framework can be used to appraise evidence before a guideline or recommendation is formulated is depicted in the flow chart (figure 1).
Flow chart depicting Grading of Recommendations Assessment, Development and Evaluation Working Group framework. RCT, randomised control trials.
Conclusion
To understand and evaluate the evidence and its quality is an indispensable skill for any clinician. There are various tools to evaluate and critically appraise the QOE in terms of quality in materials and methods and reporting of a study or evidence. To adequately assess the QOE is difficult and the most honest way to appraise the QOE is to look at the individual component of facts which form the evidence rather than trying to blindly categorise evidence into high-grade or low-grade quality. Mastering the art of critical appraisal warrants discretion, training and practice. In recent times, GRADE has become an integral element for appraising newly published systematic reviews by Cochrane. The GRADE approach provides clear, elaborate and an effective methodology for grading the QOE supporting recommendations and confers transparency for presenting evidence and developing guidelines.
Ethics statements
Patient consent for publication
Footnotes
Contributors All the authors contributed substantially to the conception of the study, design, literature search and write-up of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.