Article Text
Abstract
An evidence-based approach is considered the gold standard for health decision-making. Sometimes, a guideline panel might judge the certainty that the desirable effects of an intervention clearly outweigh its undesirable effects as high, but the body of supportive evidence is indirect. In such cases, the application of the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach for grading the strength of recommendations is inappropriate. Instead, the GRADE Working Group has recommended developing ungraded best or good practice statement (GPS) and developed guidance under which circumsances they would be appropriate.
Through an evaluation of COVID-1- related recommendations on the eCOVID Recommendation Map (COVID-19.recmap.org), we found that recommendations qualifying a GPS were widespread. However, guideline developers failed to label them as GPS or transparently report justifications for their development. We identified ways to improve and facilitate the operationalisation and implementation of the GRADE guidance for GPS.
Herein, we propose a structured process for the development of GPSs that includes applying a sequential order for the GRADE guidance for developing GPS. This operationalisation considers relevant evidence-to-decision criteria when assessing the net consequences of implementing the statement, and reporting information supporting judgments for each criterion. We also propose a standardised table to facilitate the identification of GPS and reporting of their development. This operationalised guidance, if endorsed by guideline developers, may palliate some of the shortcomings identified. Our proposal may also inform future updates of the GRADE guidance for GPS.
- COVID-19
- Evidence-Based Practice
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary box
Good practice statements (GPSs) are developed when there is high certainty that the desirable effects of an intervention clearly outweigh its undesirable effects, but the body of supportive evidence is indirect and other criteria for their development are fulfilled; under those circumstances, the application of the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach to develop a formal recommendation is inappropriate. However, the GRADE Working Group provides guidance for identifying GPS and suggests five criteria that should be explicitly considered before developing GPS.
Summary box
GPS are common actionable statements in guidelines, but are rarely labelled as GPS by guideline developers and lack explicit rationales to support their development. We encourage guideline developers to consider relevant evidence-to-decision criteria when assessing the net consequences of implementing the statement, not just direct health benefit and harms, and encourage the explicit labelling and detailed reporting of information supporting a guideline developer’s decisions for each GPS criterion.
We also propose a structured process for the development of GPS, that includes operationalisation of the GRADE guidance. Based on GPS related to guidelines to address COVID-19, we built a standardised table for developing GPS that should be explicitly considered by guideline developers.
Introduction
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach has been developed to facilitate the rigorous and transparent formulation of recommendations from evidence.1 The approach rates the certainty of evidence in an effect estimate as high, moderate, low or very low based on the evaluation of risk of bias, imprecision, indirectness, inconsistency, publication bias and domains that increase the certainty.2 After considering the criteria that determine a recommendation or decision, strong or conditional recommendations are developed, depending, partly, on the level of certainty that the benefits of an intervention outweigh its harms.
Best or good practice statements (GPS) are actionable statements deemed to be necessary for practice (desirable effects of an intervention clearly outweigh its undesirable effects) but are supported by indirect evidence that does not diminish the certainty in evidence. GPS tend to provide guidance in all aspects of healthcare practice and services such as diagnosis, treatment and health systems. An example of a GPS is the first recommendation from the American Gastroenterological Association (AGA)’s 2015 guideline on the management of asymptomatic pancreatic cysts, which reads, ‘The AGA recommends that before starting any pancreatic cyst surveillance programme, patients should have a clear understanding of programmatic risks and benefits”.3 The application of the GRADE approach for developing formal recommendations is impratical.4 The GRADE classification of ‘low’ or ‘very low’ quality due to low or very low certainty evidence (on account of indirectness) is not appropriate, given the high level of certainty in the net benefits resulting from desirable effects obviously outweighing undesirable ones. Despite explicitly acknowledging the application of GRADE to this type of scenario, as a limitation of the approach in the first paper of the GRADE JCE series,4 guideline developers have frequently misclassified GPS as ‘strong recommendations with low or very low certainty evidence’.5 6
Subsequently, the GRADE Working Group issued guidance for recognising GPS and their appropriate use.7 The guidance proposes the following five criteria for considering GPS: (1) the statement should be clear and actionable; (2) the message should be necessary for actual healthcare practice (ie, without the guidance, clinicians would fail to make the appropriate decision); (3) after considering all relevant outcomes and potential downstream consequences, implementing the GPS would result in large net positive consequences; (4) collection and summary of the evidence would be a poor use of a guideline panel’s time and resources; and (5) a well-documented, clear and explicit rationale connecting the indirect evidence should be constructed. In this guidance, GRADE cautions guideline developers from overusing GPS and issuing unintended strong recommendations in the guidance.7
During the development of a living map of COVID-19 recommendations (eCOVID19RecMap, https://COVID-19.recmap.org), we developed a taxonomy and framework for classifying actionable statements, establishing GPS as one of the five different types of recommendations (the others being formal recommendation, informal recommendation, research only recommendation and implementation consideration).8 The taxonomy defined GPS according to the guidance developed by the GRADE Working Group,7 and we proceeded to identify them in the guidelines and evaluate their development accordingly. We found that COVID-19 guideline developers frequently included actionable statements that qualify as GPS. 9 However, these statements were rarely labelled as GPS and lacked explicit information supporting their development.9 10 During this process, we identified ways that the GRADE guidance could be improved and operationalised. Hence, we sought to propose a structured framework to facilitate implementation of the guidance.
Approach
The eCOVID-19RecMap project is an international initiative that developed a living catalogue of COVID-19 guidelines and recommendations.The team has wide representation of guideline developers, healthcare professionals, policymakers, researchers and members of the public from various organisations and institutions (listed on the eCOVID19RecMap website, https://COVID-19.recmap.org/about).
We established a small working group of methodologists and guideline developers within the larger eCOVID-19RecMap team (OD, TL, MWL, ZSP, EP nd HJS). We identified criteria in the GPS guidance7 that were frequently misjudged by guideline developers. To better understand this problem, the team carefully reviewed Chapter 14 (Strong recommendations when the evidence is low quality) of the WHO handbook for guideline development,11 papers discussing the issuance of GPS suggested by team members,12–14 and practical guideline development experiences,15–17 including the European Commission Initiative for Breast Cancer.16 We developed the implementation proposal through three rounds of modifications based on information extracted from the guidelines into the GRADEpro (www.gradepro.org) app through a new module that allows the creation of GPS. We then included the GPS in the RecMap (https://covid19.recmap.org/recommendations?recommendationFormality=gps). We presented an initial draft of the proposal internally to the broader eCOVID19RecMap team and to the McMaster Guideline Interest Group and obtained feedback from officers at the WHO. We then incorporated feedback and held regular conference calls with a technical team to operationalise the guidance, followed by feedback from the authors of this paper.
Proposal to operationalise the implementation of the GRADE guidance
Scope
This proposal can aid guideline developers in identifying the need to develop GPS. If a GPS is warranted, we provide a structured process for their development and a standardised table for the transparent reporting of the development process (table 1). All the criteria must be met for the GPS to be developed. The operationalisation of the guidance is intended for all health guideline topics that can arise both in an emergency situation, such as the COVID-19 pandemic, as well as in non-emergency settings.
We reiterate the need to consider the five criteria proposed by the GRADE Working Group for developing GPS7 and suggest the following in the structured process: (1) apply the criteria in a specific order that aligns with the processes of guideline development; (2) encourage guideline developers to consider relevant evidence-to-decision (EtD) criteria when assessing the net consequences of implementing the statement18; and (3) document and publish information supporting judgements for each criterion using the provided standardised table (eg, as a stand-alone annex to the guideline). We illustrate the operationalisation of the GRADE criteria for the development of a GPS (citing an example of drug interactions in patients with COVID-19) (box 1) before we present the approach. To emphasise the need for operationalisation of the GRADE guidance, we provide examples of appropriately developed GPS that cover a wide variety of situations where guidance is required in box 2 and examples of inappropriately developed GPS in box 3. In box 3, we provide an explanation for only one of the five criteria that fails for each statement to illustrate that, in its failure, the statement is rendered inappropriate for development, even if it satisfies other criteria.
Example for application of the Grading of Recommendations, Assessment, Development and Evaluation criteria for the development of good practice statements (GPSs)
Guideline: National Institutes of Health - Coronavirus Disease 2019 (COVID-19) Treatment Guidelines (21 January 2021 version)
Statement: Any time anticoagulant or antiplatelet therapy is being used in patients with COVID-19, consideration must be given to potential drug–drug interactions with other concomitant drugs.
Topic: treatment
Message is really necessary with regard to actual healthcare practice.
Supporting information: Patients with COVID-19 requiring anticoagulants may be on other medications due to underlying comorbidities, severity of illness and for other reasons. In non-COVID-19 situations also, it is prudent that drug interactions should be considered when anticoagulants are prescribed/used because there can be harmful (even life-threatening) consequences of some interactions.
Implementing the GPS results in a large net-positive consequence after consideration of all relevant outcomes and potential downstream consequences.
Supporting information: Consideration of drug–drug interactions would likely lead to avoiding adverse events or mortality and is unlikely to cause any harm.
Collecting and summarising the evidence is a poor use of a guideline panel’s limited time, energy or resources. The opportunity cost of collecting and summarising the evidence is large and can be avoided.
Supporting information: There is overwhelming evidence that failure to consider drug interactions for people on anticoagulant therapy in or for various clinical conditions can have harmful (even fatal) consequences. There is no reason to believe that the situation would be different for patients with COVID-19. Reporting adverse event is often not conducted in formal studies. Therefore, the collection and summarisation of the evidence would be a poor use of time.
There is a well-documented clear and explicit rationale connecting the indirect evidence.
Supporting information: The rationale is that patients with COVID-19 are likely to be on several drugs; therefore, accounting for drug–drug interactions would likely reduce risk of adverse events or mortality.
Statement is clear and actionable. This criterion applies to any developed statement—all formal recommendations are clear and actionable.
Supporting information: The statement is actionable with clear instructions on what is needed (investigating drug–drug interactions) in which setting (when prescribing anticoagulants to patients with COVID-19).
This information is adapted from the guideline and was not presented accordingly.
Examples of appropriately developed good practice statements
Planning and monitoring: To minimise risk to communities, each vaccination team should practise on-site waste segregation and implement reverse logistics, where healthcare waste is taken back to the facility by the vaccination team to be disposed of properly.From WHO—Guidance on developing a national deployment and vaccination plan for COVID-19 vaccines: interim guidance.
Infection control: Cleaning should progress from the least soiled (cleanest) to the most soiled (dirtiest) areas, and from the higher to lower levels so that debris may fall on the floor and is cleaned last in a systematic manner to avoid missing any areas.From WHO—Cleaning and disinfection of environmental surfaces in the context of COVID-19.
Screening: In hospitalised patients with confirmed COVID-19, repeated upper respiratory tract and lower respiratory tract samples can be collected to demonstrate viral clearance.From WHO—Clinical management of severe acute respiratory infection (SARI) when COVID-19 disease is suspected: interim guidance, 13 March 2020.
Treatment: High-flow nasal oxygen therapy can be effectively used in patients with types I and II respiratory failure.From WHO—Effectiveness of different forms of oxygen therapy for COVID-19 management.
Diagnosis: In patients with advanced HIV with suspected or confirmed SARS-CoV-2 infection, clinicians should consider a broader differential diagnosis for clinical symptoms and consider consultation with an HIV specialist.From National Institute of Health (NIH)—Coronavirus Disease 2019 (COVID-19) Treatment Guidelines (21 January 2021 version).
Vaccination: According to the Public Health Agency of Canada, individuals with suspected, probable or confirmed COVID-19 and those who are close contacts of a case should defer influenza vaccination during their period of quarantine or isolation.From Public Health Agency of Canada (PHAC)—Guidance on the use of influenza vaccine in the presence of COVID-19.
Health service and systems: Since most immunisation staff members are occupied in the COVID-19 response, the availability of staff members and the distribution of their duties need to be reviewed carefully.From Western Pacific Regional guide for the immunisation programme and vaccine-preventable disease surveillance during the COVID-19 pandemic.
Examples of inappropriately developed good practice statements
Vaccination: Countries are strongly urged to execute only high-quality preventive vaccination campaigns that can be conducted under safe conditions, without undue harm to health workers and the community when considering lifting any temporary suspensions on preventive mass vaccination campaigns.From WHO—Framework for decision-making: implementation of mass vaccination campaigns in the context of COVID-19.Explanation: Statement not clear and not necessary for healthcare practice. The statement does not provide a clear indication of what high-quality means and does not call for action that may not be usually conducted in usual practice. Therefore, without this guidance, clinicians and policymakers will be able to make the proper decision.
Infection control: Employers may consider allowing exposed and asymptomatic critical infrastructure workers to continue to work in select instances when it is necessary to preserve the function of critical infrastructure workplaces.From COVID-19 Critical Infrastructure Sector Response Planning.Explanation: Implementing the statement may not result in large net-positive consequences after consideration of all relevant outcomes and potential downstream consequences. Reintegrating exposed workers who are not experiencing any symptoms and who have not tested positive back into onsite operations may carry considerable risk to other workers because many people with asymptomatic COVID-19 can still spread the disease, and tests are imperfect, yielding a not negligible number of false negatives.
Diagnosis: It is recommended that contacts without symptoms are tested as soon as possible after being traced, to enable early identification of any asymptomatic or presymptomatic secondary cases among contacts and to start further contact.From European Centre for Disease Prevention and Control (ECDC)—Contact tracing: public health management of persons, including healthcare workers, who have had contact with COVID-19 cases in the European Union, Third Update.Explanation: Collecting and summarising evidence would be a good use of the guideline panel’s time. There is evidence that the role of asymptomatic testing in controlling COVID-19 outbreaks in settings with vulnerable populations is evolving rapidly, and recommendations may need continuous updating. Effects of asymptomatic testing on the general population are a relevant question.
Infection control: Schools do not need to inform parents that pupil or teachers have been removed due to their symptoms.From Health Protection Surveillance Centre—Schools Pathway for COVID-19, the public health approach.Explanation: There was no well-documented clear and explicit rationale (or indirect evidence). Informing parents of students of a potential COVID-19 case allows them to take appropriate actions to protect their children and their close contacts. Therefore, the benefits behind not informing are not clear and require further explanation.
Evaluate the necessity of the message for healthcare practice
The first step that guideline developers should undertake is to frame their recommendation questions of interest by identifying the population, interventions and comparator and including them in the prioritisation process of the guideline project19 (figure 1). The actual evidence synthesis question that results from the recommendation question and serves to compile the evidence will also include the important and critical outcomes to complete the population, intervention, comparator and outcome (PICO) format. The decisions, including practical considerations about timelines and resources available, of the guideline developers will determine the choice between leaving the question out, issuing a graded recommendation or developing a GPS. Guideline developers must provide the rationale for how the statement is relevant and necessary to healthcare practice. Note that, at times, assessing whether the issue is a priority is considered prior to the guideline committee meeting is conducted to examine the evidence to inform a recommendation. Nonetheless, this rationale should be documented. If the statement would lead to an absurd or illogical alternative that does not conform with ethical norms, this signals a possible GPS, and guideline developers should determine if a scoping search for indirect evidence is needed. Lack of resources to systematically identify evidence is insufficient by itself to justify using a GPS to address a recommendation question. In such cases, guideline developers should reconsider where the PICO question situates within the prioritisation process. The availability of direct evidence for the effect of the intervention or lack of a compelling body of linked indirect evidence should discourage the issuance of a GPS and lead to a formal recommendation instead. Note also that over time, as more evidence is available on a given topic, the message provided by a GPS may not become necessary in practice, failing the first GPS criterion (the ‘necessary for healthcare practice’ criterion) and leading to the GPS not being appropriate. For example, issuing guidance for healthcare workers to use protective personal equipment when interacting with patients with COVID-19 mid-pandemic (this would not be true for the type of protective personal equipment of course as this is influenced by the evidence that supports the type of equipment).
Assess the potential consequences of implementing the statement using EtD criteria
When assessing the balance between benefits and harms, implementing the GPS should be clearly expected to yield large net (or large harm) consequences.20 Deliberating these potential consequences of implementing a GPS is crucial for their development, as they are intended to be interpreted as strong recommendations (since there is the underlying assumptions that there is high certainty in the net benefits or net harms). However, since GPSs are intended to be used in a variety of settings, issuing a GPS should not only account for health benefits and harms but should also consider all the other relevant EtD criteria, although it will do this informally.18 For example, the Advisory Committee on Immunisation Practices advises that in the first phase of vaccine distribution, persons aged 75 years and older and essential workers should be prioritised.21 This statement is driven by equity considerations aimed to reduce the burden of COVID-19 on high-risk subpopulations, maintaining maximal capacity of the healthcare system to vaccinate additional people. The urgency and the choice are based on the high mortality in the elderly due to COVID-19, for which there is evidence.22 As to vaccine effectiveness, there is strong high-quality evidence in adults but not in elderly people specifically. Therefore, the former establishes the rationale to develop the statement, while the latter presents the need for indirect evidence to support the statement. Thus, the potential large net desirable consequences should still favour the issuance of a statement as a GPS rather than a formal recommendation. Additional considerations such as assumptions or routinely collected data could be included to make a judgement and must be reported in sufficient detail.
Guideline developers may be reluctant to address all the EtD criteria, but doing so would lead the GPS to a lack of context and to have limited applicability. We suggest that they consider as many EtD criteria as possible to allow easier updating when they revisit the statements or when time or resources are no longer a restricting factor. Reasons for revisiting could be the panel’s judgement or awareness of recent direct evidence for any of the EtD criteria (eg, direct evidence of vaccine effectiveness in the elderly from population-based studies).
Determine if the collection and summary of the indirect evidence would be a poor use of the guideline development group’s time and energy
Our evaluation of GPS shows that very few guideline developers provide an explicit rationale for their decision on not conducting a formal review of the evidence.9 As stated previously, guideline developers should evaluate and document the appropriateness of collecting and summarising the indirect evidence. This decision is of utmost importance, especially in resource and time-constrained settings, such as emergencies, and should be based on the guideline developer’s expectation of only finding indirect evidence. Guideline developers should consider if it is possible to directly investigate the effects of the intervention of interest by contrasting against a direct and appropriate comparator. This may help inform the need to develop a GPS or a formal recommendation, as the comparator of an intervention in a GPS, when made explicit, is bizarre and illogical to choose. Thus, no empirical studies would be conducted to answer such questions directly if the interventions are expected to provide obviously more good than harm.
If the guideline developers decide that a search for evidence is warranted, and it yields only indirect evidence, the guideline panel members should assess if, in their opinon, they believe there is high certainty that the indirect evidence undoubtedly supports net health benefits. If so, conducting a formal review to support the statement with high-certainty indirect evidence would be considered time-consuming and unnecessary.
Identify a well-documented and explicit rationale that connects the statement to the indirect evidence
The use of the standard GRADE approach for GPS is deemed inappropriate as the lack of direct evidence classifies the statement as low or very low certainty, contradicting the high level of certainty in the net benefits which mandates high-quality rating.12 23 Accordingly, GPS are typically supported by several bodies of evidence (with indirect comparisons of the intervention). These bodies of indirect evidence should be linked, meaning they can be interpreted in combination with all inferences regarding the net desirable consequences to make a case for the high level of certainty.24 In turn, the guideline developer should develop a rationale that clearly illustrates the connection betweeen the bodies of indirect evidence. A flow diagram or an analytical framework or pathway is a good way of laying out the evidence is a good way of linking direct to indirect evidence using (Karam et al, Journal of Clinical Epidemiology, in press). This indirect evidence should also be cited and the rationale for why and how the net consequences arise should be provided.
Construct a statement that is clear and actionable
A GPS should be constructed as any recommendation is framed and expressed in actionable terms with the intended intervention, population and setting clearly specified. In the presentation of the GPS, guideline developers should ensure that users are not confused with formally graded recommendations and follow the suggestions made by the GRADE Working Group, using a separate heading for GPS or adding ‘ungraded’ next to the statement.25 However, an important concept in the operationalisation of the guidance is to separate the evaluation of a GPS (answering the question of if the statement is clear and actionable) from the actual development of a GPS. A guideline panel should clearly define the population, interventions and comparator that are the subject of the actionable statement, as they would do for a formal recommendation.
Discussion
Despite available guidance for classifying and developing GPS,4 12 recommendations in guidelines often take the form of GPS. Their rationale and supporting indirect evidence is often poorly explained, and the likelihood of large net benefit associated with the recommendation insufficiently described for users, health professionals and patients. These problems are avoidable and hence represent an ideal target for use of this proposal that has the potential to benefit several stakeholders. We now provide operationalised guidance for developing GPS (figure 1) based on the evaluation of over 800 GPSs on the mapping of COVID-19 recommendations and the gateway to their contextualisation (COVID-19.recmap.org).
Following this structured guidance will aid guideline developers in ensuring that GPSs are appropriately developed and labelled, and judgements supporting their development are transparently reported. This improves the trustworthiness of guidelines and allows clinicians and policymakers to access the best available evidence-based guidance to make their decisions. Documentation supporting the judgements made by the guideline developers using the suggested standardised table will assist researchers in monitoring and evaluating GPS and, if needed, refining the criteria accordingly.
Based on our evaluation of COVID-19 actionable statements, we suggest a GPS evaluation framework in another Dewidar et al 9. The framework consists of the five criteria, but they are arranged in an alternate order to reflect a developers and evaluator’s perspective. As stated by the GRADE Working Group,7 we emphasise the importance of avoiding the inappropriate use of GPS. Given that guideline developers’ time is at a premium, GPSs are intended to help them invest their time and resources where they are most needed. This approach can also significantly help in standardising the visualisation of GPS in guidelines,10 aiding adoption, adaption and development of guidelines. We emphasised elsewhere that GPS should be a result of a prioritisation approach.8
A potential limitation of this guidance is that we do not provide thresholds for judging the extent of indirectness and the incontestability of net beneficial consequences. However, the explicit reporting of the rationale supporting the judgement allows the development of the GPS open to debate. We also do not specify a minimum number of EtD criteria that need to be addressed to influence the judgement on consequences from implementing the statement, but this is in line with guidance for using EtDs. Ideally, guideline developers should consider all the EtD criteria when developing a GPS. However, this may not be feasible in time-constrained settings, and at times, EtD criteria of importance for the context could be prioritised. In the GPS development, we have incorporated a table for guideline developers to indicate which EtD domains were addressed. Guideline developers may later revisit this table, consider the criteria they initially elected not to use in developing the GPS and update the statement accordingly.
Future research can help determine the minimum number of EtD criteria needed to be considered, or specifically which EtD criteria are essential, when developing a GPS and when updating should be considered.
Conclusions
Our guidance builds on previous work conducted by the GRADE Working Group to provide a structured and operationalised process for developing GPS. Using the proposal can aid guideline developers in the development process and in transparently reporting their judgements. Our proposal will inform future updates of GRADE guidance for the development of GPS.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
References
Footnotes
Twitter @okdewidar
Correction notice This article has been corrected since it first published. ORCID has been added for Miloslav Klugar.
Contributors OD, TL, MWL, ZSP, EP and HJS conceived of this paper and designed the first draft of the guidance. OD and HJS presented proposals at the development meetings. OD prepared all the material for the meetings. The coauthors were involved in the meetings and provided feedback on the conceptual approach used in this study. OD and TL collated the feedback and OD prepared the first draft of the manuscript. All authors provided critical review, interpretation and approval of the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. HJS acts as guarantor accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding CIHR (FRN VR4-172741 & GA3-177732) for COVID-19 recommendation mapping. AFT is the Chairholder of the Canada Research Chair in Critical Care Neurology and Trauma.
Competing interests This work was supported by grants from Canadian Institutes of Health (FRN VR4-172741 and GA3-177732) and WHO during the conduct of the study. EA, HJS and PA-C report contribution to the development of the original five criteria for assessing the appropriateness of issuing good practice statements. The remaining authors have nothing else to declare.
Provenance and peer review Commissioned; internally peer reviewed.