Article Text
PDF
Statistics from Altmetric.com
Introduction
Since the 1980s, evidence-based medicine has been considered best practice, and evidence-based guidelines have been developed by national and international organisations. Alongside the evidence-based movement, societies and experts are developing and publishing consensus-based guidance, often labelled as consensus statements, committee opinions or recommendations. These documents provide guidance for areas with insufficient evidence-base, a high level of uncertainty and hence a large variation in practice. Both evidence-based and consensus-based guidance documents are highly relevant, but the distinction should be clear.
Evidence-based guidelines are commonly developed according to fixed methodologies, written in a manual for development and incorporating or referring to international standards, like the GRADE methodology.1 The strictness in the development of evidence-based guidelines is in strong contrast with the lack of structured methodologies and quality standards for consensus-based documents. Moreover, these documents are sometimes labelled as guidelines, which makes the reader wrongly assume that the formulated recommendations have been developed based on a standardised methodology and are supported by the scientific literature.
The European Society of Human Reproduction and Embryology (ESHRE) has recently developed a manual for the development of good practice recommendations, which are consensus-based, and are meant to complement the societies’ evidence-based guidelines.2 The reason for developing and formulating a procedure for the drafting of an ESHRE recommendation is to harmonise the different steps in such a project. This will lead to the development of ESHRE recommendations according to a structured approach and transparently labelled as consensus-based recommendations. In addition, recommendations for good practice could be instrumental in guiding research policies and priorities.
Rationale for recommendations for good practice
Evidence-based guidelines are primarily based on high-quality evidence and are appropriate for areas where such evidence is available for most of the guideline’s key questions.3 Guideline groups can formulate strong and conditional recommendations, depending on the quality of the supporting evidence and other factors including patient perspective, healthcare context and clinicians’ expertise.
In contrast, recommendations for good practice are more applicable in areas where there is an opportunity to reduce uncertainty and improve quality of care, but where evidence for most aspects is absent or limited. Topics for recommendations for good practice are different and often more practically oriented than these for evidence-based guidelines. Due to the limited scientific evidence and the more practical topics, applying an evidence-based guideline methodology to recommendation papers is often impossible, and a modified approach based on expert opinions is required.
A ‘recommendations for good practice’ document provides an opportunity for guidance when an evidence-based approach is not feasible. However, it needs to be stressed that this consensus-based approach is complementary to and does not replace the evidence-based approach. To ensure that only eligible topics are selected for recommendations papers and to distinguish from the ones that can be developed as evidence-based guidelines, the methodology includes an exploratory evidence search and a formal assessment and approval of any suggested topic by the ESHRE Executive Committee.
Development of recommendations for good practice
The development of recommendations for good practice is summarised in figure 1 and discussed ahead. These steps for development are based on the methodology for evidence-based guidelines. It is however adapted to the specificities of a recommendation using the methodology of consensus documents published by our Society and others.3–5 The similarities and differences in the development of recommendations for good practice and evidence-based guidelines are summarised in table 1.
Comparison of the key features of guidelines and recommendations for good practice
Summary of the steps for development of recommendations for good practice. ExCo, executive committee; RS, research specialist; SIG, special interest group; WG, working group.
Composition of a WG
The first step, after the Executive Committee accepts a project for development, is to set up a working group (WG) of 8–10 experts with hands-on experience, representing different geographical regions, different levels of expertise and different employment situations. When composing a group, the emphasis is put on covering all types of expertise (different settings, different countries, different laws, different preferred ways of working) rather than the numbers. All WG members should agree to the following responsibilities: (1) attend all meetings, online and in-person, (2) sign a statement of confidentiality at the start of the project, (3) declare of any conflict of interest, (4) actively contribute to the discussions, with acceptance and tolerance of varying viewpoints, and (5) approve of the final recommendations.
The chairperson of the WG should be a respected expert, with experience on the topic and in group facilitation. In addition, a research specialist is assigned to perform administrative tasks, perform literature searches and to ensure that deadlines and procedures are respected. Patient representatives, allied health providers or non-expert clinicians are generally not involved for recommendations documents and industry representatives are excluded from participation.
Scope and outline of the document
The first task of the WG is to define the scope of the document and determine the outline. To define the scope of the document, it is recommended to consider the following questions:
What is/are the overall purpose(s) of the proposed recommendations?
What are the proposed target patient population and healthcare setting?
Which interventions should be included?
Who are the target users of the proposed recommendations and who are the key stakeholders?
What is the relation to other documents?
In addition to defining which interventions/procedures/tests will be included in the document, it is recommended to write the outline of the paper. This will further aid to define which topics will or will not be included. Furthermore, a clear structure will form the basis of the entire work and avoid misinterpretation.
At this stage, the WG is also encouraged to agree on the specific methodology for data collection (survey and literature search) and set a timeline.
Preparing a draft
The WG should discuss and decide which of the recommendations could or should be supported by a formal evidence synthesis, and which can be supported only by an expert opinion. If a formal literature search and evidence synthesis are relevant, the WG should define the questions to be answered. The research specialist performs the literature searches and evidence synthesis and provides the WG members with the raw data and a summary of the evidence. The methodology used for the evidence search and synthesis is the same as for evidence-based guidelines. A literature search should also be performed to collect observational data, for instance, on the variation in clinical practice, to support an expert opinion in certain recommendations. Observational data can also be collected through a survey among practitioners. The remaining recommendations are based on expert opinion only. Examples of how data collection was approached in recent recommendations documents from our society are listed in table 2.
Examples of how data collection was approached in recent consensus documents from our society
Based on their expert opinion, available (unpublished or published) observational data and, if performed, the results of the literature search, the WG members prepare a first draft of the recommendations document. The first draft is a starting point for discussion.
WG consensus meeting(s)
During one or more consensus meetings (online or in-person), the WG members discuss the draft with the aim of reaching consensus in the group on each of the recommendations.
During the meetings, the author of each section is asked to read the document, explaining the reasoning behind the recommendations and indicating areas of uncertainty. The WG members discuss whether they agree, and if not, how to modify the recommendation. In general, informal methods for reaching consensus are applied, but if no consensus can be reached, formal methods could be used (eg, a modified Delphi approach). If the group is unable to reach consensus, this should be explicitly stated in the document and explained.
At the end of the meeting, the final list of recommendations is sent to the WG members for written comments. At this stage, WG members are asked to check whether the recommendations are well formulated, specific, unambiguous and feasible. Furthermore, inconsistencies between the recommendations should be considered, as well as the inconsistent use of terminology and differences in writing style. The chairperson of the WG and the leading authors of the sections review the written comments and address them. Modifications to the text that could affect the content (rather than linguistic corrections and improvements) should be agreed on by all WG members.
Stakeholder consultation
To ensure that a recommendation document is acceptable for end users, a stakeholder consultation is organised. The stakeholder review allows for the input from additional experts and non-experts. In addition to verifying the acceptability and adequacy of the document, stakeholder consultation can also highlight issues with the methodological quality, the language, its unambiguity, the feasibility of the recommendations, and any conflicts of interest that could have influenced the recommendations. At the start of the consultation phase, the draft is web-posted with an open invitation to review. Furthermore, stakeholders are invited to review by email. Reviewers must sign a statement of confidentiality and submit their comments within 4 weeks using the reviewer comments form. All reviewers’ names and their comments are tabulated in a stakeholder review report. The WG does not commit to altering the recommendations in all instances. However, if no change is made, the reason is also recorded in the report. The stakeholder review report is published on the ESHRE website.
Approval
After stakeholder review, all WG members and the executive committee are formally asked to approve the final version of the recommendations for good practice document.
Publication
On approval, the recommendations document is published, disseminated and promoted through available channels. To aid end users in applying the recommendations, implementation tools can be developed, for example, a pocket version of the document or an online platform with educational materials including videos, pictures and tutorials.
Developing standards: alternative methodology
When defining parameters for the laboratory or clinic, such as performance indicators, or when defining a classification system, it may be relevant to ensure that the recommendations are accepted and agreed on by a larger group of experts, rather than within the WG. To achieve the input of a larger group of stakeholders, the WG can prepare an expert panel meeting to which representatives of different countries, (national or international) organisations and/or disciplines are invited.
The approach to such documents is similar concerning topic selection, WG composition and definition of the scope and outline. However, at an expert panel meeting, the WG members are asked to present their findings from the literature and expertise, after which the details are further discussed until consensus is reached within the expert panel. A chairperson facilitates discussion and ensures consensus is reached. After the meeting, the workshop report and the recommendations of the expert panel are written down by the research specialist and/or the WG members. Another approach to include the input of a larger group of stakeholders is the development of a Delphi survey, where the stakeholders share their opinion in an iterative survey. Between each survey, they are given feedback on the responses of the entire group, which may change their views towards consensus.6 After the expert panel meeting or Delphi survey, the document is sent for stakeholder consultation and approval, although for documents where a large group of stakeholders was involved in the development phase, it can be acceptable not to organise an additional stakeholder consultation.
Discussion
Although consensus-based documents are widely produced in various fields of knowledge and their development is often supported by professional societies, their methodology is usually not standardised nor well documented. This is in contrast with evidence-based guidelines for which rigour of development is a criterion of quality (AGREE),7 and several professional societies have published a manual.4 8 9 In addition, several tools and checklist have been developed to assess guideline quality, methodological research is conducted and the quality of evidence-based guidelines is improving.10
Terminology for guidelines and consensus documents are often overlapping. A significant proportion of published guidelines from professional organisations are in fact consensus-based; these are not evidence-based nor intended to be. However, by labelling them as ‘guidelines’, these documents could be perceived as evidence-based by end users.
Expert opinion, although included as an important factor in evidence-based recommendations, should not be the sole basis for formulating recommendations in evidence-based guidelines.11 Good practice points, once used in evidence-based guidelines for recommendations with little available evidence, are now restricted to clarification and advice on recommendations, for instance, to emphasise the importance of shared decision-making about specific procedures or advise caution where there is perceived risk of harm but no available evidence.9 Restricting the use of expert opinion-based recommendations results in guidelines that are strongly evidence-based, but not always applicable to clinical practice without further information on how to execute them. In the field of assisted reproduction, for example, practical recommendations on how to perform surgical procedures in endometriosis are being written, based on a perceived need for further practical information in addition to an evidence-based guideline listing indications and different options for surgery.12
Although the current methodology is based on the methods used for recent consensus-based papers from our society, the relevance of having a manual and standardised methodology is not yet validated. Further limitations are the lack of methodological guidance or tools, and the lack of detailed methods sections in existing consensus papers. It is expected that the current manual will be revised based on further experience with the draft of recommendations for good practice in the field of reproductive science and medicine.
The methodology described in this manual for recommendation papers intends to combine the principles of evidence-based guidelines with practical consensus-based expert advice. Labelling the resulting documents as ‘recommendations for good practice’ clarifies the difference in aim and methodology, which fits the principle of transparency as a benchmark in guideline development. In addition, to further strengthen the methodological quality of recommendations for good practice developed by our society, we aim to highlight the relevance and usefulness of such documents, promote transparency on the methods used in consensus documents and encourage progress in the field. Indeed, recommendations for good practice could entail the opportunity to generate and publish evidence in these specific topics.
Acknowledgments
The authors would like to thank the members of the ESHRE Executive Committee for their constructive comments in the development of the manual for recommendations.
Footnotes
Contributors A.d’A, KT and NV: contributed to the design of the work. NV: drafted the manuscript. All authors participated in critical revision of the manuscript and approved the final version.
Funding This work was performed as part of the ESHRE Guideline Development Program.
Competing interests NV and NLC are employed by ESHRE. All authors declare no financial relationships with any organisation that might have an interest in the submitted work in the previous 3 years, no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review Not commissioned; externally peer reviewed.