Article Text
PDF
Abstract
Objectives Healthcare professionals need to take into account their knowledge, skills and attitudes to develop a focused clinical question, perform an effective search of the literature, critically appraise the evidence, and apply to the clinical context and evaluate the effectiveness of the process. To date, there is a lack of consensus on evidence-based medicine (EBM) curriculum for undergraduate healthcare students in Brazil. The aim of this study was to develop a consensus on EBM curriculum contents for healthcare schools in Brazil considering expert opinion.
Design Modified three-round Delphi methodology.
Setting Online survey.
Participants The expert panel was composed of 40 healthcare professionals from different specialties. Most of the participants (n=24; 60%) were female with the age between 30 and 44 years. Participants were also experts in the field of epidemiology, biostatistics or public health. The mean experience of experts in teaching EBM was 9.5 years.
Main outcome measures An online questionnaire consisting of 89 items related to EBM was sent to the experts. The experts ranked each item of EBM curriculum considering the importance of each item as omitted, mentioned, explained or practised. The last section of the questionnaire was composed of ‘additional content’ where the experts evaluated only if an item should be included or not, the form of offering the EBM contents and the total workload (in hours/semester). Open-ended questions were present in each section to give the opportunity to experts to insert suggestions. Items that reached values greater than or equal to 70% of agreement among experts was considered definitive for the curriculum. Items between 51% and 69% of agreement were included for the next round and those items with less than or equal to 50% of agreement were considered unnecessary and were excluded. In the third round, the EBM contents were classified according to the degree of consensus as follow: strong (≥70% of agreement), moderate (51%–69% of agreement) and weak (50% of agreement) based on the maximum consensus reached.
Results Of the 89 initial contents, 32 (35.9%) reached a strong degree of consensus, 23 (25.8%) moderate degree of consensus, two (2.2%) weak degree of consensus and 35 items were not recommended (≤50% of agreement). The workload suggested by experts should be between 61 and 90 hour/semester and an EBM curriculum should be offered with epidemiology and biostatistics as prerequisites. Regarding the importance of each item, 29 (72.5%) should be explained and 25 (27.5%) should be practised with exercises.
Conclusions The consensus on an EBM curriculum for Brazilian healthcare schools consists of 54 items. This EBM curriculum also presents the degree of consensus (strong, moderate and weak), the importance of each item (mentioned, explained and practised with exercises). A total workload of between 60 and 90 hours per semester was suggested and the EBM curriculum should be offered with epidemiology and biostatistics as prerequisites, but also EBM contents should be included within other disciplines throughout the entire undergraduate course.
- evidence-based practice
- global health
- health
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information. Additional data should be asked by email to felipe.reis@ifrj.edu.br.
Statistics from Altmetric.com
Summary box
What is already known about this subject?
Evidence-based medicine (EBM) concepts should be presented to undergraduate healthcare students.
What are the new findings?
There is a lack of an established consensus on EBM curriculum contents that should be covered in healthcare schools in Brazil.
This study developed a consensus on EBM curriculum including contents, degree of consensus, importance of each item and total workload.
How might it impact clinical practice in the foreseeable future?
This consensus on EBM curriculum can contribute to implementation in healthcare schools.
Introduction
Evidence-based medicine (EBM) is a well-designed problem-solving approach to the delivery of healthcare that integrates the best evidence from well-designed studies, patient’s preferences and values and clinician’s expertise.1 2 Healthcare professionals need to take into account their knowledge, skills and attitudes to develop a focused clinical question, perform an effective search of the literature, critically appraise the evidence and apply to the clinical context and evaluate the effectiveness of the process. On the other side, patients hope that they will receive interventions that had their effectiveness proven through research.3–5 Therefore, it is expected that the incorporation of EBM in healthcare is likely to benefit society by increasing the benefits, reducing the risks of adverse events and being cost-effective.6
Even with its well-recognised benefits, EBM seems not to be the standard of care practised consistently by healthcare professionals.7 In general, there is an important gap between the publication of the evidence and its implementation in clinical practice.8 Several studies highlight multiple barriers to implement EBM by different healthcare professionals including lack of time, inability to comprehend statistical data, lack of employer support, lack of resources, lack of interest, lack of generalisation of the results to the patient, difficulty in obtaining full-text papers and the language of publication.9–12
In order to address some of these barriers and to perpetuate EBM knowledge throughout their professional lives, it is necessary to present to healthcare students EBM principles and applicability during undergraduate training.13 Two studies proposed to identify the main contents to be addressed by an EBM curriculum in healthcare schools.4 14 However, an EBM curriculum consensus based on expert’s opinion that considers the cultural and social aspects, especially non-English and developing countries such as Brazil is still lacking. Blanco et al 15 emphasise that there is a need to obtain a national agreement on the skills required for teaching EBM. Despite some efforts to disseminate, improve and implement EBM education in Brazilian healthcare schools,16 17 to date, there is a lack of an established consensus in respect to EBM content that should be covered, the degree of consensus and the importance of each item for an EBM curriculum. Therefore, we aimed to fill this gap by developing a consensus on EBM curriculum components for healthcare schools in Brazil considering expert opinion.
Methods
Design and ethical aspects of the study
We used a modified Delphi approach to obtain a final consensus of EBM experts using three rounds.18 This study followed the recommendations of Conducting and REporting DElphi Studies.19
Professional identification strategy
In order to compose the expert panel, we searched the largest professional curricula database in Brazil-Curriculum Lattes Platform (http://lattes.cnpq.br/) in July 2018. The Lattes Platform belongs to the National Council for Scientific and Technological Development (CNPq), which aims to integrate Curricula database, Research Groups and Institutions into a single information system. The search strategy considered the following terms: “evidence-based practice”; “evidence-based medicine”, “evidence-based physiotherapy”, “evidence-based psychology”, “evidence-based nursing”, “evidence-based speech therapy”, “evidence-based physical education”, “evidence-based pharmacy”, “evidence-based nutrition”, “evidence-based occupational therapy”, “evidence-based dentistry”, “public health”, “epidemiology”, “biostatistics”, “EBM discipline”, “evidence-based practice discipline”, “evidence-based medicine discipline”, “evidence-based medicine”, “evidence-based practice course”, “EBM course”, “evidence-based clinical practice course”, “evidence-based practice”, “evidence-based practice”, “evidence-based clinical practice”.
Sample size definition
A minimum sample size of 15 experts was defined a priori for each round.20
Selection of the expert panel
The expert panel was composed of healthcare professionals from different specialties who must have significant experience in EBM both in its theoretical and practical concepts considering history of scientific activities, academic and professional training in EBM. To be considered as an expert we established the following criteria: (1) Brazilian nationality, (2) be a healthcare professional, (3) have a doctoral degree and (4) more than 5 years of teaching experience in EBM course and/or have a significant contribution to EBM research (have published an article in peer-reviewed scientific journals related to the topic in the last 3 years). The exclusion criteria were: (1) healthcare professionals who did not have any information registered in the Curriculum Lattes that characterised him as an expert, (2) those who did not respond to the invitation email within 30 days and (3) those who answered the form incompletely. The curricula of each healthcare professional with the potential to compose the expert panel were analysed independently by of two researchers (physiotherapists with knowledge about EBM, and with more than 15 years of experience). Disagreements were resolved by consensus or by arbitration by a third professional from our research team. In order to identify other healthcare professionals that have potential to be included in the expert panel, we used the snowball sampling asking participants to indicate one or more professional.
The invitation to participate in the study was sent by email every 15 days (being days 1, 15 and 30) containing all details of the study including the design, aims and procedures (the estimated time to answer the questionnaire, the importance of completing all rounds and nomination of possible candidates eligible for the study). The email addresses were collected, if available, in the personal description in the Curriculum Lattes platform or searched from publicly available sources (eg, Google search, published articles, organisation websites, disclosure of lectures or courses). When the email was not identified, a message was sent through the ‘contact’ button in the Curriculum Lattes platform. After accepting the invitation, a second email was sent containing the link to access the questionnaire of the respective round. All contacts were invited to the three rounds of the survey, including those who did not respond in the preceding rounds (exception for those who had chosen not to participate).
Modified Delphi procedures
All questions were translated and adapted into Brazilian Portuguese from Albarqouni et al 14 (for more details, see Albarqouni et al 14) by three researchers. First, all questions were translated from English into Portuguese independently by two researchers (FJR and NMF) experienced in EBM. A single version of the questionnaire was obtained by consensus. Then, this version and the original questionnaire were sent to another author (LCN) also experienced in EBM and fluent in English to compare both documents. The final version was obtained through discussion. The online survey was developed using Google Forms platform. At the beginning of each round, all participants completed the sociodemographic and professional data such as age, gender, address, profession, workplace, professional activity, activity in relation to EBM and time of experience with EBM. Subsequent sections included specific topics allocated into categories that reflect the five steps of EBM totalling 89 items: (1) introductory concepts and developing a clinical question, (2) evidence search, (3) evidence appraisal, (4) integration of the evidence into clinical practice and (5) assessment the effectiveness of the process and additional information.
When filling out the questionnaire, experts should select if certain content of the EBM curriculum should be omitted (not a priority to be included in a teaching programme), mentioned (should only be mentioned to provide common knowledge competence), explained (competence should be briefly explained, but without practical exercises) or practised (explained and practised with exercises). The additional content should be classified on the need to be included for the next round. Experts were free to participate in different rounds, but they were encouraged to participate in all rounds of the study.
The first round aimed to identify the sociodemographic profile of the experts and obtain the maximum frequency of responses to the content organised in seven categories (Introduction, Question, Research, Evaluation, Application, Reassessment and Additional Content). Considering that a universally agreed proportion of consensus does not exist for Delphi studies,18 21 we determined a priori a value greater than or equal to 70% of agreement among experts. When this agreement value was reached, the item was considered definitive for the curriculum. This level of agreement has been considered appropriate in previous studies.21–24 Items between 51% and 69% of agreement were included for the next round. Those items with less than or equal to 50% of agreement were considered unnecessary and were excluded. The final section was composed of ‘additional content’ where the experts evaluated only if an item should be included or not and the form of offering the discipline and total workload (in hours/semester). Open-ended questions were present in each section to give the opportunity to experts to insert suggestions, for example, to reformulate questions and insert new content. The main researcher (GAA) was responsible for organising the answers of the open-ended questions and analyse qualitatively (ie, grouping similar responses to create a list of items). After the analysis, these items were transformed into objective questions and included in the subsequent round in the section of additional content.
The second round consisted of survey feedback, also called controlled feedback, which consists of organising data from the previous round and present to the expert panel. For a better view of the results of the first round, one section was composed of all items that reached agreement of 70% or more, the other included items between 51% and 69% of agreement together with the second most voted option and the last section composed of the items from the additional content that reached agreement between 51% and 69%. The results of the first round about the importance of each item and the total workload were converted into options. In the first section, the expert panel was informed that those items were definitely included unless they have any strong objection to refute this inclusion. The expert panel was invited to judge the second section, the additional content and the method to offer the discipline and the total workload.
The last round consisted of presenting the results from the previous round considering the same criteria. The results of the agreement between 51% and 69%, the importance of each item of each content and workload obtained in the previous round were presented to the expert panel to identify if they strongly oppose the inclusion of these items. This round also aimed to classify the importance of each item (mentioned, explained or practised). In this round, the final EBM curriculum was established based on the degree of consensus as follow: strong (≥70% of agreement), moderate (51%–69% of agreement), weak (equal to 50% of agreement in the final round) and non-consensus (<50% of agreement). The procedures and the questionnaire used in each round were pilot tested in a small sample (n=10) of healthcare professionals with knowledge in EBM, who completed it on their own and provided feedback on content. Questions were revised as needed; once finalised, the survey was converted for web completion using Google form and sent to the experts.
Data management
The data were stored and analysed using the SPSS V.22 for Windows (SPSS). The response options of the expert panel were presented by absolute and relative frequencies. The results of the three rounds were analysed and a suggested EBM curriculum was established. This document was sent to the experts to review and provide final suggestions.
Results
Characteristics of the expert panel
The initial search on the Curriculum Lattes Platform returned 714 curricula vitae. From those, 612 were excluded because they did not meet the inclusion criteria, resulting in a final sample of 102 selected curricula. The analysis of the curricula and nominations by snowball sampling method resulted in 103 candidates. After all invitations, 5 (4.85%) professionals refused to participate in the study, and 54 (52.4%) did not answer. A total of 44 (41.9%) professionals agreed to participate and four did not answer the questionnaire. The final sample of the expert panel was composed of 40 health professionals (figure 1).
Flow diagram representing the selection of the expert panel. EBM, evidence-based medicine.
Most of the participants (n=24; 60%) were female with the age between 30 and 44 years (n=20; 50%) from the southeastern region (n=18; 47.3%). Most of the healthcare professionals who participated were physician (n=14; 37%) and nurses (n=13; 34%) and 33 (80%) of the total participants were experts in the field of epidemiology, biostatistics and public health. The mean experience of experts in teaching EBM was 9.5 years and the median was 8 years. Regarding participation in the rounds, 20 experts (52.6%) participated in the first round, 26 (65%) in the second and 27 (67%) in the third round. The demographic and professional characteristics of the participants are shown in table 1.
Demographic and professional characteristics of the expert panel
Characteristics of each round
In the first round, of the 89 contents judged by the expert panel, 11 (12.3%) reached the predefined consensus level (≥70% of agreement), eight of which were classified as ‘practised with exercises’ and three as ‘explained’. Thirty-five (39.3%) items were included for the second round (51%–69% of agreement) and 23 (25.8%) were excluded (≤50% of agreement). Of 19 items in the section related to additional concepts, two (10.5%) items obtained more than 70% of agreement, seven (36.8%) items were included for the second round (51%–69% of agreement) and ten items (52.6%) did not reach the minimum agreement (≤50%). Regarding the semester workload of an EBM curriculum, ten experts (50%) suggested between 30 and 60 hours, four (20%) between 61 and 90 hours, four (20%) between 90 and 120 hours and two (10%) 200 hours. Suggestions about the workload included the distribution of EBM contents where two (10%) experts considered that the EBM content should be distributed throughout the undergraduate course, two (10%) considered that the content should be offered in an EBM specific discipline, one (5%) that there should be an EBM discipline each semester, one (5%) that an EBM discipline should have epidemiology and biostatistics as prerequisites, and one (5%) that the contents should be part of biostatistics and epidemiology.
The second round consisted of a questionnaire with 42 contents. Of these, 17 (40.5%) contents reached the level of consensus (≥70% of agreement), being nine classified as ‘practised with exercises’ and eight as ‘explained’. In respect to the other contents, 15 (35.7%) contents were included for the third round (51%–69% of agreement) and three (7.1%) contents did not reach a consensus (≤50% of agreement). Of seven contents present in the section referring to additional concepts, one (14.2%) content obtained consensus (≥70% of agreement), two (28.6%) contents were included for the third round (51%–69% of agreement) and four (57.1%) contents did not reach a consensus (≤50% of agreement). Considering the suggestions made in the first round about the form of discipline, 15 (60%) believed that the EBM contents should be distributed throughout the undergraduate course, six (24%) considered that the contents should be offered in an EBM discipline having epidemiology and biostatistics as prerequisites and four (16%) consider the EBM contents should be part of epidemiology and biostatistics. The results of the semester workload of an EBM discipline were: 14 experts (53.8%) consider that it should be between 61 and 90 hours, six (24%) between 90 and 120 hours, five (20%) between 30 and 60 hours and one (4%) 200 hours.
The analysis of the second round resulted in a questionnaire composed of 10 contents in the third round. From those, three (30%) reached a strong agreement (≥70%), two (20%) were classified as ‘explained’, and one (10%) as ‘practised with exercises’; four (25%) reached moderate consensus (51%–69% of agreement) being three classified as ‘explained’ and one as ‘practised with exercises’; two (20%) were classified as weak consensus (50% of agreement) and one (10%) did not reach a consensus (<50% of agreement) and was excluded from the final suggestion of an EBM curriculum. Two questions with additional information involving the method of offering the EBM curriculum were included in this round according to the suggestions in the previous round. In one question, 22 (81.4%) experts believe that an EBM curriculum should be offered with epidemiology and biostatistics as a prerequisite. Figure 2 shows the results for each round.
Flow diagram of the development process of the set of basic contents of evidence-based medicine curriculum.
Establishment of consensus on contents for an EBM curriculum
The final consensus resulted in a suggestion of an EBM curriculum with 54 contents; three additional topics regarding the course workload and the importance of each item. The degree of consensus was classified as strong in 32 (35.9%) contents, moderate in 23 (25.8%), weak in two (2.2%) and 35 (39.3%) initial contents did not reach consensus and were not included. Regarding the importance of each item, the expert panel suggested that 29 items (72.5%) should be explained and 25 (27.5%) should be practised.
The consensus obtained in this study of the EBM curriculum for Brazilian healthcare schools is presented in online supplemental appendix A. The EBM curriculum is composed of Introductory EBM concepts (four items) and contents of the EBM five-step model as follow: concepts for question (two items), searching the literature (five items), critical appraisal (37 items), application (four items), evaluation (two items) and additional information (three items) (figure 3). Online supplemental table 2 shows the frequency of responses to conduct each item of the EBM curriculum. Online supplemental table 3 presents the suggestion about the additional contents to be included in the EBM curriculum.
Supplemental material
Supplemental material
Evidence-based medicine (EBM) curriculum contents according to experts.
Discussion
We aimed to develop a consensus on EBM curriculum contents for undergraduate courses in Brazilian healthcare schools. The final consensus on EBM curriculum is composed of 54 items and three additional contents with three degrees of consensus, strong (32), moderate (23) and weak (2). The consensus on the total workload was 61–90 hours per semester and epidemiology and biostatistics should be considered as prerequisites.
To the best of our knowledge, there are two studies that aimed to establish a consensus for EBM curriculum in the literature. One of the studies was based on seven steps of EBM (step 0: maintain a questioning attitude; step 1: question; step 2: search; step 3: evaluate; step 4: apply; step 5: reassess and step 6: disseminate), and the 24 competencies were designated for nursing in the USA.25 More recently, Albarqouni et al 14 aimed to develop a consensus-based set of core competencies for healthcare professionals in EBM. The authors identified 68 priorities for an EBM curriculum for healthcare professional. The main difference for our study is that while Albarqouni et al 14 also identified the content and the importance of each item, we included the degree of consensus. The establishment of the degree of consensus can provide more flexibility to the implementation of the EBM items in the curriculum. The degree of consensus provided in this study could contribute to the implementation of the EBM curriculum in healthcare schools where the number of free hours in their curriculum and economic resources are limited.
The current consensus on an EBM curriculum is in line with the literature. In order, the Sicily statement proposed that the curriculum framework for EBM should be grounded in the five-step model: asking a clinical question; collecting the most relevant evidence; critically appraising the evidence; integrating the evidence with one’s clinical expertise, patient preferences and values to make a practice decision and evaluating the change or outcome.26 27 However, in this study, the experts’ suggestions were more focused on the critically appraising the evidence (37 items). Indeed, a recent scoping review including 81 studies conducted in several countries (USA, Canada, Norway, Australia, Sweden and others), reported that the majority of evaluated EBM educational interventions focus on collecting the most relevant evidence (EBM step 2), critically appraising evidence (EBM step 3) and integrating the evidence with one’s clinical expertise (EMB step 4). Last attention was given to asking a clinical question (EBM step 1) and evaluating the change or outcome (EBM step 5).28 Other previous studies have found similar results in the UK,29 the USA and Canada.15 Apparently, evaluating the change or outcome (EBM step 5) seems to be more theoretically linked at the undergraduate level and should be considered to take place at a more advanced level.30 During the implementation of an EBM curriculum, educators/healthcare professionals should be aware that a small number of EMB programmes taught content that covers all five EBM steps. If EBM educational interventions remain focused on teaching how to locate and appraise evidence, evidence from researches may be poorly translated into clinical practice.30
The literature provides different educational interventions of varying duration, frequency and forma to implement an EBM curriculum. The consensus on EBM curriculum identified in this study is flexible considering the degree of consensus (strong and moderate) and should be adjusted according to the main curriculum. In addition, the EBM curriculum also provides useful information related to the importance of each item (mentioned, explained or practised) reflecting the degree of details for each item. The precise moment that an EBM curriculum should be introduced remains unclear. Some authors supported the EBM should be included in the first year,31 but others supported its introduction at a later stages32 and in association with clinical practice.33 34 There is also a lack in the literature about the most effective strategy to develop EBM knowledge and skills in undergraduate healthcare students. For example, research courses and workshops conducted in classrooms are the most frequently used methods for teaching EBM and information technology (mobile devices, video resources and websites),35 participation in research projects, ‘embedded librarians’ and ‘Journal clubs’ are less common.28 Another significant point to be considered is that the experts considered that an EBM curriculum should has epidemiology and biostatistics as prerequisites. This suggestion is in line with other studies that support the need for students to learn epidemiology and statistics before taking specific EBM contents.36 All this information provided here can be a guide for implementation in different healthcare schools.
The main limitations of this study involved the number of participants in the expert panel and experts from other specialties. Despite the fact we sent the invitation to more than 100 healthcare professionals in each round, only 40% agreed to participate. Although we tried to include different areas of health sciences such as psychology, dentistry, nutrition, physical therapists and exercise and sports scientist to compose the expert panel most members were doctors or nurses. We sought to reduce these limitations expanding the sample recruitment by making contact at three different times in each round and applying the snowball sampling approach. There are some possible explanations: (1) the limited number of professionals from other healthcare specialties considered experts in EBM according to our inclusion criteria (ie, have a doctoral degree and more than 5 years of teaching experience in EBM course and/or have a significant contribution to EBM research), (2) some professional data should be outdated in the Curriculum Lattes database and (3) the emails registered in the Lattes platform are not the main contact and it was not possible to find the email using another method (Google search or publications). Regardlessof this limitation, we believe that it does not interfere in the results, since a representative sample can be evaluated by the experience in the field of the expert panel and not by the number of professionals.37 Last, we did not investigate the teaching method that should be more effective to teach EBM according to the experts. Future studies should investigate the feasibility of implementing this EBM curriculum, method of teaching as well as modifications on students’ satisfaction, knowledge, skills and attitudes.
Conclusion
The consensus on an EBM curriculum for Brazilian healthcare schools consists of 54 items. This EBM curriculum also presents the degree of consensus (strong, moderate and weak), the importance of each item (mentioned, explained and practised with exercises). A total workload of between 61 and 90 hours per semester was recommended and the EBM curriculum should be offered with epidemiology and biostatistics as prerequisites, but also EBM contents should be included within other disciplines throughout the entire undergraduate course.
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information. Additional data should be asked by email to felipe.reis@ifrj.edu.br.
Ethics statements
Patient consent for publication
Ethics approval
All ethical principles were followed and the study was approved by the Ethics and Research Committee of the Instituto Federal do Rio de Janeiro (CAAE 83262217.5.0000.5268).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @lcos3060
Contributors All authors contributed to the development of this study including data collection, data analysis, text, text review and supervision.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.