Successes, lessons and opportunities: 15-year follow-up of an integrated evidence-based medicine curriculum
===========================================================================================================

* Christina S Korownyk
* G Michael Allan
* James McCormack
* Adrienne J Lindblad
* Samantha Horvey
* Michael R Kolber

## Abstract

In 2005, the Department of Family Medicine at the University of Alberta introduced an evidence-based practice curriculum into the 2-year Family Medicine Residency Program. The curriculum was based on best available evidence, had multiple components and was comprehensive in its approach. It prioritised preappraised summary evidence over in-depth evidence appraisal. This paper describes the lessons learnt over the past 15 years including components that were eventually discontinued. We also discuss additions to the programme including the development of accessible, preappraised, summarised resources. We review the difficulties associated with evaluation and the incorporation of evidence-based practice into all aspects of residency training. Future directions are discussed including the incorporation of shared decision-making at the point of care.

*   general practice
*   evidence-based practice

In 2005, the University of Alberta Evidence-Based Medicine (EBM) programme in the department of family medicine, incorporated a new evidence-based practice (EBP) curriculum into the ongoing Family Medicine Residency program. The curriculum incorporated didactic, small group and point of care teaching. The two primary goals of the EBP curriculum were to (1) train competent self-directed life-long learners with the skills to efficiently keep up to date and (2) cultivate resident’s EBP skills to enable them to solve problems encountered in daily practice.1 These aligned with the overarching family medicine residency goals of ensuring graduates are competent to provide comprehensive care in any Canadian community, prepared for the evolving needs of society and taught based on the best available evidence on patient care and medical education.2 

The Family Medicine Residency Program at the University of Alberta is a 2-year programme, which employs a blended model of block-based and longitudinal integrated clinical learning experiences. The academic portion of the residency programme consists of monthly full-day academic days, with workshops on communication skills, procedural skills and EBP interspersed throughout the 2 years. The EBP curriculum originally included five main components integrated throughout 2 years of the family medicine residency (table 1). These included a 2-day EBP workshop, quarterly 1 hour of didactic lectures, monthly journal clubs, a computerised EBP desktop and one-page Brief Evidence-based Assessments of Research (BEARs).1 

View this table:
[Table 1](http://ebm.bmj.com/content/26/5/241/T1)

Table 1 
Past and current evidence-based practice curriculum

The curriculum has been well described previously1 and addressed all five steps of EBP described in the 2005 Sicily statement.3 It was deliberately designed to address the most significant barrier to incorporating an evidence-based primary care practice, namely, time.4–9 Evidence appraisal skills remain the most frequently taught part of EBP educational interventions10 despite evidence that practicing clinicians rarely practice evidence appraisal,11 12 dedicate very limited time to answering their own clinical questions11–13 and have difficulty accessing timely and reliable resources.4 6 7 14 

Our programme encouraged residents to become more familiar with accessing and using preappraised evidence over performing in depth evidence appraisals of single papers. As EBP teaching has evolved, newer EBP competencies highlight benefits of preappraised resources yet continue to prioritise evidence appraisal.15 The objective of this paper is to highlight successes, lessons and evolution of 15 years of a Family medicine residency EBP programme in the context of contemporary research.

## Successes

The 2-day EBP workshop and 1 hour of quarterly didactic lectures are the most consistently highly rated components of the programme. The goal is to teach principles pertaining to better understand health information (and how it can be misleading) in an engaging and entertaining manner. The general philosophy is to ‘light a fire, not fill the bucket.’16 The workshop consists of lectures on therapeutic, diagnostic and systematic reviews followed by accompanying small groups where students practice these skills performing evidence appraisal on a representative paper for each. The goal is not for the residents to become experts in evidence appraisal, rather to be familiar with the process and how many ways we can be misled. We have replaced traditional database searching techniques with sessions in which students are taught how to find and use preappraised and high-quality summary resources. In this context, preappraised resources are defined as evidence that has been filtered—such that only the most relevant data are reviewed and has been subjected to a rigorous evidence appraisal process by a third party not involved in the development of the evidence. While we do not provide specific tools to assess preappraised resources, residents are provided with a number of examples of high-quality resources and encouraged to approach newly available resources with scepticism and common sense. In particular, preappraised resources should complete the same process of evidence appraisal that residents have been introduced to, saving residents substantial time. They should be free of financial and other potential conflicts of interest. Quarterly 1 hour of didactic lectures during regularly scheduled Family Medicine academic days allow for a brief review of a number of relevant research articles in the context of EBP principles. Papers are generally picked to highlight well-done randomised controlled trials with patient-oriented outcomes that are potentially practice changing. In addition, papers with significant limitations are highlighted as an example of how research can be misleading. These lectures provide examples of what residents should expect from preappraised resources.

BEARs are one-page worksheets that promote the translation of clinical uncertainties into questions and rapid searches.2 Residents complete and present at least four BEARs during their scheduled family medicine clinical rotation. They are to be completed using preappraised evidence at the point of care. BEARs provide the opportunity for learners to incorporate EBP concepts into practice and provide the opportunity for discussion in site-specific rounds. The BEAR template has previously been published17 and our research found that BEARs facilitate the use of a variety of resources in answering clinical questions and 69% of residents reported changing their practice after completing a BEAR.17 

Following the initiation of this curriculum, the EBM team noted that primary care clinicians were generally not involved in their own continuing medication education,18 were overwhelmed with information and industry-funded resources seemed common. We determined that if we are training residents to use reliable, preappraised resources—we needed to ensure that these were available to residents during residency training and upon their graduation. We wanted to demonstrate that incorporation of high-quality evidence into community practice was feasible, even in a busy community practice.

In 2008, we initiated evidence summaries on primary care questions. These ‘Tools for Practice’ (TFP) are one-page evidence synthesis, focusing on highest quality evidence and patient-oriented outcomes (available at: toolsforpractice.ca). All family medicine residents are encouraged to sign up for the one-page evidence summaries at the EBP workshop, which occurs at the beginning of residency. TFPs are free of cost and financial conflict of interest. They are distributed every 2 weeks to residents in the programme in addition to almost 40 000 clinicians worldwide. Residents are also provided with the opportunity to complete an EBP elective during their residency. During the elective, residents are mentored in evidence appraisal while contributing to the development of a future TFP.

## Lessons learned and ongoing debates

The EBP desktop, a site that provided residents with a number of evidence-based resources, was largely unused and eventually removed. Similarly, as residents became increasingly competent at searching for medical information, the librarian session during the workshop (which taught residents how to develop searching strategies) was replaced with a small group session emphasising how to quickly finding reliable answers to clinical questions.

Balancing the ability to perform an in-depth evidence appraisal on one paper versus quickly finding preappraised synthesised resources pertaining to one’s question continues to be debated. In 2005, we replaced the more comprehensive and time-consuming ‘critical appraisal topics’ with BEARs. This change was well received. However, the introduction of primary summarised evidence resources at the monthly journal club is a source of ongoing conversation. Some feel that residents should be using the journal clubs to learn more in-depth evidence appraisal of individual articles. However, we feel that in-depth assessment of a single article does not emulate future practice and in an era of ever increasing medical literature and time pressures, most clinicians favour prefiltered resources that summarise the evidence.19 20 Systematic review publications increased 2728% between 1991 and 2014 with 185 different meta-analyses of antidepressants published between 2007 and 2014.21 Some will argue that this reinforces the need for clinicians to be able to discern misinformation, with a greater focus on evidence appraisal.22 Others suggest we need to effectively triage the information.23 We would argue that prioritisation of time is key in primary care, and time expended in one area results in lost opportunity costs in another.

Our department has been very supportive of the EBM curriculum, dedicating 2 days for the EBM workshop and allowing for a regular schedule of lectures and journal club. The predominant clinical setting of the residency programme means that integration of EBP into all elements of education is largely dependent on the clinical practice of the hundreds of clinicians that contribute to the residency programme. Offers of academic faculty development have not been well attended. In 2012, in collaboration with the university and the provincial family medicine college, we initiated an annual medical education conference on evidence-based practice for community clinicians. New high-quality evidence that can be readily incorporated into primary care is reviewed in a fun and engaging manner. The conference is highly rated and has grown in size every year since its initiation. Many of the attendees are clinical preceptors for the family medicine residents, and it is our hope that the conference can provide teachers with tools and enthusiasm to model the regular incorporation of evidence in their daily practice.

## Evaluation

Evaluation is one of the biggest limitations of our programme. Evaluation of EBM knowledge or EBP is fraught with difficulties. Shaneyfelt identified 104 unique instruments for evaluating EBP education24 and Oude *et al* identified 160 instruments for assessing EBP behaviours.25 The proliferation of assessment tools continues,26 27 suggesting evaluation is complex and there is no clear consensus on how EBP programmes should be evaluated overall.15 

In the original description of the curriculum, we reported improvement in resident attitudes regarding EBP and their comfort with practical application.2 For the first 3 years, the workshop moved knowledge scores (on a 15-question quiz) from 50% (SD 18%) to 75% (SD 20%) (table 2). As this was consistent year to year, we did not continue collection. These findings are consistent with systematic reviews that report an increase in EBP knowledge and attitudes following EBP interventions.28 Although self-reported knowledge may improve, this does not necessarily translate into EBP behaviours.28 

View this table:
[Table 2](http://ebm.bmj.com/content/26/5/241/T2)

Table 2 
Two-day evidence-based practice workshop knowledge scores

Individual components of our curriculum have traditionally been evaluated independently. Evaluations generally included the use of simple rating scales that have been modified over the years and thus are not easily comparable. In addition, there is no clear national standard for EBP evaluation. Family Medicine national exams in Canada do not have EBM specifically identified as part of examination blueprint (personal communication—Brent Kvern Director, Certification and Examinations, College of Family Physicians of Canada). It is not clear if a national standard would improve outcomes, as we are not interested in collecting data on knowledge acquisition that may not result in behaviour change on graduation. We consider the ultimate measure of success to be graduating residents who regularly practice medicine based on best available evidence. Unfortunately, the measurement of this is complex, and we did not have sufficient resources to measure this in residents who had graduated from the programme.

The difficulty is compounded as the primary outcome of EBP remains debated: is EBP intended to improve health on a population level or encourage shared informed decision making, which may allow patients to make decisions that may not be consistent with ‘best evidence?’29 30 If the latter is true, assessments of EBP may be as simple as, ‘were reliable tools used for shared decision-making with the patient?’

## Moving forward

Recently experts in evidence appraisal published the 21st chapter of the GRADE guidelines—tools intended to assist with the assessment of results and certainty of evidence.S31 The need for multiple publications suggests that evidence appraisal has become increasingly complex, and in many ways a moving target. Based on 15 years experience with EBM education, we would advocate for familiarity with, not mastery of evidence appraisal skills in an EBP curriculum for primary care. Even if one were familiar with basic evidence appraisal skills, there are numerous other factors to consider that require further time and investigation (eg, How does this paper fit with others on the topic? How many other trials were registered but not published?) In our personal experience, comprehensive and timely appraisal of health information requires years of regular practice. Thus, we continue to encourage our learners to learn basics of EBM and health literacy, but strongly advocate the use of prefiltered evidence summaries (ideally written by experienced authors without financial conflicts of interest) to answer daily clinical questions.

Over the past 15 years, we have seen a gradual evolution from ‘point of care’ tools to ‘shared informed decision-making’.S32 Improving learner familiarity with shared decision-making tools is essential in moving medicine to a more inclusive decision-making setting. Our team, along with numerous other groups, has developed tools to assist with shared decision-making in clinical practice.S33–37 Increasing learner familiarity with these resources may be an opportunity to improve translation of evidence into practice and we are working to incorporate these as examples of practical evidence application into our curriculum. For example, we are incorporating these more and more into the 1-hour journal club to allow for discussion around their practical implementation into practice.

In order to engage faculty, we continue to grow our annual medical education conference on evidence-based practice for both community and academic clinicians. This medical education event is organised outside of the university setting and is designed to promote EBP and build enthusiasm for teaching and incorporation of EBP into daily practice.

In summary, the current curriculum (table 1) is a result of modifications in response to ongoing feedback and best evidence. The originally stated goals of the EBP curriculum remain relevant, encouraging familiarity of with basic EBM concepts yet emphasising the use of preappraised evidence summaries. Fifteen years later, we would add a third goal—to train residents to optimise patient-oriented outcomes in the context of shared decision-making and individual patient preferences and values.

### Supplementary data

[[bmjebm-2020-111393supp001.pdf]](pending:yes) 

## Ethics statements

### Patient consent for publication

Not required.

## Acknowledgments

The authors would like to acknowledge the Family Medicine Department at the University of Alberta, the Alberta College of Family Physicians and the PEER team for their assistance and support.

## Footnotes

*   Contributors CK, MK and AL conceived of the original idea. MA, SH and MK assisted with data collection. CK wrote the first draft. Draft review and critical revision were completed by JM, MA, SH, MK and AL. All authors gave final approval of the version to be published.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests None declared.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.   Allan GM , Korownyk C , Tan A , et al . Developing an integrated evidence-based medicine curriculum for family medicine residency at the University of Alberta. Acad Med 2008;83:581–7.[doi:10.1097/ACM.0b013e3181723a5c](http://dx.doi.org/10.1097/ACM.0b013e3181723a5c) pmid:http://www.ncbi.nlm.nih.gov/pubmed/18520465 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1097/ACM.0b013e3181723a5c&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=18520465&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

2.   Oandasan I , Working Group on Postgraduate Curriculum Review . Advancing Canada's family medicine curriculum: triple C. Can Fam Physician 2011;57:739–40.pmid:http://www.ncbi.nlm.nih.gov/pubmed/21673223 
    
    [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjg6IjU3LzYvNzM5IjtzOjQ6ImF0b20iO3M6MjA6Ii9lYm1lZC8yNi81LzI0MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.   Dawes M , Summerskill W , Glasziou P , et al . Sicily statement on evidence-based practice. BMC Med Educ 2005;5:1. [doi:10.1186/1472-6920-5-1](http://dx.doi.org/10.1186/1472-6920-5-1) pmid:http://www.ncbi.nlm.nih.gov/pubmed/15634359 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/1472-6920-5-1&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=15634359&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

4.   Zhao Y , Zhao X , Liu Y , et al . Perceptions, behaviours, barriers and needs of evidence-based medicine in primary care in Beijing: a qualitative study. BMC Fam Pract 2019;20:171. [doi:10.1186/s12875-019-1062-0](http://dx.doi.org/10.1186/s12875-019-1062-0) pmid:http://www.ncbi.nlm.nih.gov/pubmed/31810450 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/s12875-019-1062-0&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

5.   Cook DA , Pencille LJ , Dupras DM , et al . Practice variation and practice guidelines: attitudes of generalist and specialist physicians, nurse practitioners, and physician assistants. PLoS One 2018;13:e0191943. [doi:10.1371/journal.pone.0191943](http://dx.doi.org/10.1371/journal.pone.0191943) pmid:http://www.ncbi.nlm.nih.gov/pubmed/29385203 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1371/journal.pone.0191943&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

6.   Hisham R , Liew SM , Ng CJ , et al . Rural doctors' views on and experiences with evidence-based medicine: the freedom qualitative study. PLoS One 2016;11:e0152649. [doi:10.1371/journal.pone.0152649](http://dx.doi.org/10.1371/journal.pone.0152649) pmid:http://www.ncbi.nlm.nih.gov/pubmed/27031700 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1371/journal.pone.0152649&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

7.   Zwolsman S , te Pas E , Hooft L , et al . Barriers to GPs' use of evidence-based medicine: a systematic review. Br J Gen Pract 2012;62:e511–21.[doi:10.3399/bjgp12X652382](http://dx.doi.org/10.3399/bjgp12X652382) pmid:http://www.ncbi.nlm.nih.gov/pubmed/22781999 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjIvNjAwL2U1MTEiO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzI2LzUvMjQxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.   van Dijk N , Hooft L , Wieringa-de Waard M . What are the barriers to residents' practicing evidence-based medicine? A systematic review. Acad Med 2010;85:1163–70.[doi:10.1097/ACM.0b013e3181d4152f](http://dx.doi.org/10.1097/ACM.0b013e3181d4152f) pmid:http://www.ncbi.nlm.nih.gov/pubmed/20186032 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1097/ACM.0b013e3181d4152f&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=20186032&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000279378400018&link_type=ISI) 

9.   McColl A , Smith H , White P , et al . General practitioner's perceptions of the route to evidence based medicine: a questionnaire survey. BMJ 1998;316:361–5.[doi:10.1136/bmj.316.7128.361](http://dx.doi.org/10.1136/bmj.316.7128.361) pmid:http://www.ncbi.nlm.nih.gov/pubmed/9487174 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTYvNzEyOC8zNjEiO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzI2LzUvMjQxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10.  Albarqouni L , Hoffmann T , Glasziou P . Evidence-Based practice educational intervention studies: a systematic review of what is taught and how it is measured. BMC Med Educ 2018;18:177. [doi:10.1186/s12909-018-1284-1](http://dx.doi.org/10.1186/s12909-018-1284-1) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30068343 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/s12909-018-1284-1&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

11.  Ely JW , Osheroff JA , Ebell MH , et al . Analysis of questions asked by family doctors regarding patient care. BMJ 1999;319:358–61.[doi:10.1136/bmj.319.7206.358](http://dx.doi.org/10.1136/bmj.319.7206.358) pmid:http://www.ncbi.nlm.nih.gov/pubmed/10435959 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTkvNzIwNi8zNTgiO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzI2LzUvMjQxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12.  Ramos K , Linscheid R , Schafer S . Real-Time information-seeking behavior of residency physicians. Fam Med 2003;35:257–60.pmid:http://www.ncbi.nlm.nih.gov/pubmed/12729309 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=12729309&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000182521000012&link_type=ISI) 

13.  Del Fiol G , Workman TE , Gorman PN . Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Intern Med 2014;174:710–8.[doi:10.1001/jamainternmed.2014.368](http://dx.doi.org/10.1001/jamainternmed.2014.368) pmid:http://www.ncbi.nlm.nih.gov/pubmed/24663331 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

14.  Muscat DM , Patel P , Reid S , et al . Content analysis of clinical questions from Australian general practice which are prioritised for answering: identifying common question types and perceived knowledge gaps. BMJ Evid Based Med 2020;25:15–21.[doi:10.1136/bmjebm-2019-111210](http://dx.doi.org/10.1136/bmjebm-2019-111210) pmid:http://www.ncbi.nlm.nih.gov/pubmed/31235528 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiZWJtZWQiO3M6NToicmVzaWQiO3M6NzoiMjUvMS8xNSI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjYvNS8yNDEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15.  Albarqouni L , Hoffmann T , Straus S , et al . Core competencies in evidence-based practice for health professionals: consensus statement based on a systematic review and Delphi survey. JAMA Netw Open 2018;1:e180281. [doi:10.1001/jamanetworkopen.2018.0281](http://dx.doi.org/10.1001/jamanetworkopen.2018.0281) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30646073 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

16. Quote investigator. Available: [https://quoteinvestigator.com/2013/03/28/mind-fire/](https://quoteinvestigator.com/2013/03/28/mind-fire/) [Accessed 19 Mar 2020].
    
    

17.  Korownyk C , Ross S , Ma V , et al . A descriptive analysis of a novel intervention to help residents become evidence users. Med Teach 2013;35:e1546–50.[doi:10.3109/0142159X.2013.786807](http://dx.doi.org/10.3109/0142159X.2013.786807) pmid:http://www.ncbi.nlm.nih.gov/pubmed/23799951 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.3109/0142159X.2013.786807&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=23799951&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

18.  Klein D , Allan GM , Manca D , et al . Who is driving continuing medical education for family medicine? J Contin Educ Health Prof 2009;29:63–7.[doi:10.1002/chp.20008](http://dx.doi.org/10.1002/chp.20008) pmid:http://www.ncbi.nlm.nih.gov/pubmed/19288556 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1002/chp.20008&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=19288556&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

19.  Hoogendam A , Stalenhoef AFH , Robbé PFdeV , et al . Answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed. J Med Internet Res 2008;10:e29. [doi:10.2196/jmir.1012](http://dx.doi.org/10.2196/jmir.1012) pmid:http://www.ncbi.nlm.nih.gov/pubmed/18926978 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.2196/jmir.1012&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=18926978&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

20.  Alper BS , White DS , Ge B . Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med 2005;3:507–13.[doi:10.1370/afm.370](http://dx.doi.org/10.1370/afm.370) pmid:http://www.ncbi.nlm.nih.gov/pubmed/16338914 
    
    [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiMy82LzUwNyI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjYvNS8yNDEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21.  Ioannidis JPA . The mass production of redundant, misleading, and Conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485–514.[doi:10.1111/1468-0009.12210](http://dx.doi.org/10.1111/1468-0009.12210) pmid:http://www.ncbi.nlm.nih.gov/pubmed/27620683 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1111/1468-0009.12210&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=27620683&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

22.  Ioannidis JPA , Stuart ME , Brownlee S , et al . How to survive the medical misinformation mess. Eur J Clin Invest 2017;47:795–802.[doi:10.1111/eci.12834](http://dx.doi.org/10.1111/eci.12834) pmid:http://www.ncbi.nlm.nih.gov/pubmed/28881000 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1111/eci.12834&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=28881000&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

23.  Prasad V . 21St century physician: triaging the tsunami of medical information, 2018. Available: [https://www.medscape.com/viewarticle/902209#vp\_2](https://www.medscape.com/viewarticle/902209#vp_2) [Accessed 19 Mar 2020].
    
    

24.  Shaneyfelt T , Baum KD , Bell D , et al . Instruments for evaluating education in evidence-based practice: a systematic review. JAMA 2006;296:1116–27.[doi:10.1001/jama.296.9.1116](http://dx.doi.org/10.1001/jama.296.9.1116) pmid:http://www.ncbi.nlm.nih.gov/pubmed/16954491 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1001/jama.296.9.1116&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=16954491&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 
    
    [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000240298400013&link_type=ISI) 

25.  Oude Rengerink K , Zwolsman SE , Ubbink DT , et al . Tools to assess evidence-based practice behaviour among healthcare professionals. Evid Based Med 2013;18:129–38.[doi:10.1136/eb-2012-100969](http://dx.doi.org/10.1136/eb-2012-100969) pmid:http://www.ncbi.nlm.nih.gov/pubmed/23349216 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

26.  Hisham R , Ng CJ , Liew SM , et al . Development and validation of the evidence based medicine questionnaire (EBMQ) to assess doctors' knowledge, practice and barriers regarding the implementation of evidence-based medicine in primary care. BMC Fam Pract 2018;19:98. [doi:10.1186/s12875-018-0779-5](http://dx.doi.org/10.1186/s12875-018-0779-5) pmid:http://www.ncbi.nlm.nih.gov/pubmed/29935527 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/s12875-018-0779-5&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

27.  Rousselot N , Tombrey T , Zongo D , et al . Development and pilot testing of a tool to assess evidence-based practice skills among French general practitioners. BMC Med Educ 2018;18:254. [doi:10.1186/s12909-018-1368-y](http://dx.doi.org/10.1186/s12909-018-1368-y) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30413196 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1186/s12909-018-1368-y&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

28.  Saunders H , Gallagher-Ford L , Kvist T , et al . Practicing healthcare professionals' evidence-based practice competencies: an overview of systematic reviews. Worldviews Evid Based Nurs 2019;16:176–85.[doi:10.1111/wvn.12363](http://dx.doi.org/10.1111/wvn.12363) pmid:http://www.ncbi.nlm.nih.gov/pubmed/31074582 
    
    [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1111/wvn.12363&link_type=DOI) 
    
    [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Febmed%2F26%2F5%2F241.atom) 

29.  McCormack J , Elwyn G . Shared decision is the only outcome that matters when it comes to evaluating evidence-based practice. BMJ Evid Based Med 2018;23:137–9.[doi:10.1136/bmjebm-2018-110922](http://dx.doi.org/10.1136/bmjebm-2018-110922) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30002077 
    
    [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiZWJtZWQiO3M6NToicmVzaWQiO3M6ODoiMjMvNC8xMzciO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzI2LzUvMjQxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

30.  Bell NR , Thériault G , Singh H , et al . Measuring what really matters: screening in primary care. Can Fam Physician 2019;65:790–5.pmid:http://www.ncbi.nlm.nih.gov/pubmed/31722909 
    
    [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjk6IjY1LzExLzc5MCI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjYvNS8yNDEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)