Article Text
PDF
Abstract
Objectives This research project aims to determine the potential differential impact of two curricular approaches to teaching evidence-based medicine (EBM) on student performance on an EBM assignment administered during the first year of clerkship. A meaningful result would be any statistically significant difference in scores on the assignment given to measure student performance.
Design In order to assess and compare student learning under the different curricula, the principal investigator and a team of five faculty members blinded to assignment date and other possibly identifying details used a modified version of the previously validated Fresno rubric to retrospectively grade 3 years’ worth of EBM assignments given to students in clerkship rotations 1–3 (n=481) during the Internal Medicine clerkship. Specifically, EBM performance in three separate student cohorts was examined.
Setting The study took place at a large Midwestern medical school with nine campuses across the state of Indiana.
Participants Study participants were 481 students who attended the medical school and completed the Internal Medicine clerkship between 2017 and 2019.
Interventions Prior to the inception of this study, our institution had been teaching EBM within a discrete 2-month time period during medical students’ first year. During a large-scale curricular overhaul, the approach to teaching EBM was changed to a more scaffolded, integrated approach with sessions being taught over the course of 2 years. In this study, we assess the differential impact of these two approaches to teaching EBM in the first 2 years of medical school.
Main outcome measures We used clerkship-level EBM assignment grades to determine whether there was a difference in performance between those students who experienced the old versus the new instructional model. Clerkship EBM assignments given to the students used identical questions each year in order to have a valid basis for comparison. Additionally, we analysed average student grades across the school on the EBM portion of step 1.
Results Four hundred and eighty-one assignments were graded. Mean scores were compared for individual questions and cumulative scores using a one-way Welch Analysis of Variance test. Overall, students performed 0.99 of a point better on the assignment from year 1 (Y1), prior to EBM curriculum integration, to year 3 (Y3), subsequent to EBM integration (p≤0.001). Statistically significant improvement was seen on questions measuring students’ ability to formulate a clinical question and critically appraise medical evidence. Additionally, on the United States Medical Licensing Examination (USMLE) step 1, we found that student scores on the EBM portion of the examination improved from Y1 to Y3.
Conclusions Results of this study suggest that taking a scaffolded, curriculum-integrated approach to EBM instruction during the preclinical years increases, or at the very least does not lessen, student retention of and ability to apply EBM concepts to patient care. Although it is difficult to fully attribute students’ retention and application of EBM concepts to the adoption of a curricular model focused on scaffolding and integration, the results of this study show that there are value-added educational effects to teaching EBM in this new format. Overall, this study provides a foundation for new research and practice seeking to improve EBM instruction.
Trial registration number IRB approval (Protocol number 1907054875) was obtained for this study.
- evidence-based practice
- information science
Data availability statement
Data are available in a public, open access repository. The data that support the findings of this study are openly available in figshare. The doi is below. 10.6084/m9.figshare.12003087.
Statistics from Altmetric.com
Summary box
What is already known about this subject?
Evidence-based medicine (EBM) instructional interventions improve medical student skills based on pretest and post-test data.
Instructional approaches to teaching EBM are varied throughout the clinical and preclinical years.
Whether or not medical students retain EBM skills and knowledge is unclear in the long term.
What are the new findings?
Students who experienced an updated version of an EBM curriculum during their first and second years of medical school focused on progressive, scaffolded learning perform better on a measure of EBM competency in clerkships.
Step 1 EBM scores increased when the updated version of the curriculum was implemented.
How might this impact practice in the foreseeable future?
Medical educators and administrators should consider moving away from an isolated or one-shot approach to teaching EBM to medical students.
EBM instruction should be integrated into the existing curriculum and reinforced regularly throughout the preclinical years.
Introduction
Faculties are increasingly being called on to teach medical students the skills necessary to have a strong foundation in evidence-based medicine (EBM) before the students move from the classroom to the clinical phase of their education. All too often, the expectation is that these skills can be imparted in a class or two when the curriculum allows. Rarely is EBM instruction intentionally scaffolded and implemented throughout the curriculum, let alone has the impact of such an approach been assessed.1 2 Recently, the Indiana University School of Medicine (IUSM) had the opportunity to redesign its EBM curriculum. Prior to the inception of this study, IUSM had been teaching EBM within a discrete 2-month time period during medical students’ second year. During a large-scale curricular overhaul, the approach to teaching EBM was changed to a more scaffolded, integrated approach with sessions being taught over the course of 2 years.
In this paper, we assess the updated curriculum in comparison to the previous approach. Our specific question is as follows: Do undergraduate medical students who received an EBM curriculum in a new instructional model focused on scaffolding, critical thinking and longitudinal integration perform better on measures of EBM competency than students who received the (same) curriculum in a prior instructional model?
This research project demonstrates the differential impact of two instructional models for teaching EBM on student performance, measured by grades, on an EBM assignment administered during the first year of clerkships/clinical rotations. One approach features a scaffolded, progressive model of EBM instruction over the first 2 years of medical school. This is compared with the previous model of instruction, which consisted of an isolated, 2-month EBM course taught in the first year. In order to compare these approaches, the principal investigator (PI) analysed clerkship-level EBM assignment grades to determine whether there was a difference in performance between those students who experienced the old year 1 (Y1) versus the new year 3 (Y3) instructional model with one rollout year of mixed instructional approaches in between year 2 (Y2). The clerkship EBM assignments given to the students used identical questions each year (Y1, Y2 and Y3) in order to have a valid basis for comparison. The PI and a curriculum team responsible for EBM content also analysed student performance on the EBM section of USMLE step 1 to determine potential differences. The USMLE step 1 examination is typically taken by medical students at IUSM before they begin their required clerkship courses (also known as clinical rotations). Taken together, the results of this research should be used by academic health sciences faculty and administrators to plan and implement EBM curriculum in the future.
This study took place at IUSM. IUSM is currently the largest medical school in the USA with nine campuses across the state of Indiana. Prior to 2016, EBM was taught within the curriculum in 12 discrete 1-hour sessions that were spread over a 2-month period during the first year of medical school. Once again, IUSM is unique, in that it has nine campuses, and during the previous curriculum, the method by which EBM was presented was determined by each campus. At some of the campuses, the material was presented in didactic sessions, while other campuses had students complete online modules. Critical appraisal of medical literature and application to patient care were briefly discussed, but there was no ongoing assessment of the students’ ability to apply the EBM processes that they had learnt.
It was clear from examining student evaluations of EBM education submitted during the previous curriculum that isolating EBM into a separate course or classes led to a disconnect in identifying the role of EBM in patient care. The isolated nature of the previous curriculum’s approach to teaching EBM led to marginalisation of the content as well as a student learning approach of ‘dump and purge’, whereby the material was only seen by the students as important for step 1 and was not viewed as vital to their future careers as practicing physicians. It was agreed that when it came to EBM skills, students needed more practice with critical thinking and application to patient care.
With these issues in mind, a new curriculum was implemented at IUSM in 2016 for all newly matriculating students. The curriculum was centralised so that equivalent EBM curriculum was taught at all nine campuses. Initially, most of the EBM content was introduced to the new students through 14 hours of instruction provided during the first 2 weeks of medical school with EBM assignments taking place during the second year. The content was created by content experts and included biostatistics, literature searching and critical analysis.
As the new curriculum was being rolled out, IUSM also developed an EBM Thread programme. The programme was designed to identify EBM topics that needed to be vertically and horizontally integrated across the 4 years of students’ undergraduate medical education. A group of content experts, headed in part by a medical librarian, developed a set of EBM-related learning objectives for students to achieve as a result of this new curriculum. The EBM learning objectives were developed and subsequently mapped to institutional learning objectives, Liaison Committee on Medical Education accreditation standards,3 the Accreditation Council for Graduate Medical Education’s entrustable professional activities4 and EBM standards and best practices. The members of the thread then explored the curriculum to identify gaps and appropriate courses for EBM integration as well as to develop curricular interventions and assessments to help address those gaps. At this time, it was decided that students should be exposed to critical appraisal of a prognosis and systematic review/meta-analysis article, and three additional contact hours covering these topics were added to the curriculum.
After this process was complete, the EBM Thread developed an educational framework to determine where and how EBM content would be delivered. The overall goal was to develop a longitudinal, stratified and integrated EBM curriculum. EBM activities were developed to be appropriate for the students’ learning and experience level and were integrated into the educational goals of each course and clerkship. Formative assessments were added to each session, and the results of these assessments were used to inform concepts to be reviewed in upcoming sessions. As the students advanced through their education, more emphasis was progressively placed on practical applications of EBM principles to actual patients. See table 1 and online supplemental appendix A for more information on curriculum content, delivery and timing.
Supplemental material
The IUSM EBM framework has been progressively implemented since 2017. At this point, intentional EBM learning activities have been added to most classes in phase 1 (preclinical) and phase 2 (clerkships) at IUSM. Below, we review the extant national literature for understanding curricular change in EBM.
Literature review
Scholarship focused on teaching and assessment of EBM skill sets in medical students’ preclinical years is somewhat established; however, consensus on the most efficacious approach has yet to be achieved.1 2 Most EBM instruction in medical schools is provided as a stand-alone course or as modules usually early on in the curriculum rather than developed as a longitudinal, scaffolded, integrated curriculum.5 Some studies focus on medical students’ self-reported attitudes and perceived competencies, which is helpful for developing curricula but may not measure actual skills.6 Medical students’ knowledge of and ability to apply EBM concepts in their clerkship years indicates that current curricular approaches to teaching EBM during students’ preclerkship years leave students struggling with literature searching, critical appraisal and appropriate application of evidence to patient care.7
While there is not consensus regarding the most effective instructional model for teaching EBM, individual studies suggest that longitudinal EBM instruction throughout the course of undergraduate medical education improves students’ knowledge and skills.5 8 9 One study shows that integration of EBM concepts throughout the clinical or clerkship years has also shown to have an impact on knowledge and practices of learners.10 Another study discusses the improvement of student knowledge after the rollout of a longitudinal EBM curriculum evaluated using informal reports and preliminary USMLE step 1 scores.5 Additionally, evidence shows that even first-year medical students can benefit from EBM integration into the curriculum in the form of a short course or stand-alone assignment, at least in the short term.11–13
Most current assessments of an EBM-based interventions are limited to examining a stand-alone course or project and tend to use either a validated instrument such as the Fresno test14 or Berlin questionnaire,15 although some develop their own assessment tool.1 2 9 Preliminary assessments of these types of interventions show improvement from a pretest to post-test as well as learners’ perceptions of their skills. However, additional evidence shows that educational interventions employed during undergraduate medical education intended to teach EBM skills do not result in substantial retention or application of vital EBM concepts.16 In light of these findings, better evidence is needed to measure the impact of different curricular approaches to teaching EBM skillsets to medical students.
Methods
In order to assess and compare student learning under the previous, rollout and updated curricula (see table 1 for details), the PI and a team of five library faculties used a modified version of the previously validated Fresno rubric (see online supplemental appendix B) to grade 3 years’ worth of EBM assignments given to students in clerkship rotations 1–3 (n=481) during the Internal Medicine clerkship. Students generally complete the required Internal Medicine clerkship during their third year of medical school. The Fresno rubric was originally developed and validated by Ramos, Schafer and Tracz in 200314 and is widely accepted as an appropriate instrument for examining student learning in EBM. The Fresno rubric was chosen for this project over other assessment instruments due to the fact that it was developed to assess open-ended answers to clinical questions, which most closely matches the format of the assignment that students in our programme are required to complete.
Supplemental material
The Internal Medicine clerkship at IUSM implemented the focused EBM assignment that we graded in order to provide feedback to students on their EBM skills. The assignment was designed to assess students’ ability to complete tasks showcasing their skills in identifying gaps in their knowledge, asking clinical questions, acquiring information to answer those questions, assessing available evidence and applying evidence to patient care. The assignment questions are included in online supplemental appendix B. While the current assignment questions were finalised during 2017, some form of the EBM assignment has been used with third-year medical students at IUSM since 2002. Since its inception, the librarians at the Ruth Lilly Medical Library have provided feedback on the assignment directly to students.
The decision to analyse the first three rotations from each year was made to avoid potential confounding caused by student burnout, as the students complete the same EBM assignment in each of their clerkships. The questions on the assignments completed by students were the same year over year. The goal of the assignment is to model real-life situations where clinicians seek to answer questions they have after encountering different patients with the goal of decreasing the time it takes for students to answer clinical questions through practice. However, students have voiced feelings that the assignment is redundant. It is possible that the quality of assignments diminishes over the course of the third year, but this study did not address that question. The questions on the assignments completed by students were the same each year.
The PI exported all assignment data from the learning management system and coded each assignment with a year 1–3. Y1 corresponded with assignments completed by students who received the previous curriculum, Y2 with students who received the rollout curriculum and Y3 with students who received the updated curriculum. Although third-year students completed the assignments in three different years, assignment questions did not change in wording or context. Assignments were anonymised before being randomly assigned to graders, and all possible students and date-identifying information were removed. Prior to grading, all graders were required to attend two norming sessions in order to achieve consensus on interpreting and applying the rubric consistently to sample assignments.
This project used a modified Fresno rubric to grade assignments, adapted from the original to reflect the six questions included in the clerkship-level EBM assignment and the variation in patient cases presented by students. Please see online supplemental appendix B for the modified rubric and sample responses. Fresno is a validated instrument developed for use in medical education to assess a range of EBM skills.14
Study design
Our study was intentionally designed to examine potential differences in student learning across 3 years of EBM curriculum implementation at IUSM. Curricular year represents the independent variable: previous=Y1; rollout=Y2; updated=Y3. The educational data set consisted of the graders’ scores of the Internal Medicine Clerkship students’ EBM assignments (n=481) on the modified Fresno rubric. Once again, the graders scored students’ EBM learning on six questions. A total rubric score was also produced. Taken together, these seven scores represent the dependent variables (DVs). The rubric scores (DVs) ranged from 0 to 3.
A one-way Welch ANOVA with a Games-Howell post hoc test was conducted on the DVs to determine whether there was a statistically significant difference in EBM student learning, as measured by the rubric scoring for each domain, across curricular years. A Welch ANOVA with a Games-Howell post hoc test is appropriate for examining mean differences across groups when the homogeneity of variance assumption is not met. Put simply, this version of the ANOVA helps to further ensure that there are not type I errors.
We sought to examine each EBM learning domain independently, and therefore did not use a multivariate analysis of variance (MANOVA) procedure. We determined that examining student learning gains in specific EBM domains would provide faculty and other stakeholders with the most direct information for improving teaching and learning. Notwithstanding, future studies will benefit from examining the unique interactions that may occur within and across EBM domains.
Our study design met several important statistical assumptions that are important for conducting a one-way ANOVA. For example, the independence of observations assumption was met because each student EBM score is only considered within one category of the IV (ie, curriculum implementation year). Additionally, due in part to the nature of the scored rubric data, there were no statistical outliers in the data set. The data were normally distributed for each group of the instrumental variables. However, the homogeneity of variance assumption was violated across the seven DVs as determined by the Levene’s test (p<0.05). Once again, for this reason, the Welch ANOVA test was used.
Results
All 481 graded assignments were included in the analysis. Students were classified into three groups of the IUSM EBM curriculum implementation: previous=Y1, rollout=Y2 and updated=Y3. Overall, the most notable improvements were seen from Y1 to Y3. However, improvement continued from Y2 to Y3, although less drastically. The highest mean total score for this assignment, 13.41 out of a possible 18 points, was seen in Y3, an improvement of almost one full point scores for each individual question as well as total score is shown in table 2.
Students’ evidenced-based medicine (EBM) scores as assessed by modified Fresno rubric across curriculum implementation years
Results from a one-way Welch ANOVA tests show statistically significant differences in student learning gains across the curriculum implementation years on question 3, question 6 as well as the total rubric score. The following section details these specific findings.
Question 3
Question 3 on the modified Fresno rubric assessed students’ ability to formulate a clinical question using the PICO format. Gains on this question were modest but notable, especially given that students began with a proficient score in Y1. Student scores on question 3 were statistically significantly different between curriculum implementation years, Welch’s F(2, 280.591)=3.991, p<0.05. Student ability to formulate a clinical question increased from Y1 of the EBM curriculum implementation (mean (M)=2.20, SD=0.92) to Y2 (M=2.48, SD=0.71) and Y3 (M=2.44, SD=0.71). The Games-Howell post hoc analysis revealed that the mean increase from Y1 to Y2 (0.28, 95% CI (0.04 to 0.50)) was statistically significant (p<0.05) and the increase from Y1 to Y3 (0.24, 95% CI (0.00 to 0.47)) was statistically significant (p<0.05). These results are displayed in figure 1. No statistically significant differences were found in student scores between Y2 and Y3.
: Mean scores for Question 3 year over year.
Question 6
Question 6 on the modified Fresno rubric assessed students’ ability to apply information to patient care. Student scores on question 6 were also statistically significantly different between curricular years, Welch’s F(2, 293.241)=16.837, p<0.001. Student ability to apply information to patient care increased from Y1 of the EBM curriculum implementation (M=0.89, SD=0.92) to both Y2 (M=1.45, SD=0.95) and Y3 (M=1.45, SD=0.82). Similar to question 3, the Games-Howell post hoc analysis revealed that the mean increase from Y1 to Y2 (0.56, 95% CI 0.31 to 0.80) was statistically significant (p<0.001) and the increase from Y1 to Y3 was similar (0.56, 95% CI 0.30 to 0.81) was statistically significant (p<0.001). Although the Y2 and Y3 scores do not demonstrate mastery of this skill, mastery is generally not expected for students in the first year of clerkships.
These results are displayed in figure 2. Similar to question 3, no statistically significant differences were found in student scores between Y2 and Y3.
: Mean scores for Question 6 year over year.
Total rubric score
The total score on the modified Fresno rubric provided an assessment of students’ overall EBM competency. These aggregate student scores were statistically significantly different between curricular years, Welch’s F(2, 290.016)=7.919, p<0.001. Student EBM competency increased from Y1 of the curriculum implementation (M=12.42, SD=2.26) to Y2 (M=13.19, SD=2.18) and to Y3 (M=13.41, SD=1.91). Similar to both question 3 and question 6, the Games-Howell post hoc analysis revealed that the mean increase from Y1 to Y2 (0.77, 95% CI 0.15 to 1.38) was statistically significant (p<0.001) and the increase from Y1 to Y3 (0.99, 95% CI 0.40 to 1.57) was statistically significant (p<0.001). These results are displayed in figure 3. Once again, similar to the previous questions, no statistically significant differences were found in student scores between Y2 and Y3.
: Mean overall quiz scores year over year.
Additional results and validity
It is interesting that students’ EBM scores improved overall, and within two specific domains, as new EBM curriculum was implemented at IUSM over the course of 3 years. While there was some variability in performance on individual questions, the ability of the students to complete the process of assessing their patient, asking a clinical question, acquiring the best evidence, appraising the evidence and applying the evidence to patient care improved as a result of the curricular update. Results from our study demonstrate that many of the differences in students’ EBM learning gains are statistically significant. Additionally, the ANOVA results indicate that the statistical model accounts for a notable amount of variance as measured by estimates of effect size: question 3 (η2=0.021); question 6 (η2=0.069) and total score (η2=0.034). It is also noteworthy that the findings from our study are congruent with institutional data regarding student performance on EBM on the USMLE step 1 examination. The box whisker plots provided to IUSM by the National Board of Medical Examiners (NBME) illustrate student growth in EBM over the same curricular years examined in this study.
Discussion
These results indicate that the implementation of a scaffolded, longitudinal EBM teaching intervention improves the students’ ability to perform tasks related to the EBM skillsets taught in the preclinical years of medical school, particularly those related to formulating a clinical question and appraising medical literature. As a result of the novel curricular planning and implementation process, we have shown that students are not only learning but also retaining EBM knowledge and skills in key areas related to patient care. This study has a large sample size (481 assignments) and looks at data from students in three different years of the curriculum at one institution. The results of this study have implications for other schools that may want to improve student competency in EBM skills by developing course-integrated sessions that are scaffolded across all years of the undergraduate medical school curriculum.
This study does have a few limitations. The Fresno test, while validated in its original form, had to be slightly modified to fit the assignment being used at IUSM (see online supplemental appendix A) and therefore is not technically a validated instrument itself. Additionally, experts may disagree with the questions asked by the assignment and the rubric for the responses. The Fresno test, when compared with other validated means of EBM assessment, tends to result in lower scores on average.17
As with all educational assessments, there could be confounding factors that contribute to student competency. As discussed, the switch from an isolated 2-month EBM curriculum to a scaffolded, longitudinal model did entail the addition of two critical appraisal sessions, totalling 3 contact hours. Therefore, it cannot be said with total certainty that the improvements in EBM scores on the assignment and on step 1 were due solely to the curricular delivery modality. Due to time constraints, each assignment was scored by one faculty member rather than two or more faculty members. If this team is able to conduct further research on this topic, we intend to have multiple faculty grade each assignment and report on inter-rater reliability measures. Additionally, the assignments analysed were from the first three rotations for each of the academic years rather than all rotations.
Future plans include conducting additional analysis of the data gathered from this study. We plan to analyse data for all rotations rather than limiting to the first three, developing faculty development sessions for medical educators at IUSM and continuing to improve the quality of our EBM curriculum. Preliminary review of the text data for question 6 showed that students were not always performing high-quality critical appraisals. Further textual analysis of question 6 would provide IUSM with a more defined picture of the more commonly used critical appraisal approaches and their strengths and weaknesses. As mentioned in the literature review, other studies have shown that EBM skills are not retained by medical students16 and as of this time, we still lack data on whether or not students retain these concepts into residency and beyond. By examining data from all rotations, we may be able to determine whether EBM skills improve or deteriorate between the first rotation and the last rotation. This information may be helpful to educators in the clinical setting as they model EBM skills for clerkship students. This could also inform future efforts to provide faculty development and train-the-trainer sessions for our medical educators. Finally, IUSM is invested in continuous quality improvement. We will continue to build on the work that has been done and monitor our students’ progress to ensure that students are improving in their EBM and lifelong learning skills.
Data availability statement
Data are available in a public, open access repository. The data that support the findings of this study are openly available in figshare. The doi is below. 10.6084/m9.figshare.12003087.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to acknowledge the efforts of the EBM Thread team at the IU School of Medicine as well as the Curriculum Committee at the Ruth Lilly Medical Library, without whom this work would not have been possible. We also wish to sincerely thank the Medical Library Association for its commitment to supporting early-career researchers through the Research Training Institute.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @LMLibrarian
Correction notice This article has been corrected since it appeared Online First. "(IV)" has been removed from the title.
Contributors LM and AEB conceived of the presented idea. LM designed the study, submitted IRB approval, and obtained/anonymised the data. AEB and LM worked together to develop the rubric for grading. LM and DJT performed the analysis of the data and wrote the Study Design and Results section. AEB and KHL worked together to write the Background section. All authors discussed the results and contributed to the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.