Article Text
PDF
Extract
The objective of this study is to compare accreditation standards across medical and surgical specialties in the USA to determine the requirements for evidence-based medicine (EBM) and information management (life-long learning) skills education in residency training programmes in the USA. We performed a descriptive analysis of ‘milestones’ statements that form the basis for oversight of residency training of the Next Accreditation System for the 28 major specialties and the transition year residency, searching the milestone statements for competency requirements for 15 EBM or information management topics. The search was performed independently by pairs of researchers, with a third researcher independently verifying their findings. No specialty requires all 15 topics. Basic EBM knowledge is required by fewer than half of all specialties. Only 14% of milestone statements specify the ability to identify bias in clinical research trials. Most specialties (79%) require demonstration of the use of evidence in practice. MEDLINE searching capability is required by 52% of specialties; competence in using secondary sources is required by 69%. There is little emphasis (22%) on evaluation of these information sources. Evaluation of information from experts, colleagues or pharmaceutical representatives is required by only 38%. Competence in developing methods of keeping up with new evidence is specified by only four specialties (14%). The results signal a shift in teaching requirements during residency training, away from classic evidence-based and towards information management skills. However, requirements are limited to published materials and rarely extend to information from human sources.
- MEDICAL EDUCATION & TRAINING
Statistics from Altmetric.com
Introduction
In the USA, there has been a move to competency-based approach to medical education. This initiative began in 1999 when the Accreditation Council for Graduate Medical Education (ACGME) introduced the Outcome Project, which outlined six domains of clinical competency.1
Though ‘evidence-based medicine’ (EBM) knowledge and skills are not specifically mentioned in the new requirements governing residency education, the ability to ‘appraise and use scientific evidence’ is included as part of the ‘Practice-Based Learning and Improvement’ (PBLI) competency domain.
‘Information mastery’2 focuses on identifying information needs and using evidence sources at the point of care, as well the use of current awareness services to keep aware of new, at the point of care and using evidence new, relevant and valid information likely to change practice. It too is included in the PBLI competency domain, broadly characterised as, ‘use technology to optimise learning.’1
The ACGME introduced the ‘Next Accreditation System’ 10 years later in 2009. This new system of residency accountability requires the regular tracking and reporting of individual resident competency. Added to the previously defined outcomes are ‘milestone’ definitions, which are, ‘developmentally based, specialty-specific achievements that residents are expected to demonstrate at established intervals as they progress through training’.3
Each medical specialty has developed specific categories and definitions of milestones to gauge resident development of competence. The milestone descriptions are broadly outlined in the original Outcome Project but each specialty was charged with developing milestone descriptors to their unique clinical environment and scope of practice.
As a result in this requirement to specify the competence of graduates from each type of specialty residency, there is the ability to compare requirements for the teaching of EBM and information mastery skills and knowledge across medical specialties to gain an understanding of the state of the art. The goal of this project was to describe similarities and differences in these teaching requirements across major specialties in the USA.
Design
The project was a descriptive analysis to compare the knowledge, skill and attitude requirements related to EBM and information management for residency training across all major specialties. The sources for our analysis were the milestone documents published on the ACGME website for the 28 major specialties and the transition year residency.
As previously described,4 we developed a taxonomy, derived from all milestone documents, describing 15 topics arranged into five major themes. Two themes relate to classic EBM: critical appraisal skills and knowledge of clinical epidemiology principles.5 Three themes relate to the life-long learning skills of acquiring and using new information to improve decision-making and the subsequent care of patients. The first of these information management themes, identifying need for information, requires the ability to identify one's specific question (in contrast to a quest for general information) and to develop a patient-intervention-comparison-outcome (PICO) question suitable for literature searching. Answering patient-specific questions, the second theme, involves using point-of-care information resources. Keeping up with changes in medicine includes the demonstration of the use of a current awareness system.
We used this taxonomy to compare competency requirements by specialty. Specialties addressing some but not necessarily all of the content described in each topic code (see online supplementary digital content table S1; see end of document) were considered to require that content. Descriptive statistics were used to describe the presence of teaching requirements across the specialties. Further analysis was performed by grouping specialties as hospital-based specialties, medical specialties and surgical specialties, using the ACGME categorisation.
We conducted the project using directed content analysis.6 ,7 The investigators included an expert in EBM and three practising physicians who are educators in residency training programmes. All investigators have additional training in medical education and EBM.
Open coding was performed by a single researcher (AFS) for each competency descriptor to develop preliminary start codes. This stage involved assigning code words based on standard EBM and information mastery concepts (eg, ‘study design’, ‘evaluation of practice guidelines’). Using a constant comparative approach, all four researchers reviewed all descriptors separately, coding each descriptor with one or more codes, adding new codes as warranted.
The codes were reworked in an iterative process until all descriptors were represented by one or more codes and no new codes were identified (ie, saturation was reached). At this point, consolidation was performed to minimise overlap. We coalesced these codes into three distinct themes that represented the individual codes in a logical manner. The codes were continually reworked until we achieved the best fit under each theme. These codes were then applied to each specialty milestone statement by pairs of researchers. This step was performed by AFS and KAF and then sequentially refined by the group. A third researcher independently coded each milestone statement. Discrepancies between researchers were resolved through discussion.
Results
Across the 29 milestone documents, we identified 273 separate descriptors relating to the use of evidence or processing of medical information. All specialty milestone documents have at least one milestone statement for PBLI; almost all of these specify at least one EBM or information management requirement. Colorectal surgery does not have any milestone descriptors related to EBM or information management, and diagnostic radiology only has one. The topics addressed by each specialty are outlined in online supplementary digital content table S2.
Most milestone descriptors relating to the use of evidence or information are described in the PBLI milestone statements. Ten specialties have descriptors in other milestone statements; these descriptors usually relate to the application of evidence or information. For example, a patient care milestone for ophthalmology states that a resident, ‘employs routine and advanced diagnostic tests and imaging procedures according to evidence-based medicine’ (level 4, PC-3 Diagnostic Procedures).
Requirements for basic EBM knowledge vary among specialties (table 1). Less than half of all specialties require knowledge of basic indicators of study quality and relevance (table 1: EBM 1). Only 14% of specialties require competence in identifying bias in clinical research studies. The ability to critically appraise original research is required by only 52% of specialties, though it is more highly represented in medical and surgical specialties. Less than half of all specialties require demonstration of the critical appraisal of reviews (38%) and practice guidelines (41%). These skills are particularly under-represented in hospital-based specialties (18% and 27%, respectively).
EBM themes in specialty milestones
Most specialties (79%) recognise the need to demonstrate the use of evidence when making changes in practice. However, developing skills in the acquisition and use of information in clinical practice is variably represented across specialties. Half (52%) the specialties require demonstration of competence in classic literature searching (eg, MEDLINE searches). Sixty-nine per cent of all specialties, including all but one medical specialty, specify competence in the use of secondary resources used at the point of care to answer patient-specific questions. However, only 21% of specialties (medical genetics, physical medicine and rehabilitation, all three preventive medicine specialties, and psychiatry) also require demonstration of the ability to evaluate the quality of information provided by point-of-care resources (data not shown in table 1).
Most specialties do not specify competence in the evaluation of secondary resources other than databases; only 38% require competence in the evaluation of non-database resources such as experts, colleagues or pharmaceutical representatives.
Current awareness services abound in medicine, providing physicians with alerts or summaries of recently published scientific literature. The ability of residents to develop and use a system for keeping up with changes in medicine, as well as to evaluate the quality, objectivity and biases of available systems, was addressed only by family medicine, dermatology, obstetrics/gynaecology and psychiatry.
Discussion
Training in residency programmes in the USA lays the groundwork for what will be the state of clinical practice among its graduates. In this study, we compared the teaching requirements across specialty training programmes in the USA, finding wide variation. We found what seems to be a de-emphasis on requiring competence in basic EBM skills and knowledge and more of a focus on information mastery skills. However, requirements are limited to published materials and rarely extend to information from human sources of information.
The information preferred by many physicians comes from guidelines, expert consultants, colleagues and the pharmaceutical industry.8 ,9 Relatively few specialties require competence in the evaluation and use of information obtained from these sources, although evaluation methods are available.10–12
Staying abreast of the rapid changes in medicine is an important skill for life-long learning. New research findings in medicine can take over a decade to be reflected in clinical practice.13 ,14 Yet, fewer than one in five specialties (14%) require residents to demonstrate they have a mechanism to keep up with new information that should change practice. As a result, 17% of healthcare spending in the USA is associated with physician beliefs unsupported by clinical evidence.15
The limited requirements in the milestone statements for teaching these competencies may reflect, on the part of the respective committee members, either a lack of perceived need for this teaching or a perceived lack of capability to teach the content within their respective residencies.16 The committees developing the milestones for each specialty were comprised of members drawn from residency review committees, residency programme directors and residents within that specialty. These representatives likely tried to find a balance between what should be required of residency graduates and the current or near-future capability of residencies to teach and document competency in EBM and information management.
One specialty (neurology) specifies competence in ‘understanding the limitations of using evidence to guide decisions’. This statement may reflect either scepticism towards EBM or an acknowledgement of the role of shared decision-making.
However, other specialties, especially the primary care specialties, have a large need to manage their ever-evolving knowledge base.
This study is limited by the nature of the milestone descriptions. The brevity of the milestones may not reflect the complete intent of the committee members. To this end, several milestone development groups have written companion documents containing interpretations of the milestones and suggestions for implementation. We did not evaluate these ancillary documents.
In addition, the milestones represent a significant event or point in development but are not written as behavioural objectives. As a result, individual residency programmes are given considerable latitude to interpret the milestone descriptions and to develop means of assessment for determining resident progress through training. As the Next Accreditation System is implemented and residencies develop means of documenting these milestones, their use in practice may drift from their current literal meaning.
Conclusions
Wide variation exists in the requirement of teaching EBM knowledge and skills and information mastery techniques among specialty residencies in the USA. In many specialties, there is more of an emphasis on teaching information retrieval skills; required critical appraisal skills are usually on primary research and less emphasised for some of the most frequently used secondary sources such as reviews, practice guidelines, and information from consultants and colleagues. Few specialties require competency development in methods to keep up with changes in medicine, a skill crucial in this era of medicine.
References
Footnotes
Competing interests AFS was a member of the Accreditation Council on Graduate Medical Education's Expert Panel that developed example milestones and was a member of the Family Medicine Milestone Development Group.
Provenance and peer review Not commissioned; externally peer reviewed.