Context: Computer-based diagnostic decision support systems (DSSs) were developed to improve health care quality by providing accurate, useful, and timely diagnostic information to clinicians. However, most studies have emphasized the accuracy of the computer system alone, without placing clinicians in the role of direct users.
Objective: To explore the extent to which consultations with DSSs improve clinicians' diagnostic hypotheses in a set of diagnostically challenging cases.
Design: Partially randomized controlled trial conducted in a laboratory setting, using a prospective balanced experimental design in 1995-1998.
Setting: Three academic medical centers, none of which were involved in the development of the DSSs.
Participants: A total of 216 physicians: 72 at each site, including 24 internal medicine faculty members, 24 senior residents, and 24 fourth-year medical students. One physician's data were lost to analysis.
Intervention: Two DSSs, ILIAD (version 4.2) and Quick Medical Reference (QMR; version 3.7.1), were used by participants for diagnostic evaluation of a total of 36 cases based on actual patients. After training, each subject evaluated 9 of the 36 cases, first without and then using a DSS, and suggested an ordered list of diagnostic hypotheses after each evaluation.
Main outcome measure: Diagnostic accuracy, measured as the presence of the correct diagnosis on the hypothesis list and also using a derived diagnostic quality score, before and after consultation with the DSSs.
Results: Correct diagnoses appeared in subjects' hypothesis lists for 39.5% of cases prior to consultation and 45.4% of cases after consultation. Subjects' mean diagnostic quality scores increased from 5.7 (95% confidence interval [CI], 5.5-5.9) to 6.1 (95% CI, 5.9-6.3) (effect size: Cohen d = 0.32; 95% CI, 0.23-0.41; P<.001). Larger increases (P = .048) were observed for students than for residents and faculty. Effect size varied significantly (P<.02) by DSS (Cohen d = 0.20; 95% CI, 0.08-0.32 for ILIAD vs Cohen d = 0.45; 95% CI, 0.31-0.59 for QMR).
Conclusions: Our study supports the idea that "hands-on" use of diagnostic DSSs can influence diagnostic reasoning of clinicians. The larger effect for students suggests a possible educational role for these systems.