You are here

Article Text

Article menu

Download PDFPDF

Original EBM Research
Longitudinal adoption rates of complex decision support tools in primary care
Free
  1. Lauren McCullagh1,
  2. Devin Mann2,
  3. Lisa Rosen1,
  4. Joseph Kannry3,
  5. Thomas McGinn1
  1. 1Department of Medicine, Hofstra North Shore-LIJ School of Medicine, Manhasset, New York, USA
  2. 2Department of Medicine, Section of Preventive Medicine and Epidemiology, Boston University School of Medicine, Boston, Massachusetts, USA
  3. 3Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, New York, USA
  1. Correspondence to Dr Lauren McCullagh
    Department of Medicine, Hofstra North Shore-LIJ School of Medicine, 300 Community Drive, Manhasset, NY 11201, USA; lmccullagh{at}nshs.edu

Extract

Translating research findings into practice promises to standardise care. Translation includes the integration of evidence-based guidelines at the point of care, discerning the best methods to disseminate research findings and models to sustain the implementation of best practices. By applying usability testing to clinical decision support (CDS) design, overall adoption rates of 60% can be realised. What has not been examined is how long adoption rates are sustained and the characteristics associated with long-term use. We conducted secondary analysis to decipher the factors impacting sustained use of CDS tools. This study was a secondary data analysis from a clinical trial conducted at an academic institution in New York City. Study data was deidentified patients’ electronic health records (EHR). The trial was to test the implementation of an integrated clinical prediction rule (iCPR) into the EHR. The primary outcome variable was iCPR tool acceptance of the tool. iCPR tool completion and iCPR smartest completion were additional outcome variables of interest. The secondary aim was to examine user characteristics associated with iCPR tool use in later time periods. Characteristics of interest included age, resident year, use of electronic health records (yes/no) and use of best practice alerts (BPA) (yes/no). Generalised linear mixed models (GLiMM) were used to compare iCPR use over time for each outcome of interest: namely, iCPR acceptance, iCPR completion and iCPR smartset completion. GLiMM was also used to examine resident characteristics associated with iCPR tool use in later time periods; specifically, intermediate and long-term (ie, 90+ days). The tool was accepted, on average, 82.18% in the first 90 days (short-term period). The use decreases to 56.07% and 45.61% in intermediate and long-term time periods, respectively. There was a significant association between iCPR tool completion and time periods (p<0.0001). There was no significant difference in iCPR tool completion between resident encounters in the intermediate and long-term periods (p<0.6627). There was a significant association between iCPR smartset completion and time periods (p<0.0021). There were no significant associations between iCPR smartest completion and any of the four predictors of interest. We examined the frequencies of components of the iCPR tool being accepted over time by individual clinicians. Rates of adoption of the different components of the tool decreased substantially over time. The data suggest that over time and prolonged exposure to CDS tools, providers are less likely to utilise the tool. It is not clear if it is fatigue with the CDS tool, acquired knowledge of the clinical prediction rule, or gained clinical experience and gestalt that are influencing adoption rates. Further analysis of individual adoption rates over time and the impact it has on clinical outcomes should be conducted.

  • PRIMARY CARE
  • INTERNAL MEDICINE

https://doi.org/10.1136/ebmed-2014-110054

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background

    Centres such as the Knowledge Translation Canada Clearing House, the Veteran Affair's Center for Implementation Practice and Research Support, and the National Cancer Institute's Implementation Science Center strive to expand implementation science and translate research findings into practice. Implementation science focuses the methodology used to translate research findings into practice. This includes the integration of evidence-based guidelines and tools at the point of care, discerning the best methods to disseminate research findings and models to sustain the implementation of best practices.

    One of the emerging areas of implementation science is health information technology (HIT). The US federal government, policy groups and academics alike are hopeful that HIT tools will be the gateway to improving patient outcomes and quality of care.1–5 Health technology, such as electronic health records (EHRs) and clinical decision support (CDS) tools are rapidly being implemented in hospitals and clinics nationwide.6 ,7 ‘Among non-federal acute care hospitals, adoption of at least a basic EHR system has increased by over 260% and physician adoption of five core Meaningful Use functionalities—ranging from e-prescribing to CDS—has grown by at least 66%.’7 ,8

    CDS systems have shown to have some success in improving patient outcomes but implementation of EHR and CDS tools into complex healthcare systems and clinical workflow continues be challenging and poor integration can result in poor adoption.9–12 A recent systematic review of 148 randomised controlled studies of electronic CDS systems revealed that two-thirds documented low use (<50%).13 In our previous publications we document that by applying usability testing to CDS design, adoption rates of 60% can be realised.14 ,15 The rapid-cycle test-of-change improvement methodology and usability testing of iterative designs have proven to customise tools to different clinical settings and change efforts in healthcare organisations.16–19 Although usability testing has been able to improve aggregate rates of adoption, there are few studies examining the sustainability of CDS adoption rates and impact on clinical outcomes.

    Time series analysis of interventions has been conducted in behavioural intervention studies to determine their sustainability, yet few studies look at the sustained use of CDS tools.20 New conceptual models are being developed to address sustained use of CDS tools and the behavioural adaptation of users over time, but the models have not been analysed in prospective studies.21 Analysis of compliance with CDS adoption over a time period showed moderate adoption rates but poor compliance.22

    This paper's aim was to measure adoption and ongoing use in a cohort. Secondary aims were to decipher the factors impacting sustained use of CDS tools. Our study examined adoption rates of CDS tools over time and provider characteristics to predict long-term adoption. We hypothesised that as time progressed, fewer participants would use the tool and there will be a relationship between age and long-term use.

    Methods

    From November 2010 to November 2011, the iCPR randomised controlled trial was conducted within two large urban ambulatory primary care practices in New York City with a mix of providers: attendings, residents and nurse practitioners. The patient population in both practices was racially and ethnically diverse, with nearly 56% of patients identifying as Hispanic, 35% African-American, 7% White and 2% other races. Residents, attendings, fellows and nurse practitioners working in the outpatient primary care clinic at the medical centre were eligible to participate in the study.

    This was a secondary analysis of an existing data set from a randomised controlled trial (RCT) that examined the impact of an integrated clinical prediction rule (iCPR) on strep pharyngitis and pneumonia.14 ,15 ,23 The RCT study used novel usability techniques that draw on low-cost technology and real ambulatory clinical settings and created a tool that integrated CPRs into a widely used commercial EHR platform, European Prospective Investigation into Cancer and Nutrition, as a CDS tool. Clinicians were randomised using a random-number generator into two groups: namely, intervention and control. The intervention group had training and access to the iCPR tool. The control group was provided with articles on clinical prediction rules. For purposes of this study, only residents (postgraduate year 1 (PGY1), PGY2 and PGY3) in the intervention arm were included and the attending and nurse practitioners were excluded since there was not enough substantial data in these groups. The primary aim was to evaluate use of the iCPR tool by residents in the intervention14 arm over time. The primary outcome variable was iCPR tool use, which was defined as accepting or opening of the tool (yes/no). iCPR tool completion and iCPR smartest (computerised bundled set orders) completion were additional outcome variables of interest. The secondary aim was to examine characteristics associated with iCPR tool use in later time periods (ie, long-term use).

    Generalised linear mixed models (GLiMM) were used to compare iCPR use over time for each outcome of interest; namely, iCPR acceptance, iCPR completion and iCPR smartset completion. Time was measured in calendar days from the first patient encounter that triggered the iCPR tool and categorised into three time periods; short-term (defined as less than 90 days), intermediate-term (defined as 90–239 days) and long-term (defined as 240 days or longer). GLiMM was used to account for repeated measurements within a resident.

    GLiMM was also used to examine resident characteristics associated with iCPR tool use in later time periods; specifically, intermediate and long-term (ie, 90+ days). For this analysis, only encounters that triggered the tool in the intermediate and long-term periods were included and examined for tool use. Characteristics of interest included age, resident year, experience with electronic health records (yes/no) and use of best practice alerts (BPA, clinical decision support alert boxes) (yes/no). Each characteristic was examined in a separate model as independent predictors. Characteristics that were found to be associated with tool use at the significance level of p<0.10 were included in a multivariable model.

    All analyses were conducted using SAS V.9.3 (Cary, North Carolina, USA) and results were considered statistically significant at p<0.05, unless stated otherwise.

    Results

    There were a total of 70 residents in the study sample included in the intervention arm; 34 PGY1 (48.57%), 18 PGY2 (25.71%) and 18 PGY3 (25.71%). These 70 residents account for 440 strep pharyngitis or pneumonia patient encounters that triggered the iCPR tools (table 1). Table 2 describes the characteristics of the 440 encounters in each of the time periods.

    View this table:
    Table 1

    Resident characteristics

    View this table:
    Table 2

    Resident characteristics for all encounters

    iCPR tool use over time

    iCPR encounter was defined as any patient visit in which the tool would have been activated. There was a significant association between iCPR tool acceptance and time periods (p<0.0001) (table 3). The tool was accepted (opened tool/iCPR encounters), on average, 82.18% in the first 90 days (short-term period). The use decreases to 56.07% and 45.61% in intermediate and long-term time periods, respectively. Specifically, resident encounters in the short-term period had greater odds of iCPR tool acceptance as compared to resident encounters in the intermediate (odds ratio (OR) 3.61 to 95% CI 2.06 to 6.34) and long-term periods (OR 5.50, 95% CI 2.12 to 14.27). There was no significant difference in iCPR tool acceptance between resident encounters in the intermediate and long-term periods (p<0.3872).

    View this table:
    Table 3

    iCPR tool use over time

    There was a significant association between iCPR tool completion and time periods (p<0.0001). Specifically, resident encounters in the short-term period had greater odds of iCPR tool completion as compared to resident encounters in the intermediate (OR 3.68, 95% CI 2.14 to 6.33) and long-term periods (OR 4.52, 95% CI 1.79 to 11.41). There was no significant difference in iCPR tool completion between resident encounters in the intermediate and long-term periods (p<0.6627).

    There was a significant association between iCPR smartset completion and time periods (p<0.0021). Specifically, resident encounters in the short-term period had greater odds of iCPR smartset completion as compared to resident encounters in the intermediate time period (OR 2.54, 95% CI 1.49 to 4.31). There were no significant differences in iCPR smartest completion between resident encounters in the short-term and long-term periods (p<0.1109), nor resident encounters in the intermediate and long-term periods (p<0.6384).

    Characteristics of long-term iCPR tool use

    Resident year and use of other EHR were both significantly associated with iCPR acceptance and completion. Unadjusted multivariable models were created for iCPR acceptance and iCPR completion in the later time periods using these two characteristics as independent predictors (table 4). PGY1 were over five times more likely to accept and complete the iCPR tool. There were no significant associations between iCPR smartest completion and any of the four predictors of interest. Therefore, this outcome was not further evaluated in a multivariable model.

    View this table:
    Table 4

    Characteristics associated with iCPR tool use in later time periods (multivariable)

    Discussion

    Clinical decision support tools have proven to have a positive impact on patient outcomes but their adoption rates continue to remain low in many health systems.9–12 Health IT (HIT) conceptual design models examine the personal psychological and cognitive factors that affect the human-computer interface.14 ,21 ,24 What has been understudied is specific longitudinal patterns of adoption and how personal characteristics and personal preference have an impact on adoption rates. Understanding the personal attributes will facilitate HIT designers and researchers in building CDS tools that have sustained adoption and impact on the quality of care.

    In addition to overall adoption levels being low, our study demonstrates that CDS acceptance and use of the tools starts high but tends to diminish over time (p<0.0001). The residents in the short-term period had greater odds of iCPR tool acceptance compared to resident encounters in the intermediate (OR 3.61, 95% CI 2.06 to 6.34) and long-term periods (OR 5.50, 95% CI 2.12 to 14.27). Similar trends were seen in residents’ completion of the tool and smartsets (bundled set orders) over time. Interestingly, there was no significant difference in iCPR tool acceptance, completion or smartest orders between resident encounters in the intermediate and long-term periods. The predictors of long-term use were age, resident training level, experience with EHR systems and BPA. Resident training level was the only factor associated to increased adoption and completion of the iCPR tool.

    Inferences from these results may be that physicians may get saturated with the information from the tool after a certain point of time and no longer find the tool useful and therefore not use it. Clinical knowledge may also play an important factor as more experienced residents (PGY3) were less likely to use the tool. This trend was also seen in a study comparing adoption levels across residents and attending. Design models and tools tailored to personal characteristics have been shown to have higher rates of adoption and have been more effective in changing health outcomes. Adaptive design is based on customising tools that reflect the users’ knowledge in the decision-making process and presentation of data customised to preferences and clinical workflow and end-user development model, which creates point and click applications to give non-programmers the ability to tailor applications to their own needs.25 ,26 User centred design HIV behavioural messages tailored were shown to be effective in motivating patients to better manage their health.24

    Interestingly, there was no significant difference in iCPR tool acceptance and completion between resident encounters in the intermediate and long-term periods. This may be due to the small sample size of long-term encounters resulting in a lack of power or because residence use levels off during the intermediate and long-term periods. Therefore, future studies with a larger sample of long-term follow-up should be conducted in order to further examination of iCPR use at later time points.

    A limited factor to this study was that the original study design was a randomised controlled trial to measure antibiotic orders as a result of using the CDS tool, and not a true longitudinal design. A limitation of this study is that there were few encounters in the late follow-up period (>240 days) (n=28). However, statistically, it is not an issue that these encounters only occur among 10 residents because we modelled the data using generalised estimating equations and the unit of analysis is the encounter-level (not the provider-level). That being said, it is possible that those with longer follow-up have different resident-level characteristics (eg, they are all PGY1s and PGY2s—PGY3 is not included in the late follow-up). Therefore, the reason for iCPR tool use over time might certainly be due to the resident-level characteristics rather than just a function of time. However, given the nature of the study in which residents rotate in and out, this could not be controlled for.

    The data from this study suggests that over time and prolonged exposure to CDS tools, providers are less likely to utilise the tool. It is not clear if it is fatigue with the CDS tool, acquired knowledge of the clinical prediction rule, or gained clinical experience that are influencing adoption rates. It is also not clear how this will impact clinical outcomes over the long term. Rates of sustained use of CDS tools could be improved by various methods such as follow-up/refresher courses with time, allowing customisation to the users’ adaptive behaviours, or minimising triggering in during the summer season (low rates of pneumonia and strep) to minimise trigger fatigue. Yet this study will guide researchers and developers in understanding characteristics of sustained use and will guide the design of CDS tools. Further analysis of individual adoption rates over time and the impact that analysis has on clinical outcomes, such as on antibiotics ordered, should be conducted.⇓

    Figure 1
    Figure 1

    Acceptance, Completion of the iCPR tool and Bundled Set Order.

    References

    1. ↵
      CMS. EHR Incentive Program: Data and Reports. Secondary EHR Incentive Program: Data and Reports 2012. http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/DataAndReports.html
    2. ↵
      1. Blumenthal D
      . Launching HITECH. N Engl J Med 2010;362:382–5.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Linder JA,
      2. Chan JC,
      3. Bates DW
      . Evaluation and treatment of pharyngitis in primary care practice: the difference between guidelines is largely academic. Arch Intern Med 2006;166:1374–9.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      Centers for Medicare & Medicaid Services (CMS), HHS. Medicare and medicaid programs; electronic health record incentive program—stage 2. Proposed rule. Fed Regist 2012;77:13698–827.
      OpenUrl
    5. ↵
      Center for Medicare and Medicaid Services (CMS). Federal Register | Electronic Health Record (EHR) Incentive Program (CMS-0033-F). Secondary Federal Register | Electronic Health Record (EHR) Incentive Program (CMS-0033-F). http://www.federalregister.gov/regulations/0938-AP78/electronic-health-record-ehr-incentive-program-cms-0033-f-
    6. ↵
      1. King J,
      2. Patel V,
      3. Furukawa M
      . Physician adoption of electronic health record technology to meet meaningful use objectives: 2009–2012. ONC Data Brief, no 7. Washington, DC: Office of the National Coordinator for Health Information Technology, 2012.
    7. ↵
      ONC. ONC analysis of data from the 2011 American Hospital Association Survey Information Technolog Supplement. 2011.
    8. ↵
      1. Charles D,
      2. King J,
      3. Furukawa M,
      4. et al
      . Hospital adoption of electronic health record technology to meet meaningful use objectives: 2008–2012. ONC Data Brief, no 10: Washington, DC: Office of the National Coordinator for Health Information Technology, 2013.
    9. ↵
      1. Hemens BJ,
      2. Holbrook A,
      3. Tonkin M,
      4. et al
      . Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci 2011;6:89.
      OpenUrlCrossRefPubMed
    10. ↵
      1. Eccles M,
      2. McColl E,
      3. Steen N,
      4. et al
      . Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941.
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. Litvin CB,
      2. Ornstein SM,
      3. Wessell AM,
      4. et al
      . Use of an electronic health record clinical decision support tool to improve antibiotic prescribing for acute respiratory infections: the ABX-TRIP study. J Gen Intern Med 2013;28:810–16.
      OpenUrlCrossRefPubMed
    12. ↵
      1. Cresswell K,
      2. Majeed A,
      3. Bates DW,
      4. et al
      . Computerised decision support systems for healthcare professionals: an interpretative review. Inform Prim Care 2012;20:115–28.
      OpenUrlPubMed
    13. ↵
      Quality AfHRa. Enabling health care decisionmaking through clinical decision support and knowledge management. Rockville, MD: AHRQ, 2012.
    14. ↵
      1. McGinn TG,
      2. McCullagh L,
      3. Kannry J,
      4. et al
      . Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013;173:1584–91.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Li A,
      2. Kushniruk A,
      3. Kannry JL,
      4. et al
      . Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012;81:761–72. (in press).
      OpenUrlCrossRefPubMed
    16. ↵
      1. Yuan MJ,
      2. Finley GM,
      3. Long J,
      4. et al
      . Evaluation of user interface and workflow design of a bedside nursing clinical decision support system. Interact J Med Res 2013;2:e4.
      OpenUrlCrossRef
    17. ↵
      1. Berwick DM
      . Disseminating innovations in health care. JAMA 2003;289:1969–75.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Berwick DM
      . Developing and testing changes in delivery of care. Ann Intern Med 1998;128:651–6.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Horbar JD,
      2. Rogowski J,
      3. Plsek PE,
      4. et al
      . Collaborative quality improvement for neonatal intensive care. NIC/Q Project Investigators of the Vermont Oxford Network. Pediatrics 2001;107:14–22.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Hanbury A,
      2. Farley K,
      3. Thompson C,
      4. et al
      . Immediate versus sustained effects: interrupted time series analysis of a tailored intervention. Implement Sci 2013;8:130.
      OpenUrlCrossRefPubMed
    21. ↵
      1. Hsieh TH,
      2. Chen JJ,
      3. Chen LH,
      4. et al
      . Time-course gait analysis of hemiparkinsonian rats following 6-hydroxydopamine lesion. Behav Brain Res 2011;222:1–9.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Drescher FS,
      2. Chandrika S,
      3. Weir ID,
      4. et al
      . Effectiveness and acceptability of a computerized decision support system using modified Wells criteria for evaluation of suspected pulmonary embolism. Ann Emerg Med 2011;57:613–21.
      OpenUrlCrossRefPubMed
    23. ↵
      1. Mann DM,
      2. Kannry J,
      3. Edonyabo D,
      4. et al
      . Rationale, design and implementation protocol of an electronic health record Integrated Clinical Prediction Rule (iCPR) randomized trial in primary care. Implement Sci 2011;6:109. in press.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Uhrig JD,
      2. Lewis MA,
      3. Bann CM,
      4. et al
      . Addressing HIV knowledge, risk reduction, social support, and patient involvement using SMS: results of a proof-of-concept study. J Health Commun 2012;17(Suppl 1):128–45.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Ghédira C,
      2. Maret P,
      3. Fayn J,
      4. et al
      . Adaptive user interface customization through browsing knowledge capitalization. Int J Med Inform 2002;68:219–28.
      OpenUrlCrossRefPubMed
    26. ↵
      1. Lieberman H,
      2. Paterno F,
      3. Wulf V
      . End user development. Dordrecht, The Netherlands: Kluwer Publishers, 2006.
    View Abstract

    Footnotes

    • Contributors LM was involved in implementation of the project, research question, analysis and writing of the manuscript. DM and TMG were involved in the study design, data collection and decisions on the type of data analysis, interpretation of the results and provided significant guidance on the writing of the manuscript. LR conducted all the statistical analysis on the study at Hofstra North Shore LIJ. JK attended regular meetings on the study design, data collection, decisions on the type of data analysis, interpretation of the results and provided significant guidance on the writing of the manuscript.

    • Funding The original study was sponsored by Agency for Healthcare Research and Quality (7R18HS018491).

    • Competing interests None.

    • Ethics approval The procedures used have been reviewed in compliance with ethical standards of the responsible committee on human experimentation at the Feinstein Institute for Medical Research, Manhasset, NY. All research activities are in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The data will be shared on request.