ABSTRACT
OBJECTIVES
We conducted a review of the peer-reviewed literature since 2003 to catalogue reported methods of stakeholder engagement in comparative effectiveness research and patient-centered outcomes research.
METHODS AND RESULTS
We worked with stakeholders before, during and after the review was conducted to: define the primary and key research questions; conduct the literature search; screen titles, abstracts and articles; abstract data from the articles; and analyze the data. The literature search yielded 2,062 abstracts. The review was conducted on 70 articles that reported on stakeholder engagement in individual research projects or programs.
FINDINGS
Reports of stakeholder engagement are highly variable in content and quality. We found frequent engagement with patients, modestly frequent engagement with clinicians, and infrequent engagement with stakeholders in other key decision-making groups across the healthcare system. Stakeholder engagement was more common in earlier (prioritization) than in later (implementation and dissemination) stages of research. The roles and activities of stakeholders were highly variable across research and program reports.
RECOMMENDATIONS
To improve on the quality and content of reporting, we developed a 7-Item Stakeholder Engagement Reporting Questionnaire. We recommend three directions for future research: 1) descriptive research on stakeholder-engagement in research; 2) evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research; and 3) development and validation of tools that can be used to support stakeholder engagement in future work.
Similar content being viewed by others
REFERENCES
Institute of Medicine. Initial Priorities for Comparative Effectiveness Research. Washington: National Academies Press; 2009.
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and Congress. Washington: US Department of Health and Human Services; 2009.
Sox HC, Greenfield S. Comparative effectiveness research: a report from the institute of medicine. Ann Intern Med. 2009;151:203–205.
Patient Centered Outcomes Research Institute. http://www.pcori.org. Accessed March 21, 2014.
McClellan M, Benner J, Garber AM, Meltzer DO, Tunis SR, Pearson S. Comparative Effectiveness Research: Will it bend the Health Care Cost Curve and Improve Quality? The Brookings Institute: Washington; 2009.
Roehr B. More stakeholder engagement is needed to improve quality of research, say US experts. BMJ. 2010;341:c4193.
Conway PH, Clancy C. Charting a path from comparative effectiveness funding to improved patient-centered health care. JAMA. 2010;303:985–6.
Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, Conway PH, Daudelin D, Morrato EH, Leslie LK. A new taxonomy for stakeholder engagement in patient centered outcomes research. J Gen Intern Med. 2012;27:985–991.
Selker HP, Leslie LK, Wasser JS, Plaut AG, Wilson IB, Griffith JL, Tufts CTSI. Tufts CTSI. Clin Transl Sci. 2010;3:56–8.
Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;424:1758–72.
Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA group preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.
Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). http://www.prisma-statement.org. Accessed March 21, 2014.
Amico KL, Wieland ML, Weis JA, Sullivan SM, Nigon JA, Sia IG. Capacity building through focus group training in community-based participatory research. Educ Health (Abingdon). 2011;24:638.
Ammerman A, Corbie-Smith G, St George DM, Washington C, Weathers B, Jackson-Christian B. Research expectations among African American church leaders in the PRAISE! project: a randomized trial guided by community-based participatory research. Am J Publ Health. 2003;93:1720–1727.
Amtmann D, Cook KF, Johnson KL, Cella D. The PROMIS initiative: involvement of rehabilitation stakeholders in development and examples of applications in rehabilitation research. Arch Phys Med Rehabil. 2011;92:S12–19.
Angstman KB, Bender RO, Bruce SM. Patient advisory groups in practice improvement: sample case presentation with a discussion of best practices. J Amb Care Mgmt. 2009;32:328–332.
Austin SA, Claiborne N. Faith wellness collaboration: a community-based approach to address type II diabetes disparities in an African-American community. Soc Work Health Care. 2011;50:360–375.
Boutain DM. Collective knowledge sharing as a social justice strategy: the difference it made in a service project about preterm birth disparity. ANS Adv Nurs Sci. 2009;32:E68–80.
Boyer BB, Mohatt GV, Pasker RL, Drew EM, McGlone KK. Sharing results from complex disease genetics studies: a community based participatory research approach. Int J Circumpolar Health. 2007;66:19–30.
Breslin M, Mullan RJ, Montori VM. The design of a decision aid about diabetes medications for use during the consultation with patients with type 2 diabetes. Pat Educ Couns. 2008;73:465–472.
Buckler AJ, Bresolin L, Dunnick NR, Sullivan DC. A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging. Radiology. 2011;258:906–914.
Chang DI, Burton A, O’Brien J, Hurley RE. Honesty as good policy: evaluating Maryland’s medicaid managed care program. Milbank. 2003;81:389–414.
Chung B, Corbett CE, Boulet B, et al. Talking wellness: a description of a community-academic partnered project to engage an African-American community around depression through the use of poetry, film, and photography. Ethn Dis. 2006;16:S67–78.
Chung B, Jones L, Dixon EL, Miranda J, Wells K. Using a community partnered participatory research approach to implement a randomized controlled trial: planning community partners in care. J Health Care Poor Underserved. 2010;21:780–795.
Cooley WC, McAllister JW, Sherrieb K, Clark RE. The Medical Home Index: development and validation of a new practice-level measure of implementation of the Medical Home model. Ambul Pediatr. 2003;3:173–180.
Corona R, Gonzalez T, Cohen R, Edwards C, Edmonds T. Richmond Latino needs assessment: a community-university partnership to identify health concerns and service needs for Latino youth. J Commun Health. 2009;34:195–201.
Crowe JL, Keifer MC, Salazar MK. Striving to provide opportunities for farm worker community participation in research. J Agric Saf Health. 2008;14:205–219.
Curran GM, Pyne J, Fortney JC, et al. Development and implementation of collaborative care for depression in HIV clinics. AIDS Care. 2011;23:1626–1636.
Davis RM, Hitch AD, Nichols M, Rizvi A, Salaam M, Mayer-Davis EJ. A collaborative approach to the recruitment and retention of minority patients with diabetes in rural community health centers. Contemp Clin Trials. 2009;30:63–70.
Egede LE, Ellis C. Development and testing of the multidimensional trust in health care systems scale. J Gen Intern Med. 2008;23:808–815.
Fredericks JE, Bunting RF Jr. Implementation of a patient-friendly medication schedule to improve patient safety within a healthcare system. J Healthc Risk Mgmt. 2010;29:22–27.
Freysteinson WM. The ethical community consultation model as preparation for nursing research: a case study. Nurs Ethics. 2010;17:749–758.
Gallagher LP, Truglio-Londrigan M, Levin R. Partnership for healthy living: an action research project. Nurs Res. 2009;16:7–29.
Garretson M, Walline V, Heisler J, Townsend J. New medical school engages rural communities to conduct regional health assessment. Fam Med. 2010;42:693–701.
Goins RT, Garroutte EM, Fox SL, Dee Geiger S, Manson SM. Theory and practice in participatory research: lessons from the native elder care study. Gerontology. 2011;51:285–294.
Hitchcock K, Appelt K. Sustainable collaboration for community outreach: lessons from the Spanish Access to Literature/Uso Directo (SALUD) project. J Med Libr Assoc. 2008;96:58–60.
James S, Arniella G, Bickell NA, et al. Community ACTION boards: an innovative model for effective community-academic research partnerships. Prog Commun Health Part. 2011;5:399–404.
Kaplan SA, Calman NS, Golub M, Ruddock C, Billings J. The role of faith-based institutions in addressing health disparities: a case study of an initiative in the southwest Bronx. J Health Care Poor Underserved. 2006;17:9–19.
Kaplan SA, Ruddock C, Golub M, et al. Stirring up the mud: using a community-based participatory approach to address health disparities through a faith-based initiative. J Health Care Poor Underserved. 2009;20:1111–1123.
Kataoka SH, Fuentes S, O’Donoghue VP, et al. A community participatory research partnership: the development of a faith-based intervention for children exposed to violence. Ethnic Dis. 2006;16:S89–97.
Katz JR, Martinez T, Paul R. Community-based participatory research and American Indian/Alaska Native nurse practitioners: a partnership to promote adolescent health. J Amer Acad Nurs Pract. 2011;23:298–304.
Kelly PJ, Lesser J, Cheng AL, et al. A prospective randomized controlled trial of an interpersonal violence prevention program with a Mexican American community. Fam Commun Health. 2010;33:207–215.
Kim S, Flaskerud JH, Koniak-Griffin D, Dixon EL. Using community-partnered participatory research to address health disparities in a Latino community. J Prof Nurs. 2005;21:199–209.
Kogan JN, Bauer MS, Dennehy EB, et al. Increasing minority research participation through collaboration with community outpatient clinics: the STEP-BD community partners experience. Clin Trials. 2009;6:344–354.
Laditka JN, Beard RL, Bryant LL, et al. Promoting cognitive health: a formative research collaboration of the healthy aging research network. Gerontology. 2009;49:S12–17.
Leape LL, Rogers G, Hanna D, et al. Developing and implementing new safe practices: voluntary adoption through statewide collaboratives. Q Saf Health Care. 2006;15:289–295.
Legaspi A, Orr E. Disseminating research on community health and well-being: a collaboration between Alaska Native villages and the academe. Am Indian Alsk Native Ment Health Res. 2007;14:24.
Lopez ED, Lichtenstein R, Lewis A, et al. Drawing from Freirian empowerment methods to develop and use innovative learning maps: increasing enrollment of uninsured children on Detroit’s eastside. Health Prom Pract. 2007;8:181–191.
McAllister CL, Green BL, Terry MA, Herman V, Mulvey L. Parents, practitioners, and researchers: community-based participatory research with early head start. Am J Pub Health. 2003;93:1672–1679.
McGhan WF, Al M, Doshi JA, Kamae I, Marx SE, Rindress D. The ISPOR good practices for quality improvement of cost-effectiveness research task force report. Value Health. 2009;12:1086–1099.
McKeithen T, Robertson S, Speight M. Developing clinical competencies to assess learning needs and outcomes: the experience of the CS2day initiative. J Cont Educ Health Prof. 2011;31:S21–27.
Meredith LS, Branstrom RB, Azocar F, Fikes R, Ettner SL. A collaborative approach to identifying effective incentives for mental health clinicians to improve depression care in a large managed behavioral healthcare organization. Adm Pol Ment Health. 2011;38:193–202.
Noe TD, Manson SM, Croy C, McGough H, Henderson JA, Buchwald DS. The influence of community-based participatory research principles on the likelihood of participation in health research in American Indian communities. Ethnic Dis. 2007;17:S6–14.
Peterson TH, Dolan T, Hanft S. Partnering with youth organizers to prevent violence: an analysis of relationships, power, and change. Prog Commun Health Part. 2010;4:235–242.
Pfefferle SG, Cooper B, Layton D, Rohrbach S. Early collaboration for adaptation: addressing depression in low-income new mothers. J Health Care Poor Underserved. 2009;20:539–544.
Pickard AS, Lee TA, Solem CT, Joo MJ, Schumock GT, Krishnan JA. Prioritizing comparative-effectiveness research topics via stakeholder involvement: an application in COPD. Clin Pharm Ther. 2011;90:888–892.
Raja S, Ball M, Booth J, Haberstro P, Veith K. Leveraging neighborhood-scale change for policy and program reform in Buffalo. N Y Am J Prev Med. 2009;37:S352–360.
Ray J, White M, Cannon P, Bowen C, O’Rourke K. Implementing the Florida KidCare open enrollment communications campaign: a framework for mobilizing community partners to reduce the number of uninsured children. Int Q Comm Health Educ. 2006;26:365–377.
Redwood D, Lanier A, Kemberling M, Klejka J, Sylvester I, Lundgren K. Community-based participatory research in a large cohort study of chronic diseases among Alaska native adults. Prog Commun Health Part. 2010;4:325–330.
Redwood Y, Schulz AJ, Israel BA, Yoshihama M, Wang CC, Kreuter M. Social, economic, and political processes that create built environment inequities: perspectives from urban African Americans in Atlanta. Fam Commun Health. 2010;33:53–67.
Rhodes SD, Hergenrather KC, Montano J, et al. Using community-based participatory research to develop an intervention to reduce HIV and STD infections among Latino men. AIDS Educ Prev. 2006;18:375–389.
Rhodes SD, Hergenrather KC, Vissman AT, et al. Boys must be men, and men must have sex with women: a qualitative CBPR study to explore sexual risk among African American, Latino, and White gay men and MSM. Am J Men’s Health. Mar 2011;5:140–151.
Sanchez RJ, Mardekian J, Cziraky MJ, Mullins CD. Developing a collaborative study protocol for combining payer-specific data and clinical trials for CER. J Man Care Pharm. 2011;17:S34–37.
Savage CL, Xu Y, Lee R, Rose BL, Kappesser M, Anthony JS. A case study in the use of community-based participatory research in public health nursing. Publ Health Nurs. 2006;23:472–478.
Schensul JJ, Robison J, Reyes C, Radda K, Gaztambide S, Disch W. Building interdisciplinary/intersectoral research partnerships for community-based mental health research with older minority adults. Am J Commun Psychol. 2006;38:79–93.
Shah ND, Mullan RJ, Breslin M, Yawn BP, Ting HH, Montori VM. Translating comparative effectiveness into practice: the case of diabetes medications. Med Care. 2010;48:S153–158.
Spatz ES, Phipps MS, Wang OJ, et al. Expanding the safety net of specialty care for the uninsured: a case study. Health Serv Res. 2012;47:344–362.
Spoth R, Redmond C, Clair S, Shin C, Greenberg M, Feinberg M. Preventing substance misuse through community-university partnerships: randomized controlled trial outcomes 4(1/2) years past baseline. Am J Prev Med. 2011;40:440–447.
Steinman KJ, Wright V, Cooksey E, Myers LJ, Price-Spratlen T, Ryles R. Collaborative research in a faith-based setting: Columbus congregations for healthy youth. Publ Health Rep. 2005;120:213–216.
Stewart AL, Napoles-Springer AM, Gregorich SE, Santoyo-Olsson J. Interpersonal processes of care survey: patient-reported measures for diverse groups. Health Serv Res. 2007;42:1235–1256.
Stewart MK, Colley D, Huff A, et al. Participatory development and implementation of a community research workshop: experiences from a community-based participatory research partnership. Prog Commun Health Part. 2009;3:165–178.
Stiffman AR, Freedenthal S, Brown E, Ostmann E, Hibbeler P. Field research with underserved minorities: the ideal and the real. J Urban Health. 2005;82:56–66.
Stockdale SE, Mendel P, Jones L, Arroyo W, Gilmore J. Assessing organizational readiness and change in community intervention research: framework for participatory evaluation. Ethnic Dis. 2006;16:S136–145.
Sullivan M, Bhuyan R, Senturia K, Shiu-Thornton S, Ciske S. Participatory action research in practice: a case study in addressing domestic violence in nine cultural communities. J Interpers Violence. 2005;20:977–995.
Tanjasiri S, Tran J, Palmer P, et al. Developing a community-based collaboration to reduce cancer health disparities among Pacific Islanders in California. Pac Health Dialogues. 2007;14:119.
Tanjasiri SP, Tran JH, Palmer PH, Valente TW. Network analysis of an organizational collaboration for Pacific Islander cancer control. J Health Care Poor Underserved. 2007;18:184–196.
Taylor-Piliae RE, Froelicher ES. Methods to optimize recruitment and retention to an exercise study in Chinese immigrants. Nurs Res. 2007;56:132–136.
Weiner J, Aguirre A, Ravenell K, et al. Designing an illustrated patient satisfaction instrument for low-literacy populations. Am J Man Care. 2004;10:853–860.
Wennerstrom A, Vannoy SD 3rd, Allen CE. Community-based participatory development of a community health worker mental health outreach role to extend collaborative care in post-Katrina New Orleans. Ethnic Dis. 2011;21(S1):45–51.
Wong C, Mouanoutoua V, Chen MJ. Engaging community in the quality of hypertension care project with Hmong Americans. J Cult Divers. 2008;15:30–36.
Wong FY, Crisostomo VA, Bao D, et al. Development and implementation of a collaborative, multistakeholder research and practice model on HIV prevention targeting Asian/Pacific Islander men in the United States who have sex with men. Am J Pub Health. 2011;101:623–631.
Zoellner J, Connell CL, Santell R, et al. Fit for life steps: results of a community walking intervention in the rural Mississippi delta. Prog Commun Health Part. 2007;1:49–60.
NIH. Health Care Systems (HCS) Research Collaboratory. http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-12-002.html. Accessed March 21, 2013.
Lieu TA, Au D, Krishnan JA, Moss M, Selker H, Harabin A, Taggart V, Connors A. Comparative effectiveness research in lung diseases and sleep disorders: recommendations from the National Heart, Lung, and Blood Institute workshop. Am J Resp Crit Care Med. 2011;184:848–856.
Agency for Healthcare Research & Quality. Comparative Effectiveness Research Portfolio. http://www.ahrq.gov/cpi/portfolios/comparative-effectiveness/index.html. Accessed March 21, 2014.
Hartung DMGJ, Fagnan LJ, Davis MM, Stange KC. The role of practice-based research networks in comparative effectiveness research. J Comp Eff Res. 2012;1:45–55.
Community-Based Participatory Research [Web page] http://obssr.od.nih.gov/scientific_areas/methodology/community_based_participatory_research/index.aspx. Accessed March 21, 2014.
Woolf SH. The 2009 breast cancer screening recommendations of the US Preventive Services Task Force. JAMA. 2010;303:162–3.
Patient Centered Outcomes Research Institute. Funding Center. http://www.pcori.org/funding-opportunities/funding-announcements/funding-center/, Accessed March 21, 2014.
Patient Centered Outcomes Research Institute. Research We Support. http://www.pcori.org/research-we-support/. Accessed March 21, 2014.
Morrato EH, Concannon TW, Meissner P, Shah ND, Turner BJ. Dissemination and implementation of comparative effectiveness evidence: key informant interviews with clinical and translational science award institutions. J Comp Eff Res. 2013;2:185–194.
Burke JG, Jones J, Yonas M, Guizzetti L, Virata MC, Costlow M, Morton SC, Elizabeth M. PCOR, CER, and CBPR: alphabet soup or complementary fields of health research? Clin Trans Sci. 2013;6:493–496.
Acknowledgements
Contributors
Ushahsi Basu, Ridita Nizam, and Madeleine Streit screened abstracts, retrieved and reviewed full text articles, and recorded data on the articles that were included in the review. Samuel Hirshman and Shawna Beck-Sullivan assisted in preparation of the manuscript.
The authors also wish to thank Grant P. Thompson and Gerald Rasmussen of the Consumer Reports Panel, Judith Bradford of Fenway Institute & Fenway Health, Lawrence Becker of Xerox, JoAnne Grunbaum of the Centers for Disease Control & Prevention, Eleanor Perfetto of Pfizer Inc., Julie Lynch of the University of Massachusetts-Boston, and Radley (Chris) Sheldrick of Tufts Medical Center, who participated as stakeholders on this research project. Further information on these individuals and their roles on the project is presented in the Methods section and in Table 1.
Funders
The authors were support by grants from the Agency for Healthcare Research and Quality (AHRQ) (K01 HS017726 and HHSA 290 2007 10055 I) and the National Center for Advancing Translational Science (NCATS), NIH (UL1 RR025752).
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Agency for Healthcare Research and Quality or the National Institutes of Health.
Prior Presentations
The data presented in this article have not been presented at any conference or in any other peer reviewed publication.
Conflict of Interest
The authors have no competing interests that bear on the content of this manuscript.
Author Contributions
Dr. Concannon took primary responsibility for conceiving and writing the manuscript, obtaining contributions from co-authors and managing stakeholder reviews and government clearances. All co-authors made intellectual contributions during the research design and analysis stages.
Author information
Authors and Affiliations
Corresponding author
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
Online Appendix A
(PDF 61 kb)
Online Appendix B
(PDF 118 kb)
Rights and permissions
About this article
Cite this article
Concannon, T.W., Fuster, M., Saunders, T. et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. J GEN INTERN MED 29, 1692–1701 (2014). https://doi.org/10.1007/s11606-014-2878-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-014-2878-x