Review
A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews

https://doi.org/10.1016/j.ijnurstu.2009.01.009Get rights and content

Abstract

A new form of literature review has emerged, Mixed Studies Review (MSR). These reviews include qualitative, quantitative and mixed methods studies. In the present paper, we examine MSRs in health sciences, and provide guidance on processes that should be included and reported. However, there are no valid and usable criteria for concomitantly appraising the methodological quality of the qualitative, quantitative and mixed methods studies.

Objective

To propose criteria for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies or study components.

Design

A three-step critical review was conducted.

Data sources

2322 references were identified in MEDLINE, and their titles and abstracts were screened; 149 potentially relevant references were selected and the full-text papers were examined; 59 MSRs were retained and scrutinized using a deductive–inductive qualitative thematic data analysis. This revealed three types of MSR: convenience, reproducible, and systematic.

Review methods

Guided by a proposal, we conducted a qualitative thematic data analysis of the quality appraisal procedures used in the 17 systematic MSRs (SMSRs).

Results

Of 17 SMSRs, 12 showed clear quality appraisal procedures with explicit criteria but no SMSR used valid checklists to concomitantly appraise qualitative, quantitative and mixed methods studies. In two SMSRs, criteria were developed following a specific procedure. Checklists usually contained more criteria than needed. In four SMSRs, a reliability assessment was described or mentioned. While criteria for quality appraisal were usually based on descriptors that require specific methodological expertise (e.g., appropriateness), no SMSR described the fit between reviewers’ expertise and appraised studies. Quality appraisal usually resulted in studies being ranked by methodological quality.

Conclusion

A scoring system is proposed for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies for SMSRs. This scoring system may also be used to appraise the methodological quality of qualitative, quantitative and mixed methods components of mixed methods research.

Introduction

The use of mixed methods research increases in health sciences (Creswell et al., 2004). Mixed methods research is defined as a combination of qualitative and quantitative methods conducted by a researcher or researcher team, for the broad purpose of gaining breadth and depth of understanding or corroboration, within a single study or closely related studies (Johnson et al., 2007). For example, mixed methods research may combine a quantitative cross-sectional survey on the accessibility of mental healthcare with a qualitative ethnographic study to better understand the lack of access in a culturally diverse community (Groleau et al., 2007). While combining qualitative and quantitative methods constitutes a longstanding practice in evaluation and research (Greene, 2006, Pluye et al., 2009b), it has only recently been conceptualized in terms of mixed methods research (Tashakkori and Teddlie, 2003).

The division of methods within health sciences as qualitative or quantitative has its roots in the different ‘world views’ of constructivism and logical empiricism, which are usually presented as competing paradigms (Creswell and Plano Clark, 2007, Greene, 2007, Johnson et al., 2007, Pluye et al., 2009b, Teddlie and Tashakkori, 2003). Constructivism is associated with idealism, relativism and (inter)subjectivity, while logical empiricism is associated with materialism, realism and objectivity. Constructivism is most frequently associated with inductive qualitative studies, and logical empiricism is most frequently associated with deductive quantitative studies. Mixed methods may be conceived as methods that loop between constructivism and logical empiricism, and include the notion that something can be “both socially constructed and yet real” (Hacking, 1999, p. 119).

There is no consensus on criteria for appraising the methodological quality of mixed methods research (O’Cathain et al., 2008). While there are general criteria for planning, designing, reporting and globally assessing mixed methods research (Creswell and Plano Clark, 2007, O’Cathain et al., 2008), researchers must refine further the criteria for evaluating its quality (Creswell et al., 2004). In the present paper, we examine systematic Mixed Studies Reviews, and criteria for appraising the quality of qualitative, quantitative and mixed methods studies.

A mixed studies review (MSR) is a literature review that concomitantly examines qualitative, quantitative and mixed methods primary studies. We found examples of MSR in education, health, management and social sciences such as reviews to synthesize existing knowledge about an intervention or a program that is to be evaluated. We conceptualize MSR as mixed methods research where data consist of the text of publications reporting qualitative and quantitative studies and/or mixed methods studies.

In the present paper, we review MSRs in health sciences, define three categories of MSRs (convenience, reproducible and systematic), and provide guidance on processes that should be included and reported. Our main objective is to critically scrutinize the different ‘quality appraisal tools’ used in systematic MSRs, and propose a scoring system for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies in a systematic MSR. This scoring system may also be used to appraise the methodological quality of qualitative, quantitative and mixed methods components of mixed methods research.

Section snippets

Mixed studies review as an emerging form of literature review in health sciences

We use the term MSR to refer to reviews with many different names such as integrative review, meta-needs assessment, mixed approaches to evidence synthesis, mixed methods review, mixed methods synthesis, mixed research synthesis, and realist review. These terms and corresponding references are presented in Table 1. This profusion of terminology is testament to the significant desire for better comprehension and conceptualization of MSR. We suggest the term ‘Mixed Studies Review’ as a generic

Data sources and sampling

We identified and examined 59 MSRs indexed in MEDLINE up to March 2006.

Description of the sample

Of the 59 retained health-related MSRs, 56 (95%) were published after 2000. MSRs were conducted in the following disciplines: nursing (24%), psychosocial and behavioural research (19%), health services and policy research (14%), population health (8%) and aging research (8%). First authors were affiliated with US and UK universities for 23 (39%) and 17 (29%) reviews, respectively. Other first authors’ affiliation countries were Netherlands (N = 5), Sweden (n = 4), Canada (n = 3), Finland (n = 2),

Discussion

Results support the usability of the proposed 15 characteristics for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies in a SMSR context. The content validity of this initial set of criteria is supported by the literature and our review of SMSRs in health sciences. This minimum set may decrease the burden on reviewers. A revised set of criteria is presented in the next Section 5.1.

In our sample, the fact that only two of 17 SMSRs (12%)

Conclusion

The rationale for conducting mixed methods research lies in combining the strengths of both qualitative and quantitative studies. Typically, the former studies provide in-depth descriptions of complex phenomena that are context-specific, but may suggest theoretical and methodological lessons transferable to other contexts, while the latter examine observations or causal relationships that may be generalized using statistical inferences. In the present paper, we define MSR as a form of mixed

Conflict of interest

None.

Acknowledgements

Pierre Pluye and Marie-Pierre Gagnon hold a New Investigator Award from the Canadian Institutes of Health Research. This work was also supported by the “Fonds de recherche en santé du Québec”. Authors gratefully acknowledge the suggestions of Bryce Mansell, Laura Shea, Robbyn Seller and Tara Bambrick. Preliminary versions of this paper have been presented at the Conference of Qualitative Inquiry (Urbana-Champaign, 2006), the Canadian Cochrane Symposium (Ottawa, 2007), and the Mixed Methods

References (64)

  • V.J. Caracelli et al.

    Data analysis strategies for mixed-method evaluation designs

    Educational Evaluation and Policy Analysis

    (1993)
  • E.G. Carmines et al.

    Reliability and Validity Assessment

    (1979)
  • A.F. Chalmers

    What is This Thing Called Science?

    (1999)
  • I. Chalmers et al.

    A brief history of research synthesis

    Evaluation and The Health Professions

    (2002)
  • M. Clarke et al.

    Cochrane Reviewers’ Handbook 4.1.6 [Updated January 2003]

    (2003)
  • H. Cooper

    Synthesizing Research: A Guide for Literature Reviews

    (1998)
  • E. Coren et al.

    The Conduct of Systematic Research Reviews for SCI Knowledge Reviews

    (2006)
  • J.W. Creswell et al.

    Designing and Conducting Mixed Methods Research

    (2007)
  • J.W. Creswell et al.

    Designing a mixed methods study in primary care

    Annals of Family Medicine

    (2004)
  • M. Dixon-Woods et al.

    Including qualitative research in systematic reviews: opportunities and problems

    Journal of Evaluation in Clinical Practice

    (2001)
  • M. Dixon-Woods et al.

    Synthesising qualitative and quantitative evidence: a review of possible methods

    Journal of Health Services Research and Policy

    (2005)
  • EPPI-Centre: The Evidence for Policy and Practice Information and Co-ordinating Centre, 2007. EPPI-Centre Methods for...
  • A. Fink

    Conducting Research Literature Reviews: From the Internet to Paper

    (2005)
  • A. Forbes et al.

    Methodological strategies for the identification and synthesis of ‘evidence’ to support decision-making in relation to complex healthcare systems and practices

    Nursing Inquiry

    (2002)
  • J.L. Galvan

    Writing Literature Reviews: A Guide for Students of the Social and Behavioral Sciences

    (2006)
  • J. Garrard

    Health Sciences Literature Review Made Easy: The Matrix Method

    (2007)
  • J.C. Greene

    Toward a methodology of mixed methods social inquiry

    Research in the Schools

    (2006)
  • J.C. Greene

    Mixed Methods in Social Inquiry

    (2007)
  • D. Groleau et al.

    A mixed methods approach to the cultural understanding of distress and the non-use of mental health services

    Journal of Mental Health

    (2007)
  • I. Hacking

    The Social Construction of What?

    (1999)
  • A. Harden et al.

    Applying systematic review methods to studies of people's views: an example from public health research

    Journal of Epidemiology Community Health

    (2004)
  • A. Harden et al.

    Methodological issues in combining diverse study types in systematic reviews

    International Journal of Social Research Methodology

    (2005)
  • Cited by (725)

    View all citing articles on Scopus
    View full text