Review of Mixing Methods in Psychology
This collection of 12 essays attempts to justify the relevance of qualitative methods in psychological research, and to give theoretical arguments and practical advice on the integration of qualitative and quantitative methods. As with most edited books, some chapters are more illuminating and helpful than others. We highly recommend this book as an important step in the advancement of psychological methodology; we also believe this book to be of great import to the HCI community, and will focus on those chapters we believe to be of high quality and particular relevance to HCI.
Mixing Methods in Psychology:
The Integration of Qualitative and Quantitative Methods in Theory and Practice
Edited by Zazie Todd, Brigitte Berlich, Suzanne McKeown and
David D. Clarke (2004) Psychology Press
(Softcover: $31.95, Hardcover: $80)
This collection of 12 essays attempts to justify the relevance of qualitative
methods in psychological research, and to give theoretical arguments and
practical advice on the integration of qualitative and quantitative methods. As
with most edited books, some chapters are more illuminating and helpful than
others. We highly recommend this book as an important step in the advancement of
psychological methodology; we also believe this book to be of great import to
the HCI community, and will focus on those chapters we believe to be of high
quality and particular relevance to HCI.
The subjective nature of qualitative methods has often been a source of
criticism, but quantitative methods are problematic as well; Benjamin Disraeli
once commented â€œThere are lies, damn lies, and statistics.â€ In this collection
of essays, several experimental psychologists attempt to bridge the divide
between qualitative and quantitative methods, both by expressing theoretical
arguments for their integration (such as triangulation of results), and by
giving examples of research methods and experiments that successfully integrate
The first section of the book outlines theoretical and historical arguments for mixing methods. In the first chapter, Todd, Nerlich, and McKeown discuss the importance of integrating qualitative and quantitative methods. The chapter discusses a variety of reasons for using multiple methods, which can be categorized as supporting additional validity, properly staging experiments, and bridging the methodological divides.
In the second chapter, â€œComing full (hermeneutic) circleâ€, Nerlich defends qualitative methods by providing a comparative review of the history and philosophy of qualitative and quantitative method use in psychology. Although compelling and well written, this chapter will mainly be of interest to the more theoretically-grounded members of the HCI community. The third chapter, Henwoodâ€™s â€œReinventing validityâ€, also provides a justification for mixing methods, but one heavily biased by perceptions of clinical psychologists, and as such is not a critical read for the HCI community.
The second section deals with actual practice of experiments involving mixed quantitative and qualitative methods. HarrÃ© and Crystalâ€™s chapter, â€œDiscursive analysis and the interpretation of statisticsâ€, provides a model for much of the practices described in the book; a qualitative method is used initially to determine components for a later quantitative analysis.
David D. Clarkeâ€™s essay, â€œâ€™Structured judgement methodsâ€™ â€“ the best of both worlds?â€ is perhaps the most relevant to HCI theory and practice. His arguments for integrating methods, along with his own integrative methodology, line up nicely with the general trend in HCI research to resolve the separation of heuristic and formal approaches.
Clarkeâ€™s insight into the problem with single-method approaches is perhaps not novel, but well-constructed nonetheless. He notes that purely quantitative models give rise to repeatable, verifiable research with (potentially) limited practical value, while purely qualitative methods are suspect for their reliance on the bias of the observer. He also notes that quantitative methods give more readily comparable results, but results that may have been stripped of critical context and meaning.
Clarkeâ€™s proposed integrated method, named â€œstructured judgement methodsâ€ (p. 86) comes from research in forecasting. The method integrates a number of participants actively (and iteratively) making forecasts, assessing those against real outcomes, and eventually producing a quantitatively valid set of predictive heuristics. He shows that this knowledge-based approach produces much more reliable forecast models than purely quantitative (i.e. model-based) approaches.
In â€œQ method and qualiquantologyâ€, Stenner and Stainton Rogers present an engaging and well-constructed argument for including qualitative methods to bolster the validity of factor analysis. The authors do warn, however, that Q methodology itself lacks mainstream acceptance as a method, although it might be relevant for analyzing technology adoption, and we believe is flexible enough to allow for an adoption analysis to comprehend technology appropriation, should it be encountered.
In â€œValuing the â€˜value of lifeâ€™â€, Chilton et al., present a case for the use of qualitative methods alongside contingent valuation (CV) as a mechanism to analyze costs and risks. This research could be readily applied to HCI, despite the authorsâ€™ primary concern with physical safety (a factor only occasionally relevant in HCI). Primarily, the method involves providing interview-based context to informed decisions about tradeoff analysis; we believe this technique could be used to provided quantitative validation of claims analysis.
In â€œMethod and methodology on interpretive studies of cognitive lifeâ€, Vann and Cole engage in aggressively Vygotskian research of two cognitive tasks (rebus interpretation and order filling). Researchers engaged in Cognitive Work Analysis may find the combination of methods in this chapter particularly relevant.
In â€œIntegrating survey and focus group researchâ€, Todd and Lobeck discuss a key problem of mixed methods: conflicting results. Like all problematic research, this generates more questions than answers, but the authors end the chapter with a thorough analysis of the potential causes for the conflicts. This analysis can provide both comfort and useful direction to any researcher facing conflicting data.
The final section of the book deals with larger scale questions of when and how to mix methods. Millerâ€™s chapter, â€œEducational psychology and difficult behaviorâ€, deals with an historical analysis of the methods used to study student behavior in public schools in both Britain and the US. This chapter would provide a good foundation for a methodologically-aware literature review in HCI. Nicolsonâ€™s chapter, â€œTaking quality seriously: The case for qualitative feminist psychology in the context of quantitative clinical research on postnatal depressionâ€, is far too grounded is psychology to be readily taken into another domain. In addition, we are not sure if the community is quite ready for a feminist theory of HCI.
The book finishes with a chapter on â€œFuture directionsâ€, in which Todd and
Nerlich make several practical recommendations for integrating qualitative and
quantitative methods in both training and research, with particular respect to
the analysis of existing data sets. The authors conclude by asserting that this
avenue of research will increase in relevance, an assertion that we believe is
relevant for HCI, as well. Any such approach must take seriously the complexity
of mixed-method research, and this book provides useful guidance.
Much HCI research and practice is Vygotskian in nature; we actively modify the environment (system) as we research, evolving both our product and our process throughout. As a result, practices that can increase validity are of particular value for our practical domain. Merging qualitative and quantitative methods must be done carefully; otherwise, considerable time can be spent adding little or no additional validation. This book provides excellent theoretical and practical guidance on why, when, and how multiple methods can be integrated; while written for a more traditional audience of experimental psychologists, HCI researchers and practitioners can find much of value in this book.
Joshua B. Gross and Scott M. Robertson
School of Information Sciences and Technology
The Pennsylvania State University
311B IST Building The Pennsylvania State University
University Park, PA 16802 USA