Them and Us! Challenging the notion that qualitative research is somehow inferior to quantitative studies

By Dr Rosanna Cole

Are you a qualitative researcher, looking to enhance your coding protocols and show your data analysis in more depth? Do you want to discover easy to follow and transparent processes to present your data collection journey and subsequent results? 

It often feels like quantitative researchers are able to analyse complex data sets using neat tests whereas hundreds of hours of qualitative data can feel overwhelming and messy. Using multiple coders to analyse data and present the findings in succinct but thorough ways may help.

Them and Us?

Unless you are an experienced and effective mixed methods researcher, within some universities it can feel a bit ‘them and us’ because everyone has a side, right? Are you a quantitative researcher perplexed by the detail of embedded qualitative studies? Or a qualitative researcher baffled by the enormous data sets and number crunching of quantitative studies? Now, these are extremes of a research methods continuum but every scholar at some point has read an article where they don’t understand the data collection, analysis and results or have questioned the leap from those results to the final message. This is what makes research so fascinating and valuable after all, provided the study holds merit in its execution – and why we learn from one another’s scholarly studies.

A new resource to improve your data processes and present clearer findings

A new paper published in the SAGE journal Sociological Methods and Research clearly explains how qualitative researchers may like to consider using inter-rater reliability methods (IRR) in their case study research. IRR is the degree of agreement among independent observers who code or assess the same data. IRR methods may provide an opportunity to improve the transparency and consistency of qualitative case study data analysis in terms of the rigor of how codes and constructs have been developed from the raw data. The article offers useful tools to adopt to enhance qualitative work. For example, the paper provides a missing link in the literature between data gathering and analysis by expanding an existing process model from five to six stages. If you are unsure whether the IRR would suit your study, the paper identifies seven factors to determine the suitability of IRR to your work and it offers an IRR checklist which could be published within a manuscript as a demonstration of thorough and rigorous methods implementation. 

A missed opportunity

The reason there is such a huge opportunity for scholars to adopt better coding protocols is because few papers on qualitative research methods in the literature conduct IRR assessments or neglect to report them, despite some disclosure of multiple researcher teams and coding reconciliation in the work. In-depth discussion and reconciliation initiated by IRR may enhance the findings and theory that emerges from qualitative case study data analysis, where the main data source is often interview transcripts or field notes. Although scholars have offered exhaustive guides on how to select cases and collect data, far less attention has been paid to how to prepare and analyse the acquired data.

There is no ‘silver bullet’ to fix poor research

It must be noted that IRR is not a panacea to make research ‘better’, in response to criticisms of construct error and poor transparency of how data led to findings. For example, prior to embarking on the IRR process, the research design and data collection will need to be of high quality. IRR enhances some projects where consistency in interpretation, thus reducing interpretation bias, needs to be mitigated. It makes the journey from raw data to analysis more rigorous (through reconciliation discussion which is often implicit in qualitative research papers) and transparent (through explicit communication of the IRR measure) but does not necessarily lead to validity which would accurately reflect the phenomenon. The process promotes reconciliation, discussion and consensus building.

We (don’t) want to be like you

In short, qualitative researchers have much to offer the scholarly community in terms of rich, in-depth studies and the promotion of IRR is not about turning data analysis into a mechanical process. Rather, it is about providing an audit trail that gives the reader confidence in the process the researchers have gone through to unpack and reliably make sense of their rich data, allowing it to be heard and enabling the full potential of the method.

Article Details
Inter-Rater Reliability Methods in Qualitative Case Study Research
Rosanna Cole
First published online February 22, 2023
DOI: 10.1177/00491241231156971
Sociological Methods & Research

About the Author