How to Read a Paper: The Basics of Evidence-Based Medicine (34 page)

12
Herman WH, Edelstein SL, Ratner RE, et al. The 10-year cost-effectiveness of lifestyle intervention or metformin for diabetes prevention: an intent-to-treat analysis of the DPP/DPPOS.
Diabetes Care
2012;
35
(4):723–30.

13
Martin NK, Vickerman P, Miners A, et al. Cost-effectiveness of hepatitis C virus antiviral treatment for injection drug user populations.
Hepatology
2012;
55
(1):49–57.

14
Saariniemi KM, Kuokkanen HO, Räsänen P, et al. The cost utility of reduction mammaplasty at medium-term follow-up: a prospective study.
Journal of Plastic, Reconstructive & Aesthetic Surgery
2012;
65
(1):17–21.

15
Shahab L:
Cost-effectiveness of pharmacotherapy for smoking cessation
. London: National Centre for Smoking Cessation and Training (NCSCT), 2012 Available online http://www.ncsct.co.uk/usr/pub/B7_Cost-effectiveness_pharmacotherapy.pdf; accessed 5.11.13.

16
Coyle D, Coyle K, Cameron C, et al. Cost-effectiveness of new oral anticoagulants compared with warfarin in preventing stroke and other cardiovascular events in patients with atrial fibrillation.
Value in Health
2013;
16
:498–506.

17
Frederix GW, Severens JL, Hövels AM, et al. Reviewing the cost-effectiveness of endocrine early breast cancer therapies: influence of differences in modeling methods on outcomes.
Value in Health
2012;
15
(1):94–105.

18
Harris J. QALYfying the value of life.
Journal of Medical Ethics
1987;
13
(3):117–23.

19
Whitehead SJ, Ali S. Health outcomes in economic evaluation: the QALY and utilities.
British Medical Bulletin
2010;
96
(1):5–21.

20
Gold MR, Stevenson D, Fryback DG. HALYS and QALYS and DALYS, Oh My: similarities and differences in summary measures of population health.
Annual Review of Public Health
2002;
23
(1):115–34.

Chapter 12

Papers that go beyond numbers (qualitative research)

What is qualitative research?

Twenty-five years ago, when I took up my first research post, a work-weary colleague advised me: ‘Find something to measure, and keep on measuring it until you’ve got a boxful of data. Then stop measuring and start writing up'.

‘But what should I measure?’, I asked.

‘That’, he said cynically, ‘doesn’t much matter'.

This true example illustrates the limitations of an exclusively quantitative (counting-and-measuring) perspective in research. Epidemiologist Nick Black has argued that a finding or a result is more likely to be accepted as a fact if it is quantified (expressed in numbers) than if it is not [1]. There is little or no scientific evidence, for example, to support the well-known ‘facts’ that one couple in 10 is infertile or that one person in 10 is homosexual. Yet, observes Black, most of us are happy to accept uncritically such simplified, reductionist and blatantly incorrect statements so long as they contain at least one number.

Qualitative researchers seek a deeper truth. They aim to ‘study things in their natural setting, attempting to make sense of, or interpret, phenomena in terms of the meanings people bring to them’ [2], and they use ‘a holistic perspective which preserves the complexities of human behaviour’ [2].

Interpretive or qualitative research was for years the territory of the social scientists. It is now increasingly recognised as being not just complementary to but, in many cases, a prerequisite for the quantitative research with which most us who trained in the biomedical sciences are more familiar. Certainly, the view that the two approaches are mutually exclusive has itself become ‘unscientific’, and it is currently rather trendy, particularly in the fields of primary care and health services research, to say that you are doing some qualitative research—and since the first edition of this book was published, qualitative research has even become mainstream within the evidence-based medicine movement [3] [4], and, as described in Chapter 7, there have been major developments in the science of integrating qualitative and quantitative evidence in the development and evaluation of complex interventions.

The late Dr Cecil Helman, an anthropologist as well as a medical doctor, told me the following story to illustrate the qualitative–quantitative dichotomy. A small child runs in from the garden and says, excitedly, ‘Mummy, the leaves are falling off the trees’.

‘Tell me more’, says his mother.

‘Well, five leaves fell in the first hour, then ten leaves fell in the second hour …’

That child will become a quantitative researcher.

A second child, when asked ‘tell me more’, might reply, ‘Well, the leaves are big and flat, and mostly yellow or red, and they seem to be falling off some trees but not others. And mummy, why did no leaves fall last month?’

That child will become a qualitative researcher.

Questions such as ‘How many parents would consult their general practitioner when their child has a mild temperature?’, or ‘What proportion of smokers have tried to give up?’ clearly need answering through quantitative methods. But questions like ‘Why do parents worry so much about their children’s temperature?', and ‘What stops people giving up smoking?’ cannot and should not be answered by leaping in and measuring the first aspect of the problem that we (the outsiders) think might be important. Rather, we need to hang out, listen to what people have to say and explore the ideas and concerns that the individuals themselves come up with. After a while, we may notice a pattern emerging, which may prompt us to make our observations in a different way. We may start with one of the methods shown in
Table 12.1
, and go on to use a selection of others.

Table 12.1
Examples of qualitative research methods

Box 12.1, which is reproduced with permission from Nick Mays and Catherine Pope's introductory paper ‘Qualitative Research in Health Care’ [5] summarises (indeed overstates) the differences between the qualitative and quantitative approaches to research. In reality, there is a great deal of overlap between them, the importance of which is increasingly being recognised [6].

As section ‘Three preliminary questions to get your bearings’ explains, quantitative research should begin with an idea (usually articulated as a hypothesis), which then, through measurement, generates data and, by
deduction
, allows a conclusion to be drawn. Qualitative research is different. It begins with an intention to explore a particular area, collects ‘data’ (e.g. observations, interviews, documents—even emails can count as qualitative data), and generates ideas and hypotheses from these data largely through what is known as
inductive reasoning
[2]. The strength of quantitative approach lies in its
reliability
(repeatability)—that is, the same measurements should yield the same results time after time. The strength of qualitative research lies in
validity
(closeness to the truth)—that is, good qualitative research, using a selection of data collection methods, really should touch the core of what is going on rather than just skimming the surface. The validity of qualitative methods is said to be greatly improved by the use of more than one method (see
Table 12.1
) in combination (a process sometimes known as
triangulation
), by the researcher thinking carefully about what is going on and how their own perspective might be influencing the data (an approach known as
reflexivity
) [7], and—some would argue—by more than one researcher analysing the same data independently (to demonstrate
inter-rater reliability
).

Box 12.1 Qualitative versus quantitative research—the overstated dichotomy > (see reference [7])

Since I wrote the first edition of this book, inter-rater reliability has become less credible as a measure of quality in qualitative research. Appraisers of qualitative papers increasingly seek to assess the competence and reflexivity of a single researcher rather than confirm that the findings were ‘checked by someone else’. This change is attributable to two important insights. First, in most qualitative research, one person knows the data far better than anyone else, so the idea that two heads are better than one simply isn't true—a researcher who has been brought in merely to verify ‘themes’ may rely far more on personal preconceptions and guesswork than the main field worker. And second, with the trend towards more people from biomedical backgrounds doing qualitative research, it's not at all uncommon for two (or even a whole team of) naïve and untrained researchers setting up focus groups or attacking the free-text responses of questionnaires. Not only does ‘agreement’ between these individuals not correspond to quality but teams from similar backgrounds are also likely to bring similar biases, so high inter-rater reliability scores may be entirely spurious.

Those who are ignorant about qualitative research often believe that it constitutes little more than hanging out and watching leaves fall. It is beyond the scope of this book to take you through the substantial literature on how to (and how not to) proceed when observing, interviewing, leading a focus group, and so on. But sophisticated methods for all these techniques certainly exist, and if you are interested I suggest you try the excellent BMJ series by Scott Reeves and colleagues from Canada [8–12].

Qualitative methods really come into their own when researching uncharted territory—that is, where the variables of greatest concern are poorly understood, ill-defined and cannot be controlled. In such circumstances, the definitive hypothesis may not be arrived at until the study is well under way. But it is in precisely these circumstances that the qualitative researcher must ensure that he or she has, at the outset, carefully delineated a particular focus of research and identified some specific questions to try to answer (see Question One in section ‘Evaluating papers that describe qualitative research’). The methods of qualitative research allow for—indeed, they require—modification of the research question in the light of findings generated along the way—a technique known as
progressive focusing
[5]. (In contrast, as ‘d’ of section ‘Have the authors set the scene correctly?’ showed, sneaking a look at the interim results of a quantitative study is statistically invalid!).

The so-called
iterative
approach (altering the research methods and the hypothesis as you go along) employed by qualitative researchers shows a commendable sensitivity to the richness and variability of the topic. Failure to recognise the legitimacy of this approach has, in the past, led critics to accuse qualitative researchers of continually moving their own goalposts. Whilst these criticisms are often misguided, there is a danger that when qualitative research is undertaken unrigorously by naïve researchers, the ‘iterative’ approach will slide into confusion. This is one reason why qualitative researchers must allow periods away from their fieldwork for reflection, planning and consultation with colleagues.