Bad Science (10 page)

Forty years ago a man called Austin Bradford Hill, the grandfather of modern medical research, who was key in discovering the link between smoking and lung cancer, wrote out a set of guidelines, a kind of tick list, for assessing causality and a relationship between an exposure and an outcome. These are the cornerstone of evidence-based medicine, and often worth having at the back of your mind: it needs to be a strong association, which is consistent, and specific to the thing you are studying, where the putative cause comes before the supposed effect in time; ideally there should be a biological gradient, such as a dose-response effect; it should be consistent or at least not completely at odds with what is already known (because extraordinary claims require extraordinary evidence); and it should be biologically plausible.

Michael van Straten here has got biological plausibility, and little else. Medics and academics are very wary of people’s making claims on such tenuous grounds, because it’s something you get a lot from people with something to sell—specifically, drug companies.

Drug companies are very keen to promote theoretical advantages (“It works more on the Z4 receptor, so it must have fewer side effects!”), animal experiment data, or “surrogate outcomes” (“It improves blood test results; it must be protective against heart attacks!”) as evidence of the efficacy or superiority of their products. Many of the more detailed popular nutritionist books, should you ever be lucky enough to read them, play this classic drug company card very assertively. They will claim, for example, that a “placebo-controlled randomized control trial” has shown
benefits
from a particular vitamin, when what they mean is, it showed changes in a “surrogate outcome.”

For example, the trial may merely have shown that there were measurably increased amounts of the vitamin in the bloodstream after you took a vitamin, compared with placebo, which is a pretty unspectacular finding in itself; yet this is presented to the unsuspecting lay reader as a positive trial. Or the trial may have shown that there were changes in some other blood marker, perhaps the level of an ill-understood immune system component, which, again, the media nutritionist will present as concrete evidence of a real-world benefit.

There are problems with using such surrogate outcomes. They are often only tenuously associated with the real disease, in a very abstract theoretical model, and often developed in the very idealized world of an experimental animal, genetically inbred, kept under conditions of tight physiological control. A surrogate outcome can, of course, be used to generate and examine hypotheses about a real disease in a real person, but it needs to be very carefully validated. Does it show a clear dose-response relationship? Is it a true predictor of disease or merely a “covariable,” something that is related to the disease in a different way (e.g., caused
by
it rather than involved in
causing
it)? Is there a well-defined cutoff between normal and abnormal values?

All I am doing, I should be clear, is taking the feted media nutritionists at their own word. They present themselves as men and women of science, fill their columns, TV shows, and books with references to scientific research. I am subjecting their claims to the exact same level of very basic, uncomplicated rigor that I would deploy for any new theoretical work, any drug company claim and pill marketing rhetoric, and so on.

It’s not unreasonable to use surrogate outcome data, as they do, but those who are in the know are always circumspect. We’re
interested
in early theoretical work, but often the message is: “It might be a bit more complicated than that…” You’d only want to accord a surrogate outcome any significance if you’d read everything on it yourself, or if you could be absolutely certain that the person assuring you of its validity was extremely capable and was giving a sound appraisal of all the research in a given field, and so on.

Similar problems arise with animal data. Nobody could deny that this kind of data is valuable in the theoretical domain, for developing hypotheses, or suggesting safety risks, when cautiously appraised. But media nutritionists, in their eagerness to make lifestyle claims, are all too often blind to the problems of applying these isolated theoretical nuggets to humans, and anyone would think they were just trawling the Internet looking for random bits of science to sell their pills and expertise (imagine that). Both the tissue and the disease in an animal model, after all, may be very different from those in a living human system, and these problems are even greater with a lab dish model. Giving unusually high doses of chemicals to animals can distort the usual metabolic pathways, and give misleading results, and so on. Just because something can upregulate or downregulate something in a model doesn’t mean it will have the effect you expect in a person, as we shall see with the stunning truth about antioxidants.

And what about turmeric, which we were talking about before I tried to show you the entire world of applying theoretical research in this tiny grain of spice? Well, yes, there is some evidence that curcumin, a chemical in turmeric, is highly biologically active, in all kinds of different ways, on all kinds of different systems (there are also theoretical grounds for believing that it may be carcinogenic, mind you). It’s certainly a valid target for research.

But for the claim that we should eat more curry in order to get more of it, that “recent research” has shown it is “highly protective against many forms of cancer, especially of the prostate,” you might want to step back and put the theoretical claims in the context of your body. Very little of the curcumin you eat is absorbed. You have to eat a few grams of it to reach significant detectable serum levels, but to get a few grams of
curcumin
, you’d have to eat one hundred grams of
turmeric
, and good luck with that. Between research and recipe, there’s a lot more to think about than the nutritionists might tell you.

There have been an estimated fifteen million medical academic articles published so far, and five thousand journals are published every month. Many of these articles will contain contradictory claims; picking out what’s relevant—and what’s not—is a gargantuan task. Inevitably people will take shortcuts. We rely on review articles, or on meta-analyses, or textbooks, or hearsay, or chatty journalistic reviews of a subject.

That’s if your interest is in getting to the truth of the matter. What if you’ve just got a point to prove? There are few opinions so absurd that you couldn’t find at least one person with a Ph.D. somewhere in the world to endorse them for you; and similarly, there are few propositions in medicine so ridiculous that you couldn’t conjure up some kind of published experimental evidence somewhere to support them, if you didn’t mind its being a tenuous relationship and cherry-picked the literature, quoting only the studies that were in your favor.

One of the great studies of cherry-picking in the academic literature comes from an article about Linus Pauling, the great-grandfather of modern nutritionism, and his seminal work on vitamin C and the common cold. In 1993 Paul Knipschild, professor of epidemiology at the University of Maastricht, published a chapter on Pauling in the mighty textbook
Systematic Reviews
; he had gone to the extraordinary trouble of approaching the literature as it stood when Pauling was working and subjecting it to the same rigorous systematic review that you would find in a modern paper.

He found that while some trials did suggest that vitamin C had some benefits, Pauling had selectively quoted from the literature to prove his point. Where Pauling had referred to some trials that seriously challenged his theory, it was to dismiss them as methodologically flawed; but as a cold examination showed, so too were papers he quoted favorably in support of his own case.

In Pauling’s defense, his was an era when people knew no better, and he was probably quite unaware of what he was doing, but today cherry-picking is one of the most common dubious practices in alternative therapies, particularly in nutritionism, where it seems to be accepted essentially as normal practice (it is this cherry-picking, in reality, that helps characterize what alternative therapists conceive of, rather grandly, as their alternative paradigm). It happens in mainstream medicine also, but with one crucial difference: there it is recognized as a major problem, and hard work has been done to derive a solution.

That solution is a process called systematic review. Instead of just mooching around online and picking out your favorite papers to back up your prejudices and help you sell a product, in a systematic review you have an explicit search strategy for seeking out data (openly described in your paper, even including the search terms you used on databases of research papers), you tabulate the characteristics of each study you find, you measure—ideally blind to the results—the methodological quality of each one (to see how much of a “fair test” it is), you compare alternatives, and then finally you give a critical, weighted summary.

This is what the Cochrane Collaboration does on all the health care topics that it can find. It even invites people to submit new clinical questions that need answers. This careful sifting of information has revealed huge gaps in knowledge, it has revealed that “best practices” were sometimes murderously flawed, and simply by sifting methodically through preexisting data, it has saved more lives than you could possibly imagine. In the nineteenth century, as the public health doctor Muir Gray has said, we made great advances through the provision of clean, clear water; in the twenty-first century we will make the same advances through clean, clear information. Systematic reviews are one of the great ideas of modern thought. They should be celebrated.

Problematizing Antioxidants

We have seen the kinds of errors made by those in the nutritionism movement as they strive to justify their more obscure and technical claims. What’s more fun is to take our new understanding and apply it to one of the key claims of the nutritionism movement, indeed to a fairly widespread belief in general: the claim that you should eat more antioxidants.

As you now know, there are lots of ways of deciding whether the totality of research evidence for a given claim stacks up, and it’s rare that one single piece of information clinches it. In the case of a claim about food, for example, there are all kinds of different things we might look for: whether it is theoretically plausible, whether it is backed up by what we know from observing diets and health, whether it is supported by “intervention trials,” in which we give one group one diet and another group a different one, and whether those trials measured real-world outcomes, like “death,” or a surrogate outcome, like a blood test, which is only hypothetically related to a disease.

My aim here is by no means to suggest that antioxidants are
entirely
irrelevant to health. If I had a T-shirt slogan for this whole book, it would be: “I think you’ll find it’s a bit more complicated than that.” I intend, as they say, to “problematize” the prevailing nutritionist view on antioxidants, which currently lags only about twenty years behind the research evidence.

From an entirely theoretical perspective, the idea that antioxidants are beneficial for health is an attractive one. When I was a medical student—not so long ago—the most popular biochemistry textbook was called
Stryer
. This enormous book is filled with complex interlocking flowcharts of how chemicals, which is what you are made of, move through the body. It shows how different enzymes break down food into its constituent molecular elements, how these are absorbed, how they are reassembled into new, larger molecules that your body needs to build muscles, retina, nerves, bone, hair, membrane, mucus, and everything else that you’re made of; how the various forms of fats are broken down and reassembled into new forms of fat; or how different forms of molecule—sugar, fat, even alcohol—are broken down gradually, step by step, to release energy, and how that energy is transported, and how the incidental products from that process are used, or bolted onto something else to be transported in the blood, and then ditched at the kidneys, or metabolized down into further constituents, or turned into something useful elsewhere, and so on. This is one of the great miracles of life, and it is endlessly, beautifully, intricately fascinating.

When you look at these enormous, overwhelming interlocking webs, it’s hard not to be struck by the versatility of the human body and how it can perform acts of near alchemy from so many different starting points. It would be very easy to pick one of the elements of these vast interlocking systems and become fixated on the idea that it is uniquely important. Perhaps it appears a lot on the diagram; or perhaps rarely, and seems to serve a uniquely important function in one key place. It would be easy to assume that if there were more of it around, then that function would be performed with greater efficiency.

But as with all enormous interlocking systems—societies, for example, or businesses—an intervention in one place can have quite unexpected consequences; there are feedback mechanisms, compensatory mechanisms. Rates of change in one localized area can be limited by quite unexpected factors that are entirely remote from what you are altering, and excesses of one thing in one place can distort the usual pathways and flows, to give counterintuitive results.

The theory underlying the view that antioxidants are good for you is the free radical theory of aging. Free radicals are highly chemically reactive, as are many things in the body. Often this reactivity is put to very good use. For example, if you have an infection, and there are some harmful bacteria in your body, then a phagocytic cell from your immune system might come along, identify the bacteria as unwelcome, build a strong wall around as many of them as it can find, and blast them with destructive free radicals. Free radicals are basically like bleach, and this process is a lot like pouring bleach down the toilet. Once again, the human body is cleverer than anybody you know.