How We Know What Isn't So (6 page)

Many of the beliefs we hold are about
relationships
between two variables. A belief that our dreams are prophetic is really a belief about the relationship between dream content and life events. A belief that increased military spending by the U.S. was partly responsible for the recent changes in Eastern Europe is really a belief about the linkage between U.S. defense appropriations and Soviet foreign and domestic policy. Indeed, a belief in streak shooting or the hot hand (see Chapter 2) is really a belief about the relationship between the outcomes of successive shots.

Most of these relationships, and the evidence necessary to assess their validity, can be represented in the 2x2 table familiar to most social scientists. Consider once again the common belief that infertile couples who adopt a child are subsequently more likely to conceive than those who do not. The evidence relevant to this belief can be represented in the layout at the top of page 31.

In this layout, “a” represents the number of couples who adopt and then conceive, “b” represents the number who adopt and do not conceive, etc. To adequately assess whether adoption leads to conception, it is necessary to compare the probability of conception after adopting a/(a+b), with the probability of conception after not adopting, c/(c+d). There is now a large literature on how well people evaluate this kind of information in assessing the presence or strength of such relationships.
²
According to this research, although people sometimes perform such “covariation” tasks with considerable accuracy, there are as many or more occasions in which they perform poorly. A major culprit in people’s poor performance seems to be an over-reliance on instances that confirm the existence of a relationship—cells “a” and “d.” In fact, many judgments seem to be influenced almost exclusively by the information contained in cell “a.” In the example above, people are most influenced by the number (and salience) of couples who adopt and subsequently conceive. In so doing, people implicitly confuse necessary and sufficient evidence: They seem to be reasoning that if there are a fair number of such positive cases, then the phenomenon must exist, or the relationship must be valid.

In one of the most direct demonstrations of this phenomenon, two groups of people were asked different versions of the same question. One group was asked to assess whether practicing the day before a tennis match is related to winning the match, and a second group was asked to assess whether practicing the day before the match is related to
losing
. The participants were asked to indicate what information, from cells a, b, c, & d above, they thought was necessary to adequately assess whether such a relationship existed. The results were quite revealing: Those testing whether practice leads to winning emphasized the number of times players practiced and won; those testing whether practicing leads to
losing
emphasized the number of times players practiced and lost.
³

The most likely reason for the excessive influence of confirmatory information is that it is easier to deal with cognitively. Consider someone trying to determine whether cloud seeding produces rain. An instance in which cloud seeding is followed by rain is clearly relevant to the issue in question—it registers as an unambiguous success for cloud seeding. In contrast, an instance in which it rains in the absence of cloud seeding is only indirectly relevant—it is neither a success nor a failure. Rather, it represents a consequence of
not
seeding that serves only as part of a baseline against which the effectiveness of seeding can be evaluated. Additional cognitive steps are necessary to put this information to use.

Non-confirmatory information can also be harder to deal with because it is usually framed negatively (e.g., it rained when we did
not
seed), and we sometimes have trouble conceptualizing negative assertions. Compare, for example, how much easier it is to comprehend the statement “All Greeks are mortals” than “All non-mortals are non-Greeks.” Thus, one would expect confirmatory information to be particularly influential whenever the disconfirmations are framed as negations. The research literature strongly supports this prediction. People are particularly swayed by the information in “cell a” of the 2×2 table discussed above when the two variables in question are “asymmetric.” Asymmetric variables are those in which one level of the variable is simply the absence of the other, such as whether it rains or not, or whether a couple has adopted or not. Symmetric variables, on the other hand, are those in which both levels are defined by the
presence
of some attribute or set of attributes, like whether a person is male or female, or whether a university is publicly or privately funded. The influence of confirmatory information is particularly strong when both variables are asymmetric because in such cases three of the four cells contain information about the
nonoccurrence
of one of the variables, and, once again, such negative or null instances have been shown to be particularly difficult to process.
⁴
As Francis Bacon noted long ago, “It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than negatives.”
⁵

With respect to the formation of erroneous beliefs, the implications of people’s difficulties in detecting covariation should be clear. By placing too much emphasis on positive instances, people will occasionally “detect” relationships that are not there. For many of the real-world phenomena that are of greatest interest, one is sure to encounter many positive instances even when there is no relationship at all between the two variables. Although there is surely no validity to the common belief that we are more likely to need something once we have thrown it away, examples of acute longing for a discarded possession may be easy to come by. By letting necessary evidence “slip by” as sufficient evidence, people establish an insufficient threshold of what constitutes adequate support for a belief, and they run the risk of believing things that are not true.

The Tendency to
Seek
Confirmatory Information
. People exhibit a parallel tendency to focus on positive or confirming instances when they
gather
, rather than simply evaluate, information relevant to a given belief or hypothesis. When trying to assess whether a belief is valid, people tend to seek out information that would potentially confirm the belief, over information that might disconfirm it. In other words, people ask questions or seek information for which the equivalent of a “yes” response would lend credence to their hypothesis. To illustrate this tendency, consider an experiment in which participants were given a set of four cards, each of which has a letter or number on the side facing up—A, B, 2, and 3. The participants were told that each card had a letter on one side and a number on the other, and they were asked to determine, by judiciously turning over the proper cards, whether “all cards with a vowel on one side have an even number on the other.” (The reader is encouraged to take a moment to consider which cards should be turned over.)

A common response was to turn over the “A” and “2” cards. These cards were presumably chosen because of their potential to provide evidence consistent with the hypothesis. However, turning over the “2” card was uninformative because it could
only
confirm the hypothesis (a vowel on the other side would confirm it and a consonant would be irrelevant to it). The “3” card was rarely turned over, on the other hand, even though it was potentially at least as informative as any other because of its potential to invalidate the hypothesis in one quick step (a vowel on the other side guarantees that not all cards with vowels on one side have an even number on the other).
⁶

This experiment is particularly informative because it makes it abundantly clear that the tendency to seek out information consistent with a hypothesis need not stem from any
desire
for the hypothesis to be true. The people in this experiment surely did not care whether all cards with vowels on one side had even numbers on the other; they sought information consistent with the hypothesis simply because it seemed to them to be the most relevant to the issue at hand.
*

The intuition that positive instances are somehow more informative than disconfirmations can also be seen in the quotation by Holt that began this chapter. In that example, elementary school students who had 20 questions to identify an unknown number between 1 and 10,000 cheer when the teacher tells them “yes, it is between 5,000 and 10,000,” but groan when he says “no, it is not between 5,000 and 10,000,” even though the latter response is just as informative as the former. Their difficulty in recognizing the relevance of the latter response is no doubt due to the extra cognitive step that is required to put it to use—a statement that the number is not between 5,000 and 10,000 must be converted to a mental representation that it is between 1 and 5,000.

A number of investigators have examined the extent to which this tendency to seek out confirmatory information governs people’s hypothesis-testing strategies in everyday social life.
⁷
In the most common procedure used in these experiments, participants are asked to determine whether a target person possesses a certain trait (e.g., extroversion) by selecting a set of questions to ask the target from a list of questions provided by the experimenter. Much of this research, as we might expect, indicates that people sometimes perform quite well at this task. They seem sensitive to which questions discriminate most effectively between, say, introverts and extroverts, and they often prefer to ask questions that are the most discriminative.
⁸

Nevertheless, it is also clear that people sometimes perform this task rather poorly by being too inclined to ask questions for which a positive response would confirm the hypothesis.
⁹
When trying to determine if a person is an extrovert, for example, people prefer to ask about the ways in which the target person is outgoing; when trying to determine if a person is an introvert, people are more inclined to ask about the ways in which the target is socially inert.

Although a tendency to ask such one-sided questions does not guarantee that the hypothesis will be confirmed, it can produce an erroneous sense of confirmation for a couple of reasons. First, the specific questions asked can sometimes be so constraining that
only
information consistent with the hypothesis is likely to be elicited. For example, in one widely-cited study,
¹⁰
one of the questions that the participants were fond of asking when trying to determine if a person was an extrovert was: “What would you do if you wanted to liven things up at a party?” A question such as this one is clearly biased against disconfirmation: Even the most inner-directed individual has been to a party or two and can at least
discuss
how to liven one up if explicitly asked to do so. By asking such constraining questions, it is difficult for anyone, including introverts, not to sound extroverted. In fact, the experimenters in this study tape-recorded the responses of the target individuals who were asked the questions selected by the “interviewer” subjects. These tapes were then played for a group of judges. The targets who were asked questions by interviewers who were testing whether they were extroverted impressed the judges as being more extroverted than those who were asked questions by interviewers who were testing whether they were introverted. In other words, the participants tended to ask questions that produced a spurious confirmation of their initial hypotheses.

Furthermore, even if such constraining questions are not asked, a tendency to ask confirmatory questions can still produce a spurious sense of confirmation if the likelihood of a positive response to the question is high whether or not the hypothesis is true. Suppose, for example, that you want to determine if an individual is introverted, and so you ask about a characteristic that might confirm your hypothesis: “Do you sometimes feel that it is hard for you to really let yourself go at a party?” The person’s response is unlikely to be truly informative because most people, extroverts as well as introverts, would answer the same way—yes,
sometimes
it is hard to
really
let go. For the sake of illustration, suppose that 50% of the people in the world are introverts and 50% are extroverts. Suppose also that 90% of the introverts would say that it is hard for them to really let themselves go, and that 70% of the
extroverts
would also say so. Under these conditions, the question asked is indeed diagnostic of introversion (90% of the introverts would respond affirmatively as opposed to 70% of the extroverts), and it is hardly constraining (one can easily respond by saying, “No, I don’t find it hard to let myself go.”). Nevertheless, because an affirmative response is likely whether the hypothesis is true (90%) or false (70%), one is likely to conclude too often that the person is introverted. In this case, one would do so 80% of the time—(90% + 70%)/2—when the actual likelihood that a person is introverted given a positive response to this question is 56%—(90%/2)/[(90% + 70%)/2].

A similar tendency to seek out hypothesis-confirming evidence seems to exist when people search their own memories for relevant evidence, rather than asking questions of another person. In one study, participants read a story about a woman who behaved in a number of prototypically introverted and extroverted ways.
¹¹
Two days later, half of the participants were asked to assess the woman’s suitability for a job in real estate sales (a job thought to demand considerable extroversion) and the other half were asked to assess her fitness for a job as a librarian (a job thought to demand introversion). As part of their assessment, the participants were asked to recall examples of the woman’s introversion and extroversion. The particular job the woman was seeking strongly affected the evidence that the participants could recall: Those asked to assess the woman’s suitability for an extroverted job recalled more examples of the woman’s extroversion; those asked to assess her suitability for an introverted job recalled more examples of her introversion.