The Numbers Behind NUMB3RS (24 page)

 

Comparing (*) and (**), the obvious question is “Which method intercepts a higher percentage of terrorists?” The answer depends on the value of
p
, the unknown percentage of terrorists who are selected because they meet the profile. Here are some examples:

From these examples it is clear that the break-even point for CAPPS versus a purely random system is when
p
= 6 percent of terrorists are subjected to secondary screening because they meet the profile.

Now comes the heart of the matter. You might say, “Surely we can expect the percentage of terrorists who meet the profile to be larger than a paltry 6 percent!” That is where the phenomenon Chakrabarti and Strauss call the “carnival booth effect” comes in. They argue that, since the terrorist profile is fixed, and terrorist cells have members with a diversity of characteristics, a cell that wants to be successful in getting one of its members aboard a plane for an attack can use the following strategy:

• Probe the CAPPS system by sending cell members on “dry runs” to see which ones are flagged by the profile and which are not.
  
• For the actual attack mission, use members who were not flagged in the dry runs and are therefore very unlikely to be flagged by the same profile next time.
  

Chakrabarti and Strauss call this carnival booth effect because it is reminiscent of the barkers at carnival booths who call out “Step right up and see if you're a winner!” The would-be attackers who constitute a real threat are the “winners” who do not trigger secondary screening when they “step right up” to the CAPPS profiling system.

As the MIT authors explain at some length, the viability of this strategy depends on just two essential factors: the observation that the CAPPS profile itself is fixed over time—at least over short time intervals—which implies the “repeatability” of an individual's not being selected by the profile; and the recognition that terrorist cells have members with considerable diversity of characteristics, and so are likely to include at least one member who can pass the profile part of the screening system. In support of the latter claim, they describe some of the known terrorists from recent events, such as John Walker Lindh, the “American Taliban,” a nineteen-year-old from Marin County, and Richard Reid, the British citizen with an English mother and Jamaican father who single-handedly made sure that we now all have to take off our shoes before boarding an airplane.

The two MIT researchers included in their paper some more sophisticated analyses using computer simulations incorporating some variability and uncertainty in the CAPPS profile scores of each individual terrorist. For instance, they found that repeated probes would, for some individual terrorists, increase the confidence of not being flagged to a higher level than that of a randomly chosen passenger. In that case, the CAPPS probability of intercepting an actual attack by such an individual would be worse than random.

Such is the power of mathematics, that even a couple of bright college students writing a term paper can make a significant contribution to an issue as significant as airline security.

Okay, so Charlie has pulled out all the mathematical stops and as a result Don has once again nailed his suspect. That is generally the end of a
NUMB3RS
episode, but in real life it is often not the end of the mathematics. Math is used not only in crime detection, but in the courtroom as well.

One example is the use of mathematically enhanced photographs, as in the Reginald Denny beating case described in Chapter 5; another is the probability calculations that must accompany the submission of DNA profile evidence, which we looked at in Chapter 7. But there are many other occasions when lawyers, judges, and juries must weigh mathematical evidence. As our first example shows, if they get the math wrong, the result can be a dramatic miscarriage of justice.

THE BLONDE WITH THE PONYTAIL

Just before noon on June 18, 1964, in the San Pedro area of Los Angeles, an elderly woman named Juanita Brooks was walking home from grocery shopping. Using a cane, she was pulling a wicker basket containing her groceries, with her purse on top. As she made her way down an alley, she stooped to pick up an empty carton, and suddenly she felt herself being pushed to the ground. Stunned by the fall, she still managed to look up, and saw a young woman with a blond ponytail running away down the alley with her purse.

Near the end of the alley, a man named John Bass was watering the grass in front of his house when he heard crying and screaming. He looked over toward the alley and saw a woman run out of it and get into a yellow car across the street. The car started up, turned around, and took off, passing within six feet of him. Bass subsequently described the driver as a “male Negro” (this was 1964) wearing a beard and a mustache. He described the young woman as Caucasian, slightly over five feet tall, with her dark blonde hair in a ponytail.

Brooks reported the robbery to Los Angeles police, telling them her purse had contained between $35 and $40. Several days later, they arrested Janet Louise Collins and her husband Malcolm Ricardo Collins, who were ultimately charged with the crime and placed on trial in front of a jury.

The prosecutor faced an interesting challenge. Neither eyewitness, Brooks or Bass, could make a positive identification of either of the defendants. (Bass had previously failed to identify Malcolm Collins in a lineup, where he appeared without the beard he admitted he had worn in the past—but not on the day of the robbery, he said.) There was some doubt caused by the witnesses' description of the ponytailed blonde's clothing as “dark,” since the police had obtained testimony from people who had seen Janet Collins shortly before the robbery wearing light-colored clothing. How was the prosecutor to make the case to the jury that these two defendants were guilty of the purse snatching?

The prosecutor took a novel approach. He called an expert witness: a mathematics instructor at a state college. The expert testimony concerned probabilities and how to combine them. Specifically, the mathematician was asked to explain the product rule for determining the probability of the joint occurrence of a combination of events based on the individual probabilities of those events.

The prosecutor asked the mathematician to consider six features pertaining to the two perpetrators of the robbery:

Next, the prosecutor gave the mathematician some numbers to assume as the probabilities that a randomly selected (innocent) couple would satisfy each of those descriptive elements. For example, he instructed the mathematician to assume that the male partner in a couple is a “black man with a beard” in one out of ten cases, and that the probability of a man having a mustache (in 1964) is one out of four. He then asked the expert to explain how to calculate the probability that the male partner in a couple meets
both
requirements—“black man with a beard” and “man with a mustache.” The mathematician described a procedure well known to mathematicians, called the “product rule for independent events.” This says that “if two events are independent, then the probability that both events occur together is obtained by multiplying their individual probabilities.”

Thus, in the hypothetical case proposed by the prosecutor, if the events are indeed independent (we'll discuss later exactly what that means), then you can use the product rule to calculate the probability that an individual is a black man with a beard and has a mustache by multiplying the two given probabilities:

The complete list of probabilities the prosecutor asked the mathematician to assume was:

The prosecutor asked the mathematician to take those numbers as conservative assumptions, meaning that the actual probabilities would be at least this small and possibly smaller.

The mathematician then proceeded to explain how to combine these probabilities to come up with an overall probability that a random couple would satisfy all of the above description. Assuming independent events (more later), the mathematician testified that the correct calculation of the overall probability, let's call it PO, uses the same product rule, which means you multiply the individual probabilities to get the probability that the whole list applies to a random couple. When you do this, here is what you get:

One out of 12 million!

When the prosecutor gave the various odds—1 in 10, etc.—to the mathematics expert to use in calculating the overall probability, he stated that these particular numbers were only “illustrative.” But in his closing argument he asserted that they were “conservative estimates,” and therefore “the chances of anyone else besides these defendants being there,…having every similarity…, is something like one in a billion.”

The jury found Malcolm and Janet Collins guilty as charged. But did they make the right decision? Was the mathematician's calculation correct? Was the prosecutor's closing “one in a billion” claim correct? Or had the court just been party to a huge travesty of justice? Malcolm Collins said it was the latter, and appealed his conviction.

In 1968 the Supreme Court of the State of California handed down a decision in
People v. Collins, 68 Cal.2d 319
, and their written opinion has become a classic in the study of legal evidence. Generations of law students have studied the case as an example of the use of mathematics in the courtroom.

Here is what the California Supreme Court's opinion (affirmed by a six-to-one vote of the justices) said:

The majority opinion in the Collins case is a fascinating example of the interplay between two scholarly disciplines: law and mathematics. Indeed, the majority opinion took pains to say that they found “no inherent incompatibility between the [two] disciplines” and that they intended “no disparagement” of mathematics as “an auxiliary in the fact-finding process” of the law. Nevertheless, the court ruled that they could not uphold the way mathematics was used in the Collins case.

The Supreme Court's devastating deconstruction of the prosecution's “trial by mathematics” had three major elements:

• Proper use of “math as evidence” versus improper use (“math as sorcery”)
  
• Failure to prove that the mathematical argument used actually applies to the case at hand
  
• A major logical fallacy in the prosecutor's “one in a billion” claim about the chances of the defendants being innocent
  

Let's see just what went wrong with the prosecution's case.

MATHEMATICS: EVIDENCE OR SORCERY?

The law recognizes two principal ways in which an expert's testimony can provide admissible evidence. An expert can testify as to their own knowledge of relevant facts, or they can respond to hypothetical questions based on valid data that has already been presented in evidence. So, for example, an expert could testify about the percentages—in Los Angeles, say—of cars that are yellow, or of women who are blondes, provided there exists statistical data to support that testimony. And a mathematician can respond to hypothetical questions such as “How would you combine these probabilities to determine an overall probability?”—provided those hypotheticals are based on valid data. In the Collins case, however, the Supreme Court found that, the prosecution “made no attempt to offer any such evidence” of valid probabilities.

Moreover, the court pointed out that the prosecution's mathematical argument rested on the assumption that the witnesses' descriptions were 100 percent correct in all particulars and that no disguises (such as a false beard) were employed by the true perpetrators of the crime. (The trial record contained disputes about light versus dark clothing worn by the young woman, and about whether or not the defendant had a beard.)

The court pointed out that it is traditionally the function of juries to weigh the reliability of witness descriptions, the possibility of disguise by the perpetrators, and the like. But these considerations are not ones that can be assigned numerical probabilities or likelihoods. Moreover, the Supreme Court believed that the appeal of the “mathematical conclusion” of odds of 1 in 12 million was likely to be too dazzling in its apparent “scientific accuracy” to be discounted appropriately in the usual weighing of the reliability of the evidence. The court wrote: “Confronted with an equation which purports to yield a numerical index of probable guilt, few juries could resist the temptation to accord disproportionate weight to that index….” That is at the heart of the “sorcery” that the Supreme Court found in the prosecution's case.

WAS THE COURT'S MATH CORRECT?

Leaving aside the question of whether it was permissible to use mathematics in the way the original court allowed, there is the issue of whether the math itself was correct. Even if the prosecution's choice of numbers for the probabilities of individual features—black man with a beard, and so on—were supported by actual evidence and were 100 percent accurate, the calculation that the prosecutor asked the mathematician to do depends on a crucial assumption: that in the general population these features occur
independently
. If this assumption is true, then it is mathematically valid and sensible to use the product rule to calculate the probability that the couple who committed the crime, if they were
not
Malcolm and Janet Collins, would by sheer chance happen to match the Collinses in each of these factors.

That crucial assumption of independence means that if we think of the individual probabilities as representing fractions of the general population, then those fractions continue to apply in sequence as we look at each fraction in turn. Let's consider an example that is similar and slightly easier to work with. Suppose that witnesses to a crime said that the perpetrator drove a black Honda Civic that was “lowered”—fitted with special springs that make the body sit closer to the ground.

Ignoring the likely possibility that witnesses might also identify other features of the perpetrator, let's assume that we know, accurately and based on solid data, that in the Los Angeles area 1 out of 150 cars is a black Honda Civic and 1 out of 200 is lowered. The product rule says that to determine the fraction of cars that are black Honda Civics that have been lowered, we multiply:

 

1/150 × 1/200 = 1/30,000.

 

But this calculation is based on the assumption that the fraction of cars that have been lowered is the same for black Honda Civics as it is for all other makes and colors. If that were true, we could say that the descriptive features “black Honda Civic” and “lowered” occur independently. There is, however, the possibility that owners of black Honda Civics are more likely than owners of most other cars to have them customized by lowering. The correct calculation of the probability that a car in L.A. is a black Honda Civic that's been lowered (assuming we have good data to determine these numbers) is as follows.

 

Suppose that

 

fraction of cars that are black Honda Civics = 1/150

 

fraction of black Honda Civics that are lowered = 1/8

 

Then, the fraction of cars that are “lowered” black Honda Civics is