Statistics for Dummies (30 page)
The margin of error is a measure of how close you expect your sample results to represent the entire population being studied. (Or at least it gives an upper limit for the amount of error you should have.) Because you're basing your conclusions about the population on your one sample, you have to account for how much those sample results could vary, just due to chance.
Another view of margin of error is that it represents the maximum
expected
distance between the sample results and the actual population results (if you'd been able to obtain them through a census). Of course if you had the absolute truth about the population, you wouldn't be trying to do a survey, would you?
Just as important as knowing what the margin of error measures is realizing what the margin of error does
not
measure. The margin of error does
not
measure anything other than chance variation. That is, it doesn't measure any bias or errors that happen during the selection of the participants, the preparation or conduct of the survey, the data collection and entry process, or the analysis of the data and the drawing of the final conclusions.
| HEADS UP | Bigger samples don't always mean better samples! A good slogan to remember when examining statistical results is "garbage in equals garbage out." No matter how nice and scientific the margin of error may look, remember that the formula that was used to calculate it doesn't have any idea of the quality of the data that the margin of error is based on. If the sample proportion or sample average was based on a |
The Gallup Organization addresses the issue of what margin of error does and doesn't measure in a disclaimer that it uses to report its survey results. The organization tells you that besides sampling error, surveys can have additional errors or bias due to question wording and some of the logistical issues involved in conducting surveys (such as missing data due to phone numbers that are no longer current). This means that even with the best of intentions and the most meticulous attention to details and process control, stuff happens. Nothing is ever perfect. But what you need to know is that the margin of error can't measure the extent of those other types of errors.
Guesstimating with Confidence
Chapter List
- Chapter 11:
The Business of Estimation—Interpreting and Evaluating Confidence Intervals - Chapter 12:
Calculating Accurate Confidence Intervals - Chapter 13:
Commonly Used Confidence Intervals—Formulas and Examples
Anytime someone gives you a statistic by itself, he or she hasn't really given you the full story. The statistic alone is missing the most important part: by how much that statistic is expected to vary. All good estimates contain not just a statistic but also a margin of error. This combination of a statistic plus or minus a margin of error is called a confidence interval. Confidence intervals go beyond a single statistic; instead, they offer important information about the accuracy of the estimate.
This part gives you a general, intuitive look at confidence intervals: their function, formulas, calculations, influential factors, and interpretation. You also get quick references and examples for the most commonly used confidence intervals.
The Business of Estimation—Interpreting and Evaluating Confidence Intervals
Most statistics are used to estimate some characteristic about a population of interest, such as average household income, the percentage of people who buy Christmas gifts online, or the average amount of ice cream consumed in the United States every year (maybe that statistic is better left unknown). Such characteristics of a population are called
parameters.
Typically, people want to estimate (take a good guess at) the value of a parameter by taking a sample from the population and using statistics from the sample that will give them a good estimate. The question is, how do you define "good estimate"?
The best guess would be no guess at all — go out and actually come up with the parameter itself. You can't determine the exact value of a parameter without conducting a census of the entire population — a daunting and expensive task in most cases. Statisticians, however, remain unfazed by the challenge and often say, "Being a statistician means never having to say you're certain; you only have to get close." Of course, statisticians want to be confident that their results are as close as they can be to the truth, within a certain time frame and budget, and "close" is easier to accomplish than you may think. As long as the process is done correctly (and in the media, it often isn't!), an estimate can get very close to the parameter. This chapter gives
you an overview of confidence intervals (the type of estimates used and recommended by statisticians), why they should be used (as opposed to just a one-number estimate), how to interpret a confidence interval, and how to spot misleading estimates.
Read any magazine or newspaper or listen to any newscast, and you hear a number of statistics, many of which are estimates of some quantity or another. You may wonder how they came up with those statistics: In some cases, the numbers are well researched; in other cases, they're just a shot in the dark. Here are some examples of estimates that I came across in one single issue of a leading business magazine. They come from a variety of sources:
26 million folks play golf at least once a year.
6.7 percent of U.S. homes were purchased without a down payment.
Even though some jobs are harder to get these days, some areas are really looking for recruits: Over the next eight years, 13,000 nurse anesthetists will be needed. Pay starts from $80,000 to $95,000.
The average number of bats used by a Major League baseball player per season is 90.
7.4 million U.S. residents took cruise vacations in 2002. Of those people, about 4 percent of them visited the ship's medical staff.
The Lamborghini Murcielago can go from 0 to 60 mph in 3.7 seconds with a top speed of near 205 mph.
Some of these estimates are easier to obtain than others. Here are some observations I was able to make about those estimates:
How do you know that 26 million people play golf once at least once a year? Actually, this one may not be too hard to get, because golfers must always sign in whenever they play at a golf course. So, a survey of golf course sign-in sheets could give a good estimate of how many people play at least once a year. (The only hard part would be not doublecounting people you've counted before on previous sign-in sheets.)
A survey may be able to estimate the percentage of cruisers who need medical attention or the percentage of homes purchased without a down payment. If the survey is done correctly (see
Chapter 16
), these data are probably pretty accurate.How do you estimate how many nurse anesthetists are needed over the next eight years? You can start by looking at how many will be retiring in that time; but that won't account for growth. A prediction of the need in the next year or two would be close, but eight years into the future is much harder to do.
The average number of bats used per Major League baseball player in a season could be found by surveying the players themselves, the people who take care of their equipment, or the bat companies that supply the bats.
Determining car speed is more difficult, but could be conducted as a test with a stopwatch. And they should use many different cars (not just one) of the same make and model.
| HEADS UP | Not all statistics are created equal. To determine whether a statistic is reliable and credible, don't just take it at face value. Think about whether it makes sense and how you would go about formulating an estimate. If the statistic is really important to you, find out what process was used to come up with the estimate. |
The U.S. Census Bureau estimates the
median
household income for the United States and breaks it down by each state in its yearly report from the Current Population Survey. Why estimate the median (middle number) and not the mean (average) household income? (See
Chapter 5
for more discussion on the mean versus the median.) Because household income tends to be skewed, with many folks at the lower end of the income spectrum, and fewer individuals at the extreme upper end.
To estimate the median household income, the Census Bureau takes a random sample of about 28,000 households and asks questions (household income is among those questions). Based on the sample data of 28,000 homes, the Bureau calculates the median household income for this sample: For the year 2000, the sample median household income was $42,228.
The Census Bureau uses the sample median household income (a statistic) to estimate the median household income for the whole U.S. (the parameter). Yet because the Bureau knows that a sample can't possibly reflect the entire population completely accurately, it includes a
margin of error
(see
Chapter 10
) with the results. This plus or minus (±) that's added to any estimate, helps put the results into perspective. When you know the margin of error, you have an idea of how much error may be in the estimate, due to the fact that it was based on a sample of the population and not on the entire population.
| TECHNICAL STUFF | Because the Bureau didn't sample the entire population and knows that the sample may not represent the entire population perfectly, the Bureau calculates a margin of error for the median of the sample group and includes that as part of the estimate. For the year 2000, the margin of error of the sample median household income was $258. The U.S. Census Bureau therefore estimates that in 2000, the median household income was $42,228, plus or minus $258 or $42,228 ± $258. This represents the confidence interval for the median household income of the United States (see the following section about confidence intervals). |
| HEADS UP | Note the margin of error is fairly small in the above example; that's because of the high sample size used (you get what you pay for!). See |
The best way to estimate a parameter (a characteristic of an entire population) is to come up with a statistic, plus or minus a margin of error, that's based on a large sample. In this way, your result presents an estimate based on your sample, along with some indicator of how much that estimate could vary from sample to sample.
A statistic plus or minus a margin of error is called a
confidence interval:
The word
interval
is used because your result becomes an interval. For example, say the percentage of kids who like baseball is 40 percent, plus or minus 3.5 percent. That means the percentage of kids who like baseball is somewhere between 40%
−
3.5% = 36.5% and 40% + 3.5% = 43.5%. Thus, the lower end of the interval is your statistic minus the margin of error, and the upper end is your statistic plus the margin of error.The word
confidence
is used because you have a certain amount of confidence in the process by which you got your interval. This is called your
confidence level.
You can find formulas and examples for the most commonly used confidence intervals in
Chapter 13
.
Suppose you, a research biologist, are trying to catch a fish using a hand net, and the size of your net represents the width of your confidence interval. (The width is twice the margin of error, due to adding and subtracting.)
Suppose your confidence level is 95%. What does this really mean? It means that if you scoop this particular net into the water over and over again, you'll catch a fish 95% of the time. Catching a fish here means your confidence interval was correct and contains the true parameter (in this case the parameter is represented by the fish itself).
But does this mean that on any given try you have a 95% chance of catching a fish? No. Is this confusing? It certainly is. Here's the scoop (no pun intended). On a single try, say you close your eyes before you scoop your net into the water. At this point, your chances of catching a fish are 95%. But then go ahead and scoop your net through the water with your eyes still closed. After that's done, however, you have only two possible outcomes; you either caught a fish or you didn't; probability isn't involved.
Likewise, after data have been collected and the confidence interval has been calculated, you either captured the true population parameter or you didn't. So you're not saying you're 95% confident that the parameter is in the interval, because you either captured it or you didn't. What you are 95% confident about is the process by which you're getting the data and forming the confidence interval in the long run. You know that this process will result in intervals that capture the mean 95% of the time. The other 5% of the time, the data collected in the sample just by chance had abnormally high or low values in it and didn't represent the population. In those cases, you won't capture the parameter.
So you know that with the size and composition of your net, you're going to catch a fish 95% of the time. On any given try, however, you either catch a fish or you don't.
| HEADS UP | Confidence level, sample size, and population variability all play a role in influencing the size of the margin of error and the width of a confidence interval. But the margin of error, and hence the width of a confidence interval, is meaningless if the data that went into the study were biased and/or unreliable. The best advice is to look at how the data were collected before accepting a reported margin of error as the truth (see |
