Statistics for Dummies (28 page)
Leaving Room for a Margin of Error
Good survey and experiment researchers always include some measure of how accurate their results are, so that consumers of the information they generate can put the results into perspective. This measure is called the
margin of error
— it's a measure of how close the sample result should be to the population parameter being studied.
Thankfully, many journalists are also realizing the importance of the margin of error in assessing information, so reports that include the margin of error are beginning to appear in the media. But what does the margin of error really mean, and does it tell the whole story?
This chapter looks at the margin of error and what it can and can't do to help you assess the accuracy of statistical information. It also examines the issue of sample size; you may be surprised at how small a sample can be used to get a good handle on the pulse of America — or the world — if the research is done correctly.
The margin of error is a term that you may have heard of before, most likely in the context of survey results. For example, you may have heard someone
report, "This survey had a margin of error of plus or minus three percentage points." You may have wondered what you're supposed to do with that information, and how important it really is. The truth is, the survey results themselves (with no margin of error) are only a measure of how the
sample
of selected individuals felt about the issue; they don't reflect how the
entire population
may have felt, had they all been asked (what a job that would be!). The margin of error helps you measure how close you are to the truth about the entire population being studied, while still using only the information gathered from a sample of that population.
As discussed in
Chapter 3
, a sample is a representative group taken from the population you're interested in studying. Results based on a sample won't be exactly the same as what you would've found for the entire population, because when you take a sample, you don't get information from everyone in the population. But if the study is done right (see
Chapter 17
for more about designing good studies), the results from the sample should be close to the actual values for the entire population.
| TECHNICAL STUFF | The margin of error doesn't mean someone made a mistake; all it means is that you didn't get to sample everybody in the population, so you expect your sample results to be "off" by a certain amount. In other words, you acknowledge that your results could change with subsequent samples, and are only accurate to within a certain range, which is the margin of error. |
Consider one example of the type of survey conducted by some of the leading polling organizations, such as The Gallup Organization. Suppose its latest poll sampled 1,000 people from the United States, and the results show that 520 people (52%) think the president is doing a good job, compared to 48% who don't think so. Suppose Gallup reports that this survey had a margin of error of plus or minus 3%. Now, you know that the majority of the people in this
sample
approve of the president, but can you say that the majority of
all Americans
approve of the president? In this case, you can't. Why not?
If 52% of
those sampled
approve of the president, you can expect that the percent of the
population of all Americans
who approve of the president will be 52%, plus or minus 3%. So you can say that somewhere between 49% and 55% of all Americans approve of the president. That's as close as you can get with your sample of 1,000. But notice that 49%, the lower end of this range, represents a minority, because it's less than 50%. So you really can't say that a majority of the American people support the president, based on this sample. You can only say that between 49% and 55% of all Americans support the president, which may or may not be a majority.
| HEADS UP | Think about the sample size for a moment. Isn't it interesting that a sample of only 1,000 Americans out of a population of more than 288,000,000 can lead you to be within plus or minus only 3% on your survey results? That's incredible! So for large populations, to get a really good idea of what's happening, you |
| Tip | For a quick-and-dirty way to get a rough idea of what the margin of error is for any given sample size, simply take the sample size |
The margin of error is the amount of "plus or minus" that is attached to your sample result when you move from discussing the sample itself to discussing the whole population that it represents. Therefore, you know that the general formula for the margin of error contains a "±" in front of it. So, how do you come up with that plus or minus amount (other than taking a rough estimate, as shown above)? This section shows you how.
Sample results vary, but by how much? According to the central limit theorem (see
Chapter 9
), when sample sizes are large enough, the distribution of the sample proportions (or the sample averages) will follow a bell-shaped curve (or normal distribution — see
Chapter 8
). Some of the sample proportions (or sample averages) will overestimate the population value and some will underestimate it, but most will be close to the middle. And what's the middle? If you averaged out the results from all of the possible samples you could take, the average would be the real population proportion, in the case of categorical data, or real the population average, in the case of numerical data. Normally, you don't have the time or the money to look at all of the possible sample results and average them out, but knowing something about all of the other sample possibilities does help you to measure the amount by which you expect your one sample proportion (or average) to vary.
Standard errors are the basic building block of the margin of error. The
standard error
of a statistic is basically equal to the standard deviation of the data divided by the square root of
n
(the sample size). This reflects the fact that
the sample size greatly affects how much that sample statistic is going to vary from sample to sample. (See
Chapter 9
for more about standard errors.)
The number of standard errors you wish to add or subtract to get the margin of error depends on how confident you wish to be in your results (this is called your
confidence level
). Typically, you want to be about 95% confident, so the basic rule is to add or subtract about 2 standard errors (1.96, to be exact) to get the margin of error. This allows you to account for about 95% of all possible results that may have occurred with repeated sampling. To be 99% confident, you add and subtract about 3 standard errors (2.58, to be exact). (See
Chapter 12
for more discussion on confidence levels and number of standard errors.)
| TECHNICAL STUFF | To be |
 |
Percentage Confidence | Z-Value |
|---|---|
80 | 1.28 |
90 | 1.64 |
95 | 1.96 |
98 | 2.33 |
99 | 2.58 |
|
When the polling question asks people to choose from a range of answers (for example, "Do you approve, disapprove, or have no opinion about the president's performance?"), the statistic used to report the results is the
proportion of people from the sample who fell into each group, otherwise known as the
sample proportion
, or
sample percentage
. The general formula for margin of error for your sample proportion is
, where
is the sample proportion,
n
is the sample size, and Z is the appropriate Z-value for your desired level of confidence (from
Table 10-1
).
Here are the steps for calculating the margin of error for a sample percentage:
Find the sample proportion,
, and the sample size,
n
.Multiply
× (1 -
).Divide the result by
n.Take the square root of the calculated value.
You now have the standard error.
Multiply the result by the appropriate Z-value for the confidence level desired.
Refer to
Table 10-1
. The Z-value is 1.96 if you want to be about 95% confident in your results.
Looking at an example involving whether Americans approve of the president, you can find the margin of error. First, assume you want a 95% level of confidence, so Z = 1.96. The number of Americans in the sample who said they approve of the president was found to be 520. This means that the sample proportion,
is 520 ÷ 1, 000 = 0.52. (The sample size,
n
, was 1, 000.) The margin of error for this polling question is calculated in the following way:
| HEADS UP | A sample proportion is the decimal version of the sample percentage. In other words, if you have a sample percentage of 5%, you must use 0.05 in the formula, not 5. To change a percentage into decimal form, simply divide by 100. After all your calculations are finished, you can change back to a percentage by multiplying your final answer by 100%. |
