Topic 1.12 Notes – Potential Problems with Sampling

What Sampling Bias Is

A population is the whole group you want to learn about. A sampling frame is the list or source you actually sample from. The respondents or observed units are the people selected who actually give data.

Bias shows up when the method itself has a built-in problem. This flowchart ties those vocabulary words to the main places bias can enter the process.

Where sampling bias can enter a study

• Bias means a systematic error. If you repeated the method many times, the statistic would tend to miss in the same direction.
• That is different from random sampling variability. Even a good random sample can be off by chance.
• A sample statistic estimates a population parameter, so bias means the statistic tends to be consistently too high or too low for that parameter.

A couple of traps show up all the time:

• Random selection helps with selection problems, but it does not fix a bad sampling frame, nonresponse, or bad answers.
• Small samples can be unbiased. They just have more variability.
• Large samples do not cure bias. A huge bad sample is still bad.

Main Sources of Bias in Sampling

Voluntary response bias

This happens when people choose themselves to be in the sample.

• Common examples include online polls, call-in polls, and public survey links.
• People with strong opinions often respond more.
• This is self-selection into the sample.

If a school posts “Rate the cafeteria!” on social media, students with very positive or very negative opinions are more likely to answer.

Undercoverage bias

This happens when part of the population is missing from the sampling frame or is less likely to be selected.

• You can still have undercoverage even if the selection is random from that frame.
• Examples include using only email, only phone numbers, or only one place/time.

If you survey district families using only school email accounts, families with poor internet access may be missed.

Nonresponse bias

This happens when people were chosen, but data were never obtained.

• They may refuse, not answer, or be unreachable.
• Bias appears if nonrespondents differ from respondents on the variable being studied.

If selected students who work late shifts are hard to reach, a survey about weekly work hours may underestimate the true mean.

Response bias

This is about bad data, not bad selection. The responses or measurements tend to differ from the truth.

Sources include:

• Question wording bias
  • Leading, loaded, ambiguous, double-negative, or double-barreled questions
• Self-reported responses
  • People may forget, guess, or give socially desirable answers
• Biased measurements
  • A faulty scale or instrument systematically reads too high or too low

Question wording is a classic source of response bias. A leading question can nudge people toward a particular answer, even when the neutral version is asking about the same issue.

Leading vs. neutral question wording

Convenience sampling

A convenience sample uses whoever is easiest to reach.

• The researcher picks the sample for ease.
• This is different from voluntary response, where individuals choose to participate.
• Easy-to-reach people may differ from the population.

How to Identify the Bias in a Scenario

Use this sequence in your head:

1. Ask whether the problem is selection or accuracy.
   • Selection problem: voluntary response, undercoverage, nonresponse, convenience sample
   • Accuracy problem: response bias
2. Ask where the breakdown happens.
   • Before selection and some group is missed = undercoverage
   • People opt in themselves = voluntary response
   • Selected first, then no data = nonresponse
   • Data collected but distorted = response bias
   • Chosen because easy to reach = convenience
3. Explain it in context.
   • Say which group is overrepresented, underrepresented, or inaccurate
   • Connect that to the variable being measured

Figuring Out the Direction of Bias

Naming the bias is only half the job. You also may need the direction.

Show these three links:

1. Who is missing, overrepresented, or giving distorted data
2. Whether that group tends to have larger or smaller values
3. How that pushes the statistic relative to the parameter

Example: surveying gym-goers about weekly exercise time

• gym-goers are overrepresented
• they likely exercise more than typical residents
• the sample mean is likely too high

If the context does not tell you how the missing group differs, say bias is possible but the direction is unknown.

On AP Stats, “the sample is biased” is too weak. Say the source, the group affected, and the effect on the estimate.

Common Mix-Ups and Fast Fixes

• Undercoverage vs nonresponse
  Undercoverage means they were missing from the frame or less likely to be selected. Nonresponse means they were selected but did not answer.
• Voluntary response vs nonresponse
  Voluntary response means people opt in. Nonresponse means they were picked first.
• Convenience sampling vs undercoverage
  Convenience is the method. Undercoverage is a likely result of that method.
• Random sampling vs random assignment
  Random sampling supports generalizing to a population. Random assignment supports cause-and-effect.
• Biggest trap: saying “it was random” as if that fixes everything. It doesn’t fix poor coverage, low response, or bad measurement.