Topic 3.4 Notes – Potential Problems with Sampling

1. What Bias in Sampling Is

The definition you need locked in:

Bias occurs when certain responses are systematically favored over others.

“Systematically” means the method consistently pushes results in one direction. It’s not random error. It’s built into the process.

If a sample is biased, it is not representative of the population. That means conclusions about the population are unreliable.

Here’s the part students forget:

• A larger sample size does not fix bias.
• A huge biased sample just gives you a very precise wrong answer.

On tests, always pause and ask:

• Who is missing?
• Who is more likely to respond?
• Was chance used to select people?

The way data are collected determines what we’re allowed to conclude. If the collection method is flawed, the conclusion is flawed.

2. Major Sources of Bias in Sampling

Let’s walk through the main types you’re responsible for.

Voluntary Response Bias (Self-Selection)

This happens when the sample is made up entirely of people who choose themselves to participate.

Examples:

• Online polls
• Call-in radio surveys
• QR code surveys posted publicly

People with strong opinions are more likely to respond.

Why it’s biased:
Those who care deeply are overrepresented. People who don’t care stay silent.

If you see “vote on our website” or “call in now,” that’s a red flag for voluntary response bias.

Undercoverage Bias

This happens when part of the population has little or no chance of being selected.

Usually this is because the sampling frame leaves someone out.

Examples:

• Surveying registered voters to study all adults.
• Emailing a survey to company employees but excluding remote contractors.

Why it’s biased:
The missing group may respond differently than the included group.

Key idea: Some group is systematically left out.

Nonresponse Bias

Here, individuals were selected for the sample, but data were not obtained from some of them.

Important distinction:

• They were chosen.
• They didn’t respond.

The problem is not just a low response rate. The issue is that nonresponders differ from responders.

Example:
You randomly select 800 students to survey about sleep habits. Only 300 respond. If exhausted students are too tired to fill it out, your results may overestimate average sleep.

Why it’s biased:
The final responding group no longer represents the originally selected sample.

Students mix this up with voluntary response a lot. Remember:

• Voluntary → people choose themselves.
• Nonresponse → people were chosen but didn’t respond.

Response Bias (Problems with the Question or Process)

This bias comes from how the data are collected, not who was selected.

Two big causes:

Question Wording Bias

• Leading questions (“Don’t you agree…?”)
• Loaded language
• Confusing wording

Small wording changes can shift results.

Self-Reported Responses

People may:

• Lie
• Misremember
• Give socially desirable answers

If a survey asks about cheating, income, or illegal behavior, response bias is possible.

The key clue is that something about the question or setting influences answers.

Convenience Sampling

This is when the sample is chosen because it’s easy.

Examples:

• Surveying your own class.
• Asking shoppers at one grocery store.

There is no random selection.

Why this matters:
Without randomness, some people have zero chance of being selected. That opens the door to bias.

3. Random vs Non-Random Sampling

Random sampling

• Uses chance.
• Reduces bias.
• Allows generalization to the population (if other conditions are met).

Non-random sampling

• Includes voluntary response and convenience samples.
• Introduces potential bias.
• Does not justify generalizing results to the entire population.

On AP questions, if randomness is not mentioned, assume bias is possible.

4. How to Identify Bias on the Exam

When you’re given a scenario, mentally run this checklist:

1. Who is the population?
2. How were individuals selected?
3. Is any group left out?
4. Who might ignore the survey?
5. Could wording influence responses?

Then do two things in your answer:

• Name the specific bias.
• Explain how it affects results in context.

Example phrasing:
“This could lead to an overestimate of average weekly exercise among all students at the school because students who exercise frequently may be more likely to respond.”

You must say who is favored and how results shift.

5. Common AP Mistakes

• Saying “the sample is too small” when the real issue is bias.
• Confusing voluntary response with nonresponse.
• Naming the bias but not explaining its effect.
• Claiming results “prove” something when the sample was non-random.

Bias means systematic favoritism in responses → nonrepresentative sample → misleading conclusions.