Topic 3.1 Notes – Introducing Statistics: Do the Data We Collected Tell the Truth?

When Data Tell the Truth

Statistics exists because we usually can’t measure an entire population, so we collect data from a sample and use it to draw conclusions.

But here’s the catch:
Those conclusions are only as good as the method of data collection.

Data are trustworthy when:

• The method relies on chance (randomness is built in).
• The sample is representative of the population.
• Bias is minimized.
• It makes sense to generalize results to the population.

If the method does not rely on chance, the conclusions are untrustworthy. Even a huge sample size cannot fix a biased method.

The question you should automatically ask on any quiz, FRQ, or MCQ is:

How were the data collected?

If that answer is weak, everything that follows is weak.

Random Sampling and Random Assignment

Chance-based methods are the reason statistical inference works at all. There are two different types, and mixing them up is one of the most common mistakes students make.

Random Sampling

Random sampling is about who gets chosen from the population.

• Every individual has a known chance of being selected.
• On average, it produces a representative sample.
• It allows us to generalize results to the population.

You should recognize these names (details come later in the unit):

• Simple random sample (SRS)
• Stratified sample
• Cluster sample
• Systematic sample
• Multistage sample

They all have one thing in common: chance is built into selection.

Think of random sampling as solving this problem:
“How do we make sure our sample looks like the population?”

Random Assignment

Random assignment is about who gets which treatment in an experiment.

• Participants are randomly placed into groups.
• Groups are similar except for the treatment.
• This allows cause-and-effect conclusions.

Random assignment solves a different problem:
“How do we know the treatment caused the difference?”

Here’s the comparison that must be crystal clear:

You can have:

• Random sampling but no random assignment
• Random assignment but no random sampling
• Both (ideal)
• Neither (weakest)

On FRQs, students often say “random” without specifying which one. The grader will not give credit for that.

Methods That Do Not Rely on Chance

If there is no randomness in how data are collected, bias is likely.

Convenience Sampling

Choosing individuals who are easiest to reach.
Example pattern: surveying students in one hallway during lunch.

Voluntary Response Sampling

People choose themselves to participate.
Common with online polls or call-in surveys.

Self-Selection Bias

The people who volunteer differ in important ways from those who don’t.
Often linked to voluntary response situations.

Undercoverage

Some groups in the population are left out entirely.
Example pattern: surveying only landline numbers in a population where many people only use cell phones.

All of these create bias, which means systematic favoritism toward certain outcomes.

Large sample size does not fix bias.
If you ask the wrong group 10,000 times, you just get a very precise wrong answer.

How to Evaluate Data Collection on the Exam

When you’re given a study, mentally walk through this:

1. Who is the population?
   Who are we trying to make a conclusion about?
2. How were subjects selected?
   Was a random method used?
3. Can we generalize?
   Only if random sampling was used.
4. Can we say cause and effect?
   Only if random assignment was used.

Your answers must connect method → bias (or lack of bias) → conclusion.

For example:

• If no random sampling:
  “Because the sample was not randomly selected, the results may be biased and cannot be generalized to the population.”
• If no random assignment:
  “Because treatments were not randomly assigned, a cause-and-effect conclusion cannot be made.”

Notice how the conclusion directly follows from the method.

Common Mistakes and Traps

• Believing a large sample guarantees accuracy
• Confusing random sampling with random assignment
• Ignoring how participants were recruited
• Assuming survey results automatically apply to everyone
• Overlooking voluntary response bias in online polls

This topic is the foundation for everything in inference. If the data collection doesn’t rely on chance, the rest of the statistics doesn’t matter.