Topic 7.1 Notes – Introducing Statistics: Why Should I Worry About Error?

Random Variation and Uncertainty

Every time you take a different random sample from the same population, you get slightly different results. That natural difference is random variation.

Imagine repeatedly sampling 40 students from your school and recording the average number of hours they sleep. The averages would bounce around a bit. That bounce is normal. It does not mean you did anything wrong.

Here’s the key idea:

• Even with a perfectly random sample and correct calculations, you can still reach the wrong conclusion.
• Statistical inference manages risk. It never eliminates it.

This is what leads to the idea of errors in hypothesis testing.

Type I and Type II Errors

In a hypothesis test, there are only two decisions:

• Reject $H_0$
• Fail to reject $H_0$

But in reality, the null hypothesis is either true or false. That creates four possibilities.

Here’s the full picture:

Type I Error

A Type I error happens when you reject a true null hypothesis.

You conclude there is convincing evidence of a difference or effect when none actually exists.

The probability of a Type I error is:

$$\alpha$$

The significance level you choose, often 0.05, represents the long-run probability of rejecting a true null.

If $\alpha = 0.05$, that means:

If the null hypothesis is actually true and we repeated this test many times, about 5% of the time we would incorrectly reject it.

On quizzes and FRQs, you must describe this in context. For example:

A Type I error would be concluding that the new tutoring program increases test scores when in reality it does not.

Type II Error

A Type II error happens when you fail to reject a false null hypothesis.

There really is an effect, but your test does not detect it.

The probability of a Type II error is:

$$\beta$$

Related to this is power:

$$\text{Power} = 1 - \beta$$

Power is the probability of correctly rejecting a false null. High power means your test is good at detecting real differences.

In context, this sounds like:

A Type II error would be failing to detect that the tutoring program improves scores when it actually does.

Students mix these up constantly. A helpful memory trick:

• Type I = saying something is happening when it is not.
• Type II = missing something that is happening.

Tradeoffs Between Errors

You cannot shrink both types of error just by adjusting settings. There is a tradeoff.

Changing $\alpha$

• Increase $\alpha$
  • Easier to reject $H_0$
  • More Type I error
  • Less Type II error
  • Higher power
• Decrease $\alpha$
  • Harder to reject $H_0$
  • Less Type I error
  • More Type II error
  • Lower power

If a situation makes Type I errors very serious, you lower $\alpha$. That decision increases the chance of Type II error.

That question shows up a lot. They describe a scenario and ask which error is worse.

Increasing Sample Size

Increasing $n$:

• Decreases standard error
• Makes real effects easier to detect
• Decreases $\beta$
• Increases power
• Does not change $\alpha$

If you want fewer Type II errors without increasing Type I errors, increasing sample size is the clean solution.

Errors From More Than Randomness

Not all mistakes come from hypothesis testing decisions.

Sampling Error

This is the natural difference between a sample statistic and the true population parameter due to random variation.

• Happens even with random sampling
• Gets smaller with larger samples
• Is unavoidable

Bias

Bias is systematic error. It pushes results consistently in one direction.

Common sources:

• Undercoverage (part of the population left out)
• Voluntary response or convenience samples
• Response bias (leading questions, lack of anonymity)
• Confounding variables in experiments

Increasing sample size does not fix bias. A large biased sample is still biased.