Topic 3.8 Notes – Potential Errors When Performing Tests

What Type I Error, Type II Error, and Power Are

In a test, $H_0$ is the null claim and $H_a$ is the alternative claim you’re looking for evidence for. Your sample leads to a decision, but reality is separate from that decision.

That’s exactly what this 2-by-2 decision grid is showing:

Hypothesis testing outcomes

• Type I error means you reject $H_0$ when $H_0$ is actually true.
  • You say there is convincing evidence for $H_a$, but that claim is false.
• Type II error means you fail to reject $H_0$ when $H_0$ is actually false.
  • You miss a real effect or difference.
• Power is the probability that the test correctly rejects a false null hypothesis.

One wording issue matters a lot on tests. Reject $H_0$ means “the data give convincing evidence for $H_a$.” Fail to reject $H_0$ means “the data do not give convincing evidence for $H_a$.” It does not mean you proved $H_0$.

Probabilities Attached to the Errors

The probability of a Type I error is the significance level, $\alpha$.

$$P(\text{Type I error} \mid H_0 \text{ true})=\alpha$$

Common choices are $0.01$, $0.05$, and $0.10$. This is chosen before collecting data.

The probability of a Type II error is $\beta$. Unlike $\alpha$, $\beta$ usually depends on which true value under $H_a$ you mean. Missing a tiny real difference is more likely than missing a huge one.

The connection to power is

$$\text{Power}=1-\beta$$

So if power is $0.80$, then $\beta=0.20$. If $\beta=0.12$, then power is $0.88$.

Common trap:

• $\alpha$ is not the probability that $H_0$ is true.
• Power is not the probability that $H_a$ is true.

How to Identify and Write Each Error in Context

This is where AP Stats loves to be picky. You need the error in words tied to the situation.

Suppose $H_0: p=0.30$ and $H_a: p>0.30$, where $p$ is the true proportion of students at a school who buy lunch.

1. Translate the hypotheses.
   • $H_0$ says 30% buy lunch.
   • $H_a$ says more than 30% buy lunch.
2. For a Type I error
   • Decision made: conclude there is convincing evidence that more than 30% buy lunch.
   • Truth: the true proportion is actually 30%.
3. For a Type II error
   • Decision made: conclude there is not convincing evidence that more than 30% buy lunch.
   • Truth: the true proportion is actually greater than 30%.

Include these pieces every time:

• the population
• the parameter/variable
• the actual claim or value
• whether evidence was found or not found

Common mistakes:

• saying “accept $H_0$”
• giving only a consequence like “the school orders too much food”
• mixing up the test decision with what is actually true

What Changes $\beta$ and Power

Four things lower $\beta$ and raise power:

• larger sample size
• smaller standard error
• a true value farther from the null
• larger $\alpha$

This picture helps tie those ideas together. It shows the null and alternative distributions for a right-tailed test, along with the cutoff that separates the rejection region from the non-rejection region.

Power, $\beta$, and $\alpha$ on overlapping sampling distributions

Why each one works:

• Larger $n$ shrinks variability, so real differences are easier to detect.
• Smaller standard error means less overlap between null and alternative distributions.
• A true value close to the null is hard to detect. Farther away is easier.
• Larger $\alpha$ moves the cutoff so rejection is easier, so power goes up, but Type I error also goes up.

Choosing $\alpha$, Choosing Sample Size, and Exam Pitfalls

If a Type I error would be very serious, choose a smaller $\alpha$. If a Type II error would be very serious, plan for a larger sample size and stronger power.

A common target is power $\approx 0.80$, so $\beta \approx 0.20$, but context decides what matters most. Neither error is automatically worse.

These errors can happen even when the test is done correctly. They are different from bad arithmetic, biased sampling, or failed conditions. Also, statistical significance only says the data give evidence against $H_0$. It does not say the effect is large or important in real life.