Topic 2.8 Notes – Least Squares Regression

1. The Least Squares Regression Line

When you model a linear relationship, you write it as

$$\hat{y} = a + bx$$

• $x$ = explanatory variable
• $y$ = response variable
• $\hat{y}$ = predicted value of $y$

The least squares regression line is the line that minimizes the sum of the squared residuals.

A residual is

$$\text{residual} = y - \hat{y}$$

So for each data point, you:

• Find how far it is from the line (the residual)
• Square that value
• Add them all up

The LSRL makes that total as small as possible.

Why square?

• Prevents positive and negative residuals from canceling
• Penalizes large errors more heavily

One fact you should know cold:
The LSRL always passes through $(\bar{x}, \bar{y})$.

That connection to the means shows up on quizzes more than people expect.

2. The Components of the LSRL

Everything in the model connects back to summary statistics you already know.

Slope $b$

$$b = r\left(\frac{s_y}{s_x}\right)$$

Where:

• $r$ = correlation
• $s_y$ = standard deviation of $y$
• $s_x$ = standard deviation of $x$

What this tells you:

• The sign of $b$ always matches the sign of $r$
• The size depends on strength of relationship and relative spread

Interpretation

“The predicted change in (y in context) is ___ (units of y) for each 1 (unit of x).”

Example:
If $b = 2.4$ when modeling quiz score (points) from study time (hours):

“For each additional hour of study time, the predicted quiz score increases by 2.4 points.”

Always include:

• The word predicted
• Units
• Context

Students lose points every year by leaving out “predicted.”

Y-Intercept $a$

$$a = \bar{y} - b\bar{x}$$

This is the predicted value of $y$ when $x = 0$.

Interpretation:

“The predicted value of (y) is ___ when (x) is 0.”

Important: sometimes this makes no sense.

If you’re modeling college GPA from number of absences, what does “0 absences” mean? That’s fine.
But if you’re modeling house price from square footage, what does 0 square feet mean? Not realistic.

If 0 is outside the data range, don’t force meaning onto it.

Coefficient of Determination $r^2$

$$r^2 = (\text{correlation})^2$$

Also called R-squared.

• Between 0 and 1
• No direction
• Measures model fit

Interpretation:

“___% of the variation in (y) is explained by its linear relationship with (x).”

If $r^2 = 0.64$:

“64% of the variation in exam scores is explained by their linear relationship with hours studied.”

Two common mistakes:

• Saying “64% of exam scores are caused by…”
• Saying “64% of the points lie on the line.”

It’s about variation explained, not causation and not number of points.

3. Estimating the Regression Line from Summary Stats

If you’re given $r, s_x, s_y, \bar{x}, \bar{y}$, here’s the flow.

1. Compute slope
   $$b = r\left(\frac{s_y}{s_x}\right)$$
2. Compute intercept
   $$a = \bar{y} - b\bar{x}$$
3. Write the equation
   $$\hat{y} = a + bx$$

Example numbers:

Suppose
$r = 0.5$, $s_y = 10$, $s_x = 4$, $\bar{x} = 6$, $\bar{y} = 70$

$$b = 0.5 \left(\frac{10}{4}\right) = 1.25$$

$$a = 70 - 1.25(6) = 62.5$$

Regression line:

$$\hat{y} = 62.5 + 1.25x$$

On your calculator, LinReg gives:

• a (intercept)
• b (slope)
• r
• R-Sq

Always use R-Sq, not adjusted R-Sq.

4. Interpreting and Comparing Models

When reading regression output, you should be ready to interpret:

• Slope → rate of predicted change
• Intercept → predicted value at $x=0$
• r → strength and direction of linear relationship
• r² → percent of variation explained

When comparing two models:

Stronger linear model usually has:

• $|r|$ closer to 1
• Larger $r^2$
• Smaller unexplained variation

If slope magnitude decreases, the rate of predicted change becomes weaker.
If $r$ moves closer to 0, the linear relationship is weaker.