Simple Linear Regression

1 Simple Linear Regression

Simple linear regression is one of the most foundational and widely used techniques in statistics and data analysis. It provides a formal, mathematically grounded method for examining the relationship between two continuous variables — one that plays the role of the predictor and one that represents the outcome we are trying to understand or forecast. Before diving into equations and procedures, it helps to understand what the method is actually doing conceptually: it is asking, "if I know the value of one variable, how well can I predict the value of another, and what does the pattern of that prediction look like?"

The variable used to make the prediction is called the predictor variable, commonly denoted X. It is also referred to as the independent variable, the explanatory variable, or the regressor. The variable being predicted is called the outcome variable, commonly denoted Y, and is also known as the dependent variable or the response variable. The terminology "independent" and "dependent" reflects the idea that Y's value is modeled as depending on X, though it is important to remember that regression describes association, not causation, unless the data come from a carefully controlled experiment.

The central goal of simple linear regression is to find the best-fitting straight line through a scatter of data points — a line that summarizes how changes in X are systematically associated with changes in Y. The word "simple" distinguishes this case from multiple linear regression, which involves two or more predictor variables. Here, we work with exactly one predictor and one outcome. The method assumes that the relationship between X and Y follows a straight-line (linear) pattern. This is an important assumption: if the true relationship is curved or otherwise non-linear, a straight line will give a poor and potentially misleading summary of the data.

The Regression Equation

The mathematical form of a simple linear regression model is written as:

Ŷ = b₀ + b₁X

Each element of this equation has a specific and important meaning. Ŷ (read "Y-hat") is the predicted value of Y for a given value of X. The hat symbol is a standard statistical convention indicating that this is an estimate or prediction rather than an observed, exact value. Real data points will not all fall perfectly on the line — the line represents the average predicted value of Y at each level of X.

b₀ is the intercept of the regression line. Geometrically, it is the point where the line crosses the vertical Y-axis — in other words, it is the predicted value of Y when X equals zero. For example, if a regression model predicts a person's weight (Y) from their height in inches (X), the intercept would be the predicted weight when height is zero inches. In many real-world situations, X = 0 is outside the range of the data or is physically impossible, so the intercept may lack a meaningful practical interpretation. It still plays a critical mathematical role in anchoring the position of the line.

b₁ is the slope of the regression line. It represents the average change in Y for each one-unit increase in X. If the slope is 3.5, it means that for every additional one unit of X, the predicted value of Y increases by 3.5 units, on average. The slope is the most substantively interesting coefficient because it directly quantifies the direction and strength of the linear relationship between the two variables.

To make this concrete, consider a researcher studying the relationship between hours of study (X) and exam score (Y). Suppose the estimated regression equation is:

Ŷ = 45 + 5X

Here, b₀ = 45 means that a student who studies zero hours is predicted to score 45 points. b₁ = 5 means that each additional hour of study is associated with a predicted increase of 5 points on the exam. A student who studies 4 hours would be predicted to score Ŷ = 45 + 5(4) = 65 points.

Estimating the Regression Line: The Least Squares Method

Given a dataset with observed pairs of (X, Y) values, there are infinitely many possible straight lines one could draw through the scatter plot. The question is: which line is the best? The standard answer in classical statistics is to use the ordinary least squares (OLS) method, which defines "best" in a very precise and mathematically tractable way.

For any candidate line, we can compute a residual for each data point. A residual is the vertical distance between the observed Y value and the Y value predicted by the line for that same X value:

eᵢ = Yᵢ - Ŷᵢ

A positive residual means the line underestimated that observation; a negative residual means it overestimated. Residuals are the "errors" the model makes on the observed data. The least squares method finds the unique line that minimizes the sum of the squared residuals:

Minimize: Σ(Yᵢ - Ŷᵢ)² = Σeᵢ²

Why square the residuals rather than simply summing them? Because positive and negative residuals would otherwise cancel each other out, making a line with large errors appear to have none. Squaring ensures all contributions are positive, and it also penalizes large errors more heavily than small ones — a property that often aligns with our intuitions about what a good fit means.

Using calculus, it can be shown that the OLS estimates of the slope and intercept are given by the formulas:

b₁ = Σ[(Xᵢ - X̄)(Yᵢ - Ȳ)] / Σ[(Xᵢ - X̄)²]

b₀ = Ȳ - b₁X̄

Here, X̄ and Ȳ are the sample means of X and Y, respectively. Notice that b₀ is calculated after b₁, using the fact that the least squares line always passes through the point (X̄, Ȳ) — the means of the two variables. This is a useful property to remember: the regression line is anchored at the centroid of the data.

The OLS method has several attractive properties. Under standard regression assumptions (linearity, constant variance of errors, independence of observations, and normally distributed errors), OLS estimates are unbiased — meaning that on average across many samples they will equal the true population values — and they are also the most efficient among all linear unbiased estimators, a result known as the Gauss-Markov theorem.

Interpreting the Slope and Intercept

Interpreting regression coefficients carefully and in context is one of the most important skills in applied statistics. The slope b₁ carries the primary inferential message of the model. It tells us both the direction and the magnitude of the linear relationship:

• A positive slope (b₁ > 0) indicates a positive association: as X increases, predicted Y also increases. For instance, more fertilizer applied to crops might be associated with higher yield.
• A negative slope (b₁ < 0) indicates an inverse or negative association: as X increases, predicted Y decreases. For instance, increasing distance from a city center might be associated with lower property values.
• A slope of zero would indicate no linear relationship between X and Y — the regression line would be perfectly horizontal.

Crucially, the slope must always be interpreted with reference to the units of measurement. If X is measured in years and Y in thousands of dollars, then a slope of 2.3 means that each additional year is associated with an average increase of $2,300 in Y. Changing the units of X or Y (e.g., converting years to months) will change the numerical value of the slope, even though the underlying relationship is unchanged.

The intercept b₀ anchors the line but requires careful thought about interpretability. Consider a regression predicting annual salary (Y, in thousands of dollars) from years of work experience (X). If b₀ = 38, this says that a person with zero years of experience is predicted to earn $38,000. Here, X = 0 is plausible (a new hire), so the intercept has a sensible meaning. But in other contexts — for instance, predicting a person's blood pressure from their body weight where a weight of zero is impossible — the intercept is mathematically necessary but practically uninterpretable. Analysts should be cautious and explicit about this distinction.

Assessing Model Fit with R-Squared

Knowing that a regression line exists is not enough — we want to know how well it actually fits the data. The primary measure of model fit in simple linear regression is the coefficient of determination, R².

R² is defined as the proportion of the total variation in Y that is explained by the linear regression on X. It is computed as:

R² = 1 - (SSResidual / SSTotal)

where:
  SSTotal    = Σ(Yᵢ - Ȳ)²       (total variation in Y)
  SSResidual = Σ(Yᵢ - Ŷᵢ)²      (unexplained variation after fitting the model)
  SSRegression = SSTotal - SSResidual  (variation explained by the model)

R² ranges from 0 to 1. An R² of 0 means the predictor X explains none of the variation in Y — the regression line is no better than simply predicting the mean of Y for every observation. An R² of 1 means the predictor explains all variation in Y — every data point falls exactly on the regression line, which is essentially never observed in real data.

For example, if R² = 0.72, this means that 72% of the variability in Y is accounted for by the linear relationship with X, and the remaining 28% is due to other factors not captured by the model.

An important mathematical relationship exists in simple linear regression: R² equals the square of the Pearson correlation coefficient r. If the correlation between X and Y is r = 0.85, then R² = 0.85² = 0.7225, meaning the model explains approximately 72.25% of the variance in Y. This connection makes intuitive sense: the stronger the linear correlation, the better the regression line fits the data.

However, R² should not be used in isolation. A high R² does not guarantee that the model is correctly specified, that the assumptions of linear regression are met, or that the model will generalize well to new data. Conversely, a modest R² may still be highly meaningful in fields like social science or medicine where human behavior is inherently variable and hard to predict. R² must always be interpreted alongside other diagnostics and in the context of the subject matter.

Testing the Significance of the Regression Relationship

Even if OLS produces a non-zero slope estimate, we need to ask: could this apparent relationship simply be due to random chance in the sample? Hypothesis testing provides the formal framework for answering this question.

The standard null and alternative hypotheses are:

• H₀: β₁ = 0 — The population slope is zero; X has no linear relationship with Y.
• H₁: β₁ ≠ 0 — The population slope is not zero; there is a linear relationship between X and Y.

Note the use of β₁ (the Greek letter beta) to denote the true, unknown population parameter, as opposed to b₁, which is the estimate computed from our sample. To test this hypothesis, we calculate a t-statistic:

t = b₁ / SE(b₁)

where SE(b₁) is the standard error of the slope estimate — a measure of how much b₁ would vary across different random samples from the same population. This t-statistic follows a t-distribution with n − 2 degrees of freedom (we lose two degrees of freedom for estimating b₀ and b₁).

Equivalently, an F-test can be used, and in simple linear regression the F-statistic equals the square of the t-statistic: F = t². Both tests produce the same p-value and conclusion. The F-test framework becomes more natural when extending to multiple regression.

The p-value from the test tells us the probability of observing a slope as large as (or larger than) what we observed, assuming the null hypothesis is true. If p < 0.05 (using the conventional significance threshold), we reject H₀ and conclude that the slope is statistically significantly different from zero. This means the data provide sufficient evidence that X and Y are linearly related in the population.

Consider a practical example: a study examines whether advertising expenditure (X, in thousands of dollars) predicts monthly sales revenue (Y, in thousands of dollars). The OLS estimate yields b₁ = 4.2 with a standard error of 0.6, giving t = 4.2 / 0.6 = 7.0. With a large sample, this would correspond to an extremely small p-value (far below 0.05), providing strong evidence that advertising expenditure is a significant predictor of sales.

It is important not to conflate statistical significance with practical significance. A very large sample can make even a tiny slope statistically significant. Always assess whether the size of the slope is meaningful in practical terms, not just whether it is different from zero.

Using the Model for Prediction

One of the most useful applications of a fitted regression model is making predictions for new observations. Once b₀ and b₁ have been estimated and the model has been validated, predicting a Y value for a given X is straightforward: simply substitute the desired X value into the regression equation.

Returning to the study-hours example (Ŷ = 45 + 5X), if a new student plans to study 7 hours, the predicted exam score would be:

Ŷ = 45 + 5(7) = 45 + 35 = 80 points

However, this single predicted value — called a point estimate — does not convey any information about the uncertainty surrounding the prediction. There are two important types of intervals used to express this uncertainty:

• A confidence interval for the mean response gives a range of plausible values for the average Y at a given X. It reflects uncertainty in estimating the true regression line.
• A prediction interval gives a range within which we expect a single new individual observation to fall. It is always wider than a confidence interval because it must account for both the uncertainty in the line's position and the natural variability of individual data points around the line.

Both intervals are narrowest near X̄ (the mean of X) and widen as X moves farther from the center of the data, reflecting the fact that predictions become less reliable as we move away from the region where we have observed data.

A critical warning applies to prediction: extrapolation — using the model to predict Y for values of X far outside the range of the observed data — is dangerous. The linear relationship that holds within the observed range may not continue beyond it. For instance, a model showing that crop yield increases linearly with fertilizer application in the range of 0–100 kg/hectare may break down completely at 500 kg/hectare, where over-fertilization could reduce yield. Predictions should only be made within (or very close to) the range of X values used to build the model.

The following table summarizes the key components of a simple linear regression analysis and their roles:

In summary, simple linear regression offers a coherent and powerful framework — from data to equation to interpretation to prediction — for exploring and quantifying linear relationships between two variables. Its value lies not just in the formulas themselves, but in the disciplined thinking it requires: clearly defining what is being modeled, honestly assessing how well the model fits, rigorously testing whether observed patterns are real, and exercising caution when using the model to make predictions.