↑ Week 12/Module 11 - Regression Analysis

Module 11: Introduction to Regression Analysis

Supporting Lectures:
EGN3443 Module 11 - Introduction to Regression Analysis

1. Simple Linear Regression Model

Definition

A simple linear regression model is a statistical method used to understand the relationship between two variables by fitting a linear equation to observed data. It attempts to model the relationship between a dependent (outcome) variable and an independent (predictor) variable.

Mathematical Representation

The simple linear regression model is expressed as: Y = β₀ + β₁X + ε

Where:

Example Calculation

Suppose we want to predict a student's final exam score (Y) based on hours studied (X):

Web References

2. Least Squares Estimation

Definition

Least Squares Estimation is a method for finding the best-fitting line by minimizing the sum of the squared vertical distances between the observed data points and the predicted line.

Calculation Process

  1. Calculate the predicted values using the regression line

  2. Compute the residuals (differences between observed and predicted values)

  3. Square the residuals

  4. Sum the squared residuals

  5. Find the line that minimizes this sum

Example

Given data points:

The least squares method finds the line that minimizes the sum of squared vertical distances from these points to the line.

Calculation Formula

β₁ = Σ((X - X̄)(Y - Ȳ)) / Σ((X - X̄)²) β₀ = Ȳ - β₁X̄

Web References

https://www.scribbr.com/statistics/simple-linear-regression/ 

3. Regression Assumptions

Key Assumptions

  1. Linearity: The relationship between X and Y is linear

  2. Independence: Observations are independent of each other

  3. Homoscedasticity: Constant variance of residuals

  4. Normality: Residuals are normally distributed

  5. No Multicollinearity: Independent variables are not highly correlated

Diagnostic Techniques

Web References

4. Coefficient of Determination (R²)

Definition

R² measures the proportion of variance in the dependent variable that is predictable from the independent variable. It ranges from 0 to 1.

Calculation

R² = 1 - (Sum of Squared Residuals / Total Sum of Squares)

Interpretation

Example

If R² = 0.65 for a height prediction model, it means 65% of the variation in height can be explained by the predictor variable.

Web References

5. Interpreting Regression Outputs

Key Components to Analyze

  1. Coefficients

  2. P-values

  3. Confidence Intervals

  4. Standard Error

  5. F-statistic

Interpretation Steps

  1. Check coefficient significance (p-value < 0.05)

  2. Examine R² value

  3. Assess model fit through residual analysis

  4. Validate underlying assumptions

Example Output Interpretation

Dependent Variable: Sales

Independent Variable: Advertising Spend

Intercept: 10,000 (base sales when no advertising)

Slope: 5 (For every $1 spent on advertising, sales increase by $5)

R²: 0.75 (75% of sales variation explained by advertising)

P-value: 0.001 (statistically significant relationship)

Detailed Explanation of Output Interpretation

Regression results indicate the relationship between an independent variable and a dependent variable, and can be interpreted through coefficients, standard errors, p-values, and R-squared values. Coefficients show the change in the dependent variable for each unit change in the independent variable. Standard errors reflect the uncertainty in the estimated coefficients. P-values determine if a coefficient is statistically significant. R-squared measures how well the model fits the data, with higher values indicating better fit. 

1. Understanding Coefficients:

Sign:
A positive coefficient indicates a positive relationship, meaning as the independent variable increases, the dependent variable also tends to increase. A negative coefficient suggests a negative relationship, with the dependent variable decreasing as the independent variable increases. 
Magnitude:
The size of the coefficient represents the strength of the relationship. A larger coefficient indicates a stronger effect of the independent variable on the dependent variable. 
Interpretation:
Consider a simple linear regression where the coefficient for "hours studied" is 0.5. This means that for every additional hour studied, the exam score is predicted to increase by 0.5 points. 

2. Assessing Significance with p-values:

Null Hypothesis:
Regression analysis tests the null hypothesis that there is no relationship between the independent and dependent variables. 
P-value:
The p-value is the probability of observing the estimated coefficient (or a more extreme value) if the null hypothesis were true. 
Significance:
A low p-value (typically below 0.05) indicates that the coefficient is statistically significant, meaning the observed relationship is unlikely due to random chance. 
Interpretation:
If a variable has a p-value of 0.02, it's considered statistically significant, suggesting a real relationship between the variable and the dependent variable. 

3. Evaluating Model Fit with R-squared:

R-squared:
R-squared measures the proportion of variance in the dependent variable that is explained by the independent variable(s).
Interpretation:
An R-squared value of 0.75 means that 75% of the variation in the dependent variable can be explained by the model, indicating a good fit.
Limitations:
R-squared alone doesn't reveal whether the model is appropriate or if all predictors are truly useful. 

4. Assessing Standard Error and Confidence Intervals:

Standard Error:
The standard error quantifies the uncertainty in the estimated coefficient. A smaller standard error indicates a more precise estimate. 
Confidence Intervals:
Confidence intervals provide a range within which the true population parameter (e.g., the coefficient) is likely to fall. 
Interpretation:
If a coefficient has a standard error of 0.1 and a 95% confidence interval of [0.2, 0.4], this suggests that the true coefficient likely falls between 0.2 and 0.4. 

5. Checking Assumptions:

Linearity: The relationship between the independent and dependent variables should be linear.
Homoscedasticity: The variance of the errors should be constant across all levels of the independent variable.
Normality: The errors should be normally distributed.
Multicollinearity: Independent variables should not be highly correlated with each other.
Interpretation: Violations of these assumptions can lead to unreliable regression results. 
Example:
Consider a regression analysis predicting salary (dependent variable) based on years of experience (independent variable).
Coefficient:
The coefficient for years of experience might be 5000, meaning for each additional year of experience, salary is predicted to increase by $5000. 
P-value:
If the p-value is less than 0.05, the coefficient is statistically significant, suggesting that years of experience has a real effect on salary. 
R-squared:
An R-squared value of 0.65 means that 65% of the variance in salary is explained by years of experience. 
Standard Error:
A small standard error for the coefficient indicates a reliable estimate of the effect of experience on salary. 

Additional Learning Resources

Recommended Software