Module 12: Multiple Regression Analysis

Supporting Lectures:

EGN3443 Module 12 - Multiple Regression Analysis

1. Multiple Regression Models

Definition

Multiple regression is a statistical technique that uses multiple independent variables to predict a single dependent variable. It extends simple linear regression by allowing more than one predictor variable in the model.

Key Characteristics

• Predicts a continuous outcome variable

• Uses two or more independent (predictor) variables

• Allows for more complex and nuanced predictions

• Helps understand the relationship between multiple factors and an outcome

General Model Equation

Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε

Where:

• Y = Dependent variable

• X₁, X₂, ..., Xₖ = Independent variables

• β₀ = Intercept

• β₁, β₂, ..., βₖ = Regression coefficients

• ε = Error term

Example Scenario

Predicting house prices based on:

• Square footage (X₁)

• Number of bedrooms (X₂)

• Neighborhood crime rate (X₃)

Web References

• https://en.wikipedia.org/wiki/Linear_regression

• Indeed.com (yes, the job site)

2. Parameter Estimation and Interpretation

Estimation Methods

1. Ordinary Least Squares (OLS): Most common method

   • Minimizes the sum of squared residuals

   • Provides unbiased estimates of regression coefficients

Coefficient Interpretation

• Each coefficient (βᵢ) represents the change in the dependent variable for one unit change in the independent variable, holding other variables constant

Example Calculation

# Hypothetical house price prediction model

# Y = Price

# X1 = Square footage

# X2 = Number of bedrooms

# Estimated coefficients

β₀ (Intercept) = 50,000

β₁ (Square footage) = 200

β₂ (Number of bedrooms) = 15,000

# Prediction equation

Price = 50,000 + 200(Square Footage) + 15,000(Number of Bedrooms)

# Example calculation

Price = 50,000 + 200(2000) + 15,000(3)

       = 50,000 + 400,000 + 45,000

       = $495,000

Hypothesis Testing

• t-tests for individual coefficients

• Assess statistical significance of each predictor

Web References

• Statistics How To - Regression Coefficients

3. Model Selection Techniques

Common Techniques

1. Forward Selection
   

   • Start with no variables

   • Add most significant predictors sequentially

2. Backward Elimination
   

   • Start with all variables

   • Remove least significant predictors

3. Stepwise Regression
   

   • Combination of forward and backward methods

   • Adds and removes variables based on statistical significance

Evaluation Metrics

• R-squared (R²)

• Adjusted R-squared

• Akaike Information Criterion (AIC)

• Bayesian Information Criterion (BIC)

Python Example

from sklearn.feature_selection import f_regression

from sklearn.linear_model import LinearRegression

# Perform feature selection

F_scores, p_values = f_regression(X, y)

# Select top predictors based on F-scores

Web References

• https://www.jmp.com/en/statistics-knowledge-portal/what-is-multiple-regression/variable-selection 

• Scikit-learn Model Selection

4. Multicollinearity

Definition

Multicollinearity occurs when independent variables are highly correlated with each other.

Detection Methods

1. Variance Inflation Factor (VIF)

   • Measures how much the variance of an estimated regression coefficient increases

VIF Calculation

from statsmodels.stats.outliers_influence import variance_inflation_factor

# Calculate VIF for each predictor

vif_data = pd.DataFrame()

vif_data["Variable"] = X.columns

vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

Interpretation

• VIF > 5-10 indicates significant multicollinearity

• Potential solutions:

  • Remove correlated predictors

  • Use regularization techniques

  • Collect more data

Web References

• Statistics How To - Multicollinearity

• Penn State - Multicollinearity

5. Polynomial Regression

Definition

Polynomial regression fits a nonlinear relationship between variables using polynomial equations.

General Model

Y = β₀ + β₁X + β₂X² + β₃X³ + ... + ε

Example

from sklearn.preprocessing import PolynomialFeatures

from sklearn.linear_model import LinearRegression

# Create polynomial features

poly = PolynomialFeatures(degree=2)

X_poly = poly.fit_transform(X)

# Fit regression model

model = LinearRegression()

model.fit(X_poly, y)

Use Cases

• Modeling curved relationships

• Capturing non-linear patterns in data

Cautions

• Risk of overfitting

• Careful selection of polynomial degree

Web References

• https://en.wikipedia.org/wiki/Polynomial_regression 

• Scikit-learn Polynomial Features

Recommended Learning Resources

1. Textbooks:
   

   • "Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences" by Jacob Cohen

   • "Regression Analysis by Example" by Samprit Chatterjee

2. Online Courses:
   

   • Coursera: Statistical Inference and Regression Models

   • edX: Regression Analysis