Residual analysis is a statistical technique used to assess the quality and assumptions of a regression model by examining the differences between observed and predicted values.
Residuals: The difference between actual observed values and values predicted by the regression model
Residual = Observed Value - Predicted Value
Graphical Methods
Residual plots
Q-Q plots
Scatter plots of residuals vs. predicted values
Statistical Tests
Durbin-Watson test for autocorrelation
Breusch-Pagan test for heteroscedasticity
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
# Sample data
X = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)
y = np.array([2, 4, 5, 4, 5])
# Fit linear regression
model = LinearRegression()
model.fit(X, y)
# Calculate residuals
y_pred = model.predict(X)
residuals = y - y_pred
# Plot residuals
plt.scatter(y_pred, residuals)
plt.title('Residual Plot')
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
Influence diagnostics identify data points that have a significant impact on the regression model's parameters and overall fit.
Leverage
Measures how far an independent variable deviates from its mean
High leverage points can disproportionately influence regression results
Cook's Distance
Measures the influence of a data point on the regression coefficients
Values > 1 typically indicate influential observations
import numpy as np
from statsmodels.stats.outliers_influence import OLSInfluence
import statsmodels.api as sm
# Sample regression model
X = sm.add_constant(X)
model = sm.OLS(y, X).fit()
# Compute influence measures
influence = OLSInfluence(model)
leverage = influence.hat_matrix_diag
cooks_distance = influence.cooks_distance[0]
Transformations are mathematical modifications applied to variables to improve the regression model's fit, linearity, or meet statistical assumptions.
Log Transformation
Helps linearize exponential relationships
Reduces the impact of outliers
Square Root Transformation
Useful for count data
Stabilizes variance
Box-Cox Transformation
Systematic method to determine optimal transformation
import numpy as np
from scipy import stats
# Log transformation
log_transformed = np.log(X)
# Box-Cox transformation
transformed_data, lambda_param = stats.boxcox(X)
Cross-validation is a resampling method used to evaluate a model's performance and generalizability.
K-Fold Cross-Validation
Divides data into K equal subsets
Uses K-1 folds for training, 1 for testing
Leave-One-Out Cross-Validation (LOOCV)
Uses a single observation as validation set
Repeats process for each observation
Stratified K-Fold
Maintains class distribution in each fold
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LinearRegression
# K-Fold Cross-Validation
kf = KFold(n_splits=5)
model = LinearRegression()
scores = cross_val_score(model, X, y, cv=kf)
Prediction intervals estimate the range within which a future individual observation is likely to fall, considering both model uncertainty and inherent variability.
Wider than confidence intervals
Accounts for individual prediction variability
Formula:
Prediction Interval = Predicted Value ± t * SE * √(1 + 1/n)
import numpy as np
import scipy.stats as stats
def prediction_interval(X, y, X_new, confidence=0.95):
model = np.polyfit(X, y, 1)
y_pred = np.polyval(model, X_new)
# Calculation details omitted for brevity
return y_pred, interval
# Usage
pred, pred_interval = prediction_interval(X, y, X_new)
This guide covers essential techniques for validating regression models, ensuring robust and reliable statistical analysis.