Applying Regression to Real-World Datasets

1 Applying Regression to Real-World Datasets

Regression analysis moves from textbook exercises to genuine insight only when applied to real-world data. Real datasets are messy, multifaceted, and full of surprises — missing entries, skewed distributions, correlated predictors, and outcomes that resist tidy linear relationships. Working through these challenges is exactly what transforms a theoretical understanding of regression into a practical, transferable skill. This topic walks through every stage of an applied regression project: selecting and preparing data, defining the research question, building and interpreting the model, diagnosing assumption violations, refining the specification, and finally communicating results to an audience that may have little statistical background.

Selecting and Preparing a Real-World Dataset

The foundation of any regression project is a well-chosen dataset. A good dataset for regression has a clearly identifiable dependent variable — something you genuinely want to understand or predict — along with one or more candidate predictors that have a plausible connection to that outcome. Classic examples include predicting house sale prices from square footage, number of bedrooms, and neighborhood characteristics; forecasting patient hospital readmission from age, diagnosis codes, and length of stay; or explaining student exam scores from study hours, prior GPA, and socioeconomic indicators.

Before any modeling begins, the data must be thoroughly inspected and cleaned. Raw datasets almost always contain problems that, if ignored, will distort regression coefficients and invalidate statistical tests.

• Missing values: Identify every column with missing entries and decide on a strategy. Simple options include listwise deletion (dropping any row with a missing value), which is safe when missingness is rare and random, and mean or median imputation, which replaces missing values with the column's central tendency. More sophisticated approaches such as multiple imputation or model-based imputation are warranted when missingness is substantial or systematic. For example, in a housing dataset, if the garage_size field is blank only for older homes, imputing with the overall mean would introduce a systematic error; a better strategy might be imputing from a subset of comparable homes.
• Outliers: Extreme values can exert disproportionate influence on the least-squares regression line. Use scatter plots, box plots, and z-scores (flagging values beyond ±3 standard deviations) to locate outliers. Investigate each one: Is it a data-entry error (a house recorded as 50,000 square feet instead of 5,000)? If so, correct or remove it. Is it a genuine extreme case? If so, consider whether it belongs in the analysis or should be handled with a robust regression method.
• Duplicates and inconsistencies: Check for duplicate rows that could artificially inflate sample size and bias estimates. Standardize categorical variable formats — for instance, "New York," "new york," and "NY" should all map to a single consistent label before encoding.
• Descriptive statistics and visualization: Before running any model, compute means, medians, standard deviations, and ranges for every variable. Plot histograms to understand distributions, scatter plots to visualize pairwise relationships between predictors and the outcome, and a correlation matrix heatmap to get an early sense of multicollinearity. These exploratory steps often reveal the need for transformations before modeling even begins.

Consider a concrete example. Suppose you download a publicly available dataset of used car listings with variables including price (the outcome), mileage, age, engine_size, fuel_type, and brand. Initial inspection finds that 8% of price values are missing (handled by imputing from comparable brand/age/mileage combinations), three rows have mileage = 0 (likely data errors, removed), and fuel_type is coded inconsistently across rows. After cleaning, a histogram of price reveals strong right skew, suggesting a log transformation may be appropriate — a note to revisit during model refinement.

Identifying Variables and Formulating a Research Question

A precise research question anchors every modeling decision that follows. Without one, variable selection becomes arbitrary and interpretation loses direction. A well-formed research question specifies the outcome, the key predictors of interest, and the population or context to which findings will apply.

• Defining the dependent variable: The dependent variable (also called the outcome, response, or criterion variable) is what you are trying to explain or predict. It must be measured on a continuous scale for standard linear regression. In the car example, price is the natural dependent variable because understanding what drives price is directly actionable for buyers, sellers, and platforms alike.
• Identifying independent variables: Independent variables (predictors, explanatory variables, covariates) should be chosen based on both theoretical reasoning and empirical availability. Theory first: what factors are known or hypothesized to influence car prices? Mileage (higher mileage → lower price), age (older → lower price), engine size (larger → often higher price in certain segments), fuel type, and brand reputation are all defensible candidates. Avoid including variables purely because they are available; every additional predictor consumes degrees of freedom and risks overfitting.
• Simple versus multiple regression: If a single dominant predictor explains most variance and the research question is focused (e.g., "How much does mileage alone predict price?"), simple linear regression suffices. When multiple predictors simultaneously influence the outcome and controlling for confounders matters, multiple regression is necessary. For the car dataset, omitting age when estimating the effect of mileage would likely produce a biased coefficient because older cars tend to have more mileage; multiple regression isolates each predictor's independent contribution.
• Variable scales and encoding: Linear regression requires that the dependent variable be continuous. Predictors can be continuous (mileage, age) or categorical. Categorical variables must be dummy-coded (also called one-hot encoding): a variable with k categories is represented by k − 1 binary (0/1) columns, with one category serving as the reference group. For fuel_type with categories Petrol, Diesel, and Electric, you would create two dummies — is_diesel and is_electric — with Petrol as the implicit baseline. The coefficient on is_diesel then represents the average price difference between Diesel and Petrol vehicles, holding all else constant.

A well-formed research question for this example might be: "Which combination of mileage, vehicle age, engine size, fuel type, and brand best predicts the sale price of used cars listed on this platform, and what is the independent contribution of each factor?"

Building the Regression Model on Real Data

With a clean dataset and a clear research question, the next step is to estimate the model. In Python, the statsmodels library provides detailed output suited for inference, while scikit-learn is often preferred for predictive modeling pipelines. In R, the built-in lm() function is the standard tool. The example below uses Python with statsmodels.

import pandas as pd
import statsmodels.api as sm

# Load cleaned dataset
df = pd.read_csv('used_cars_clean.csv')

# Define predictors and outcome
X = df[['mileage', 'age', 'engine_size', 'is_diesel', 'is_electric']]
y = df['log_price']   # log-transformed price

# Add intercept constant
X = sm.add_constant(X)

# Fit the model
model = sm.OLS(y, X).fit()

# Display full regression table
print(model.summary())

Running this procedure yields a regression summary table. The key outputs include:

• Estimated coefficients (β̂): One for each predictor plus the intercept. These quantify the direction and magnitude of each predictor's relationship with the outcome.
• Standard errors: Measure the precision of each coefficient estimate. Smaller standard errors indicate more reliable estimates.
• t-statistics and p-values: Used to test whether each coefficient is significantly different from zero.
• R-squared and Adjusted R-squared: Indicate how much variance in the outcome the model accounts for.
• F-statistic: Tests whether the model as a whole explains significant variance, i.e., whether at least one predictor is related to the outcome.

Before moving to interpretation, verify that the model converged (no warnings about singular matrices or failed optimization) and that the number of observations matches expectations after cleaning.

Interpreting Regression Output in Context

Interpreting regression output requires translating mathematical coefficients into statements that are meaningful within the substantive domain. Below is an illustrative (hypothetical) output table for the used car model.

Several interpretation principles deserve emphasis:

• Ceteris paribus (all else equal): Every coefficient describes the relationship between one predictor and the outcome holding all other predictors constant. The coefficient on mileage tells you the expected change in log-price for a one-unit increase in mileage among cars that are otherwise identical in age, engine size, fuel type, etc. This is the key advantage of multiple regression over simple bivariate analysis.
• Log-transformed outcomes: When the dependent variable is log-transformed, coefficients are interpreted as proportional changes. A coefficient of −0.087 on age means each additional year is associated with a multiplicative change of e^−0.087 ≈ 0.917 in price — roughly an 8.3% reduction. For small coefficients (|β| < 0.1), the percentage approximation β × 100% is accurate enough for communication.
• R-squared: If R² = 0.74, the model explains 74% of the variance in log-price. This is a measure of fit, not of whether individual predictors are causal. A high R² with non-significant predictors can occur when predictors are highly correlated; a low R² with significant predictors can occur in noisy domains where many unmeasured factors influence the outcome.
• Statistical versus practical significance: A predictor can be highly statistically significant (very small p-value) yet have a negligible real-world effect, especially with large samples. Always consider the magnitude of coefficients alongside their significance. Conversely, a coefficient that is practically large but not statistically significant may simply reflect insufficient data.
• Relating findings to the research question: The model shows that vehicle age is the single strongest depreciating factor (largest standardized coefficient magnitude), that electric vehicles command a substantial premium, and that mileage matters but less so than age when both are controlled for simultaneously. These findings are directly actionable: a used car platform could use this model to flag mispriced listings or provide sellers with price guidance.

Evaluating Model Fit and Assumptions with Real Data

A regression model is only as trustworthy as its underlying assumptions. Violating these assumptions does not always invalidate the model, but it does affect the reliability of coefficient estimates, standard errors, and significance tests. The four core assumptions of ordinary least squares (OLS) regression are linearity, independence of errors, homoscedasticity (constant variance of residuals), and normality of residuals.

• Linearity: The relationship between each predictor and the outcome (after accounting for other predictors) should be approximately linear. Check this with partial regression plots (added-variable plots) or by plotting residuals against each predictor. A curved pattern in a residuals-vs-predictor plot signals non-linearity. For example, if plotting residuals against mileage shows a U-shape, the true relationship may be quadratic, requiring a squared term.
• Homoscedasticity: Residual variance should be roughly constant across all fitted values. A fan-shaped pattern in the residuals vs. fitted values plot — where spread grows with fitted values — indicates heteroscedasticity. This is common when the outcome is a dollar amount (price variance tends to grow with price level), which is precisely why log-transforming price often stabilizes variance. The Breusch-Pagan test provides a formal statistical check.
• Normality of residuals: OLS coefficient estimates are unbiased regardless of the residual distribution, but p-values and confidence intervals rely on the normality assumption. Assess this with a Q-Q plot (quantile-quantile plot) of residuals — points should fall approximately on the diagonal line. The Shapiro-Wilk test is appropriate for small samples; for large samples (n > 1,000), the central limit theorem renders minor departures from normality practically inconsequential.
• Multicollinearity: When two or more predictors are highly correlated with each other, their individual coefficients become unstable and standard errors inflate, making it difficult to discern the independent contribution of each variable. Detect multicollinearity with a correlation matrix (flag pairs with |r| > 0.80) and with Variance Inflation Factors (VIF). A VIF above 5 warrants attention; above 10 is typically considered severe. In the car dataset, mileage and age are likely correlated (r ≈ 0.65), but not so severely as to require action. If you were to include both engine_size and a highly correlated cylinder_count, VIF values would likely exceed 10, necessitating the removal of one.
• Independence of errors: Residuals should not be correlated with each other. This assumption is most often violated in time-series data (autocorrelation) or clustered data (e.g., students within schools). The Durbin-Watson statistic tests for first-order autocorrelation; values near 2 indicate no autocorrelation.

A practical diagnostic workflow in Python:

import matplotlib.pyplot as plt
import scipy.stats as stats
import numpy as np

fitted = model.fittedvalues
residuals = model.resid

# 1. Residuals vs. Fitted — check linearity and homoscedasticity
plt.scatter(fitted, residuals, alpha=0.4)
plt.axhline(0, color='red', linestyle='--')
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted')
plt.show()

# 2. Q-Q plot — check normality
stats.probplot(residuals, dist="norm", plot=plt)
plt.title('Normal Q-Q Plot')
plt.show()

# 3. VIF — check multicollinearity
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif_data = pd.DataFrame()
vif_data['Feature'] = X.columns
vif_data['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)

Refining and Improving the Model

Rarely does the first model specification turn out to be the best one. Diagnostic results, theoretical reconsideration, and comparison of fit statistics all drive iterative refinement.

• Variable transformations: If the residuals-vs-fitted plot shows a fan shape even after log-transforming the outcome, consider transforming skewed predictors as well. Mileage, for instance, is often right-skewed; applying a log transformation (log_mileage) can linearize the relationship and reduce heteroscedasticity. Square root transformations are milder alternatives for moderately skewed variables. When a predictor's relationship with the outcome is clearly non-linear (U-shaped or inverted-U), adding a squared term (e.g., age²) creates a polynomial regression that captures the curvature while remaining estimable with standard OLS.
• Adding or removing predictors: If theory suggests an important variable was omitted, add it and check whether it is significant and whether model fit improves. Be cautious about adding variables purely to inflate R²; use adjusted R², which penalizes for additional predictors, or AIC (Akaike Information Criterion) to compare models of different sizes. Lower AIC indicates a better balance between fit and parsimony.
• Interaction terms: Sometimes the effect of one predictor depends on the level of another. For example, the price premium for an electric vehicle might be larger for newer cars than for older ones. Including an interaction term is_electric × age tests and quantifies this conditional relationship. Interpreting interactions requires care: each interacting variable's coefficient now represents its effect only when the other is zero.
• Influential observations: Cook's Distance measures how much each observation influences the estimated coefficients. Observations with Cook's D greater than 4/n are often considered influential and warrant closer inspection — they may be legitimate data points from an unusual segment or they may be errors requiring correction.
• Transparency and reproducibility: Document every decision: why a variable was removed, what transformation was applied and why, which model specification was selected as final. This documentation allows others (and your future self) to audit the analysis, reproduce results, and extend the work. Version-controlled scripts (e.g., in a Git repository) and annotated notebooks (Jupyter, R Markdown) serve this purpose well.

A structured comparison of candidate models aids the selection process:

Model M4 is selected as the final specification: it has the best adjusted R² and lowest AIC, the interaction term is theoretically defensible (electric vehicle technology and battery concerns make older electrics proportionally less attractive), and diagnostic plots show no major assumption violations.

Communicating Regression Findings to Stakeholders

The most technically sophisticated model delivers no value if its findings cannot be understood and acted upon by decision-makers. Translating regression results into clear, accessible communication is a distinct and critical skill.

• Plain-language summaries: Replace statistical jargon with concrete statements. Instead of "the coefficient on age is −0.087 (p < 0.001)," say: "On average, a car loses about 8% of its value for every additional year of age, even when mileage, engine size, and fuel type are the same." This framing retains accuracy while being immediately interpretable by a non-statistician.
• Visualizations: A well-designed chart communicates in seconds what a table of coefficients communicates in minutes. Useful visualizations for regression results include:
  • Coefficient plots: Display each predictor's coefficient and confidence interval as a horizontal bar or dot plot, ordered by effect size, making it easy to see which predictors matter most and in which direction.
  • Partial regression plots: Show the relationship between one predictor and the outcome after controlling for other predictors — especially useful for continuous predictors.
  • Predicted vs. actual plots: Plot model predictions against observed values; points close to the 45° diagonal indicate good fit.
  • Scenario-based predictions: Create a small table or bar chart showing predicted prices for illustrative car profiles (e.g., a 3-year-old electric with 30,000 km vs. a 3-year-old petrol with the same mileage), making the model's implications concrete.
• Acknowledging limitations: Honest communication includes disclosing what the model does not account for. In the car example: the dataset may not include condition rating or service history, both of which affect price; the model applies to the specific platform and time period of the data and may not generalize to other markets; and the model establishes association, not causation — higher engine size predicts higher price, but recommending that someone install a larger engine to increase resale value would be a misuse of these findings.
• Actionable recommendations: Regression findings should ideally translate into decisions. For a used car platform, recommendations from this model might include: implementing an automated pricing tool that flags listings more than 15% above or below model predictions as potentially mispriced; advising sellers that reducing asking price to reflect age depreciation will likely reduce time-to-sale; and prioritizing the acquisition of electric vehicle inventory given the demonstrated price premium. Grounding recommendations in the actual coefficient estimates makes them more credible and precisely targeted.

The entire applied regression workflow — from raw data to communicated insight — is iterative and reflexive. Early exploratory choices inform model building; diagnostic results loop back to data preparation; stakeholder feedback may reframe the research question. Mastering this full cycle, rather than any single step within it, is what defines competence in applied regression analysis.