60 Regression Diagnostics and Model Evaluation
A regression model that fits the sample data well is not automatically a good model — its coefficients and predictions are only trustworthy if the assumptions introduced earlier in this module actually hold. Regression diagnostics are the set of checks, mostly based on the model’s residuals (the differences between observed and predicted values), used to validate a fitted model before relying on it.
60.1 Residual Analysis
The residual for observation \(i\) is: \[ e_i = Y_i - \hat{Y}_i \]
Residuals are the raw material of almost every diagnostic check:
- Residuals vs Fitted Plot: Plotting residuals against predicted values should show a random, patternless scatter around zero. A curved pattern suggests the true relationship is non-linear; a funnel shape suggests non-constant variance.
- Normal Q-Q Plot: Plots the residuals against theoretical normal quantiles; points falling close to a straight diagonal line support the normality assumption.
60.2 Checking Each Assumption
Linearity
Checked visually using the residuals-vs-fitted plot; a systematic curve indicates the relationship is not well captured by a straight line.
Homoscedasticity (Constant Variance)
Checked visually (a funnel-shaped spread of residuals indicates heteroscedasticity) or formally with tests such as the Breusch-Pagan test.
Normality of Residuals
Checked with a Q-Q plot or formally with the Shapiro-Wilk test.
Independence of Errors
Especially important for time-ordered data; checked with the Durbin-Watson test, which detects autocorrelation in residuals.
Multicollinearity
Checked using the Variance Inflation Factor (VIF) for each predictor in a multiple regression model: \[ VIF_j = \frac{1}{1 - R_j^2} \] where \(R_j^2\) is the R-squared obtained from regressing predictor \(X_j\) on all other predictors. A VIF above about 5 or 10 signals problematic multicollinearity.
60.3 Model Evaluation Metrics
R-squared and Adjusted R-squared
Introduced in the previous two chapters; describe the proportion of variance explained by the model, with Adjusted \(R^2\) correcting for the number of predictors.
Root Mean Squared Error (RMSE)
Measures the typical size of prediction error, in the same units as the outcome variable: \[ RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2} \]
Mean Absolute Error (MAE)
The average absolute size of the prediction errors, less sensitive to large outliers than RMSE: \[ MAE = \frac{1}{n}\sum_{i=1}^{n}|Y_i - \hat{Y}_i| \]
A lower RMSE or MAE indicates a better-fitting model; the two are typically compared across candidate models trained and evaluated on the same data.
60.4 Regression Diagnostics in R and Python
60.5 Best Practices for a Trustworthy Regression Model
- Always inspect residual plots — never rely on \(R^2\) alone to judge a model’s adequacy.
- Report Adjusted \(R^2\), RMSE, and/or MAE alongside coefficients so readers can judge both fit and predictive accuracy.
- Check VIF for every predictor in a multiple regression model before interpreting individual coefficients.
- Remember that a statistically significant coefficient does not, by itself, establish causation — regression describes association captured in observational data unless the data comes from a designed experiment.
- Validate the model on data it was not fitted on (a hold-out sample) whenever the goal is prediction rather than pure explanation.
Closing This Module
With regression analysis, this book has now moved from describing data (Module III), through testing formal hypotheses about it (Module IV), to predicting future outcomes from it (this module). The predictive techniques introduced here — simple regression, multiple regression, and the diagnostics that validate them — are the foundation for the more advanced predictive and prescriptive analytics techniques used in professional practice.
Summary
| Concept | Description |
|---|---|
| Residual Diagnostics | |
| Residual | The difference between an observed value and the value predicted by the regression model |
| Residuals vs Fitted Plot | Plots residuals against fitted values to check for non-linearity or non-constant variance |
| Normal Q-Q Plot | Plots residuals against theoretical normal quantiles to check the normality assumption |
| Assumption Checks | |
| Linearity Check | Verified visually via the residuals-vs-fitted plot for systematic curvature |
| Homoscedasticity Check | Verified visually or with tests such as Breusch-Pagan for a funnel-shaped residual spread |
| Normality Check | Verified with a Q-Q plot or formally with the Shapiro-Wilk test |
| Independence Check (Durbin-Watson) | Verified with the Durbin-Watson test, especially important for time-ordered data |
| Multicollinearity and VIF | Variance Inflation Factor measures how much a predictor's variance is inflated by correlation with other predictors |
| Evaluation Metrics | |
| R-squared / Adjusted R-squared | Proportion of variance in the outcome explained by the model, with the adjusted version penalising extra predictors |
| RMSE | Root Mean Squared Error, the typical size of prediction error in the outcome's own units |
| MAE | Mean Absolute Error, the average absolute prediction error, less sensitive to outliers than RMSE |
| Best Practices | |
| Correlation vs Causation Caveat | A significant regression coefficient shows association, not proof that X causes Y, outside designed experiments |
| Hold-out Validation | Evaluating a model's predictive accuracy on data it was not fitted on |