59 Multiple Linear Regression
Multiple linear regression extends simple linear regression to two or more independent variables, allowing an outcome to be predicted from — and explained by — several predictors simultaneously. This is closer to how most real business outcomes actually work: sales depend on more than just advertising spend; employee performance depends on more than just tenure.
59.1 The Multiple Regression Model
\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_k X_k + \varepsilon \]
Where each \(\beta_j\) (\(j = 1, \ldots, k\)) represents the change in \(Y\) for a one-unit increase in \(X_j\), holding all other predictors constant. This “holding constant” interpretation is what distinguishes a multiple-regression coefficient from a simple correlation — it isolates each predictor’s own contribution.
As in simple regression, the coefficients are estimated by minimising the sum of squared residuals, though with more than one predictor the calculation is normally done using matrix algebra rather than by hand — which is exactly why software is used from this point onward.
59.2 Adjusted R-squared
Because ordinary \(R^2\) mechanically increases every time a new predictor is added — even a useless one — multiple regression reports Adjusted R-squared, which penalises the addition of predictors that do not meaningfully improve the model: \[ R^2_{adj} = 1 - (1 - R^2)\frac{n-1}{n-k-1} \] where \(n\) is the sample size and \(k\) is the number of predictors. Adjusted \(R^2\) is the preferred metric for comparing models with different numbers of predictors.
59.3 Worked Example
Extending the advertising example, the company also records the number of active sales staff each month alongside advertising spend and sales revenue:
| Month | Advertising Spend (X₁) | Sales Staff (X₂) | Sales Revenue (Y) |
|---|---|---|---|
| 1 | 10 | 4 | 25 |
| 2 | 15 | 5 | 32 |
| 3 | 20 | 5 | 40 |
| 4 | 25 | 6 | 41 |
| 5 | 30 | 7 | 50 |
| 6 | 35 | 8 | 55 |
With two predictors, the least-squares coefficients are best obtained using software rather than hand calculation. Fitting this model (shown in R and Python below) gives approximately: \[ \hat{Y} = 8.99 + 0.72\,X_1 + 2.19\,X_2 \]
Interpretation: Holding the number of sales staff constant, each additional ₹1,000 of advertising spend is associated with about ₹0.72 lakh more in sales revenue. Holding advertising spend constant, each additional sales staff member is associated with about ₹2.19 lakh more in sales revenue.
59.4 Multiple Linear Regression in R and Python
Transition to Regression Diagnostics
Fitting a regression model is only half the job — the model’s coefficients and predictions are only trustworthy if the underlying assumptions (linearity, independence, homoscedasticity, normal residuals, limited multicollinearity) actually hold. The final chapter of this module covers regression diagnostics and model evaluation, the checks every fitted model should pass before being used for prediction or decision-making.
Summary
| Concept | Description |
|---|---|
| Foundations | |
| Multiple Linear Regression | Regression with two or more independent variables predicting one dependent variable |
| Multiple Regression Equation | Y equals beta-zero plus the sum of beta-j times X-j for each predictor, plus an error term |
| Holding Other Predictors Constant | Each coefficient represents the effect of its predictor with all other predictors fixed |
| Least Squares in Multiple Regression | Coefficients are estimated by minimising the sum of squared residuals, typically via matrix algebra |
| Model Fit | |
| Adjusted R-squared | R-squared adjusted for the number of predictors and sample size, penalising unhelpful additions |
| Why Adjusted R-squared Is Preferred | Ordinary R-squared always increases with more predictors even if they add no real explanatory power |
| Interpretation | |
| Coefficient Interpretation | Each coefficient shows the predicted change in Y per unit increase in that predictor, others held fixed |
| Prediction with Multiple Predictors | Substituting specific values for every predictor into the fitted equation to forecast Y |
| In R and Python | |
| R lm() with Multiple Predictors | R formula syntax that fits a multiple regression model by adding predictors with a plus sign |
| Python np.linalg.lstsq() | NumPy least-squares solver used to fit a multiple regression model from a design matrix |