| Concept | Description |
|---|---|
| Foundations | |
| Regression Analysis | A predictive analytics technique modelling the relationship between a dependent and one or more independent variables |
| Correlation vs Regression | Correlation measures strength/direction of association; regression predicts and quantifies the effect of X on Y |
| Simple Linear Regression | Regression with exactly one independent variable and one dependent variable |
| Multiple Linear Regression | Regression with two or more independent variables predicting one dependent variable |
| Logistic Regression | Regression used when the dependent variable is categorical, typically binary |
| The Model | |
| Regression Equation | Y equals beta-zero plus beta-one times X plus an error term |
| Intercept | The predicted value of Y when X equals zero |
| Slope | The change in Y for a one-unit increase in X |
| Error Term | The portion of variation in Y not explained by the predictor(s) |
| Method of Least Squares | The estimation method that minimises the sum of squared differences between observed and predicted Y |
| Assumptions | |
| Linearity Assumption | The relationship between predictors and the outcome should be linear |
| Independence Assumption | Observations and their errors should be independent of one another |
| Homoscedasticity Assumption | The variance of the errors should be constant across all levels of the predictor(s) |
| Normality of Errors Assumption | The residuals should be approximately normally distributed |
| Multicollinearity | In multiple regression, predictors should not be too highly correlated with one another |
57 Introduction to Regression Analysis
Regression analysis is a predictive analytics technique used to model the relationship between a dependent (outcome) variable and one or more independent (predictor) variables. Where correlation (introduced in Module IV) only measures the strength and direction of an association, regression goes a step further: it produces an equation that can be used to predict the outcome variable and to quantify exactly how much it changes as each predictor changes.
57.1 From Correlation to Regression
- Correlation answers “how strongly are these two variables related?” using a single number, \(r\), between -1 and 1.
- Regression answers “given a value of X, what value of Y do we predict, and how much does Y change per unit change in X?”
- Regression assumes a direction of influence — one or more predictor variables (X) are used to explain or predict an outcome variable (Y) — while correlation makes no such assumption.
57.2 Types of Regression
Simple Linear Regression
Models the relationship between exactly one independent variable and one dependent variable using a straight line. Example: Predicting monthly sales from advertising spend alone.
Multiple Linear Regression
Extends simple regression to two or more independent variables, allowing the combined and individual effect of several predictors to be estimated. Example: Predicting monthly sales from advertising spend, price, and number of sales staff.
Logistic Regression
Used when the dependent variable is categorical (typically binary, such as purchase / no purchase) rather than continuous. Logistic regression is covered separately from the linear-regression chapters that follow, but is mentioned here for completeness within the predictive-analytics landscape.
57.3 The General Idea: Fitting a Line (or Plane) to Data
For simple linear regression, the model takes the form: \[ Y = \beta_0 + \beta_1 X + \varepsilon \]
Where:
- \(Y\) is the dependent variable being predicted.
- \(X\) is the independent (predictor) variable.
- \(\beta_0\) is the intercept — the predicted value of \(Y\) when \(X = 0\).
- \(\beta_1\) is the slope — the change in \(Y\) for a one-unit increase in \(X\).
- \(\varepsilon\) is the error term, capturing the variation in \(Y\) not explained by \(X\).
The values of \(\beta_0\) and \(\beta_1\) are estimated from sample data using the method of least squares, which chooses the line that minimises the sum of squared differences between the observed values of \(Y\) and the values predicted by the line — covered in detail in the next chapter.
57.4 Key Assumptions of Linear Regression
- Linearity: The relationship between the predictors and the outcome is linear.
- Independence: Observations (and their errors) are independent of one another.
- Homoscedasticity: The variance of the errors is constant across all levels of the predictor(s).
- Normality of Errors: The residuals (errors) are approximately normally distributed.
- No (or Limited) Multicollinearity: In multiple regression, predictors should not be too highly correlated with each other.
Regression diagnostics — covered in the final chapter of this module — provide the tools to check whether these assumptions hold for a fitted model.
57.5 Business Applications of Regression
- Marketing: Predicting sales revenue from advertising spend across different channels.
- Finance: Estimating a stock’s return based on market indices (as in the Capital Asset Pricing Model).
- Human Resources: Predicting employee performance ratings from tenure, training hours, and engagement scores.
- Operations: Forecasting production cost based on volume, raw-material price, and labour hours.
- Real Estate: Estimating property prices from size, location, and number of rooms.
Transition to Simple Linear Regression
Having introduced the regression model and its assumptions, the next chapter works through simple linear regression end-to-end: fitting the least-squares line by hand, interpreting the coefficients, and assessing how well the model fits using \(R^2\).