Coefficient of Determination (R²)

The coefficient of determination, R-squared, is a key metric for evaluating regression models. It provides an intuitive measure of how much of the variability in the outcome is captured by the model. It is derived from two key components. The first is the Total Sum of Squares ([latex]SS_{tot} = \sum_i (y_i – \bar{y})^2[/latex]), which measures the total variance in the dependent variable [latex]y[/latex]. The second is the Residual Sum of Squares ([latex]SS_{res} = \sum_i (y_i – \hat{y}_i)^2[/latex]), which measures the variance left unexplained by the model, where [latex]\hat{y}_i[/latex] is the predicted value.

The formula [latex]R^2 = 1 – SS_{res}/SS_{tot}[/latex] can be interpreted as the percentage of total variance that is ‘explained’ by the regression model. For instance, an R² of 0.75 means that 75% of the variability in the outcome can be accounted for by the predictors in the model. In simple linear regression, R² is simply the square of Pearson’s correlation coefficient (r) between the observed and predicted values.

However, R² has a significant limitation: it never decreases when a new predictor variable is added to the model, even if the new variable is irrelevant. This can be misleading and encourage overfitting. To counteract this, the Adjusted R-squared is often used. It modifies the R² value to account for the number of predictors in the model, providing a more accurate measure of goodness of fit for multiple regression.