Supervised Learning & Regression Mathematics

Supervised learning constitutes the foundational pillar of modern statistical machine learning and empirical pattern recognition. In the supervised learning paradigm, an algorithm is provided with an inductive training dataset comprising paired feature vectors and ground-truth target annotations. The mathematical objective is to deduce a mapping function—termed a hypothesis—that minimizes expected risk and generalizes accurately to unseen observations drawn from the same underlying probability distribution.

Within supervised learning, regression problems specifically address tasks where the target variable is continuous, real-valued, and unbounded or semi-bounded (Y ⊆ ℝ). This guide provides an exhaustive, university-grade examination of regression mathematics, establishing rigorous derivations for Ordinary Least Squares (OLS), gradient-based optimization mechanics, the bias-variance decomposition, and L1/L2 regularization geometries.

Formal Problem Formulation & Empirical Risk Minimization#

Let the input feature space be denoted by X ⊆ ℝ^d, where each observation x = [x_1, x_2, ..., x_d]^T represents a d-dimensional column vector of measurable attributes. Let the target output space be denoted by Y ⊆ ℝ. We assume the existence of an unknown joint probability distribution D over X × Y from which our training dataset S of m independent and identically distributed (i.i.d.) observations is sampled:

S = { (x^(1), y^(1)), (x^(2), y^(2)), ..., (x^(m), y^(m)) } ~ D^m

The goal of learning is to select a hypothesis function h_θ: X → Y from a hypothesis class H parametrized by weight vector θ ∈ ℝ^d (and bias scalar b ∈ ℝ) that minimizes the Generalization Error (also called True Risk):

R(h_θ) = E_(x,y)~D [ L(h_θ(x), y) ] = ∫ L(h_θ(x), y) · p(x, y) dx dy

where L(y_hat, y) is a task-specific loss function measuring discrepancy between predicted output y_hat = h_θ(x) and ground-truth y.

Because the underlying distribution D is fundamentally unknown in practice, we cannot compute the true risk directly. Instead, statistical learning theory relies on the principle of Empirical Risk Minimization (ERM): minimizing the average empirical loss across the observed training set S:

R_emp(h_θ) = (1 / m) · ∑ [i = 1 to m] L(h_θ(x^(i)), y^(i))

+---------------------------------------------------------------------------------------+
|                          Empirical Risk Minimization Pipeline                         |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   True Data Distribution D(X, Y) ---> Sample m i.i.d. instances ---> Training Set S   |
|                                                                             |         |
|                                                                             v         |
|   Hypothesis Space H <--- Optimization Algorithm (OLS / SGD) <--- Loss Function L     |
|          |                                                                            |
|          v                                                                            |
|   Selected Predictor h_θ(x) ---> Evaluation on Unseen Test Distribution (Generalization)
|                                                                                       |
+---------------------------------------------------------------------------------------+

Linear Regression & Ordinary Least Squares (OLS)#

In linear regression, the hypothesis function h_θ(x) assumes a linear relationship between input attributes and the continuous target:

h_θ(x) = θ_0 + θ_1 · x_1 + θ_2 · x_2 + ... + θ_d · x_d = θ_0 + ∑ [j = 1 to d] θ_j · x_j

To vectorize notation cleanly, we adopt the standard convention of appending an artificial intercept feature x_0 = 1 to each observation vector, yielding an augmented feature vector x ∈ ℝ^(d + 1) and parameter vector θ ∈ ℝ^(d + 1):

x = [1, x_1, x_2, ..., x_d]^T
θ = [θ_0, θ_1, θ_2, ..., θ_d]^T
h_θ(x) = θ^T x = x^T θ

For the entire dataset of m observations, we stack row vectors to construct the Design Matrix X ∈ ℝ^(m × (d + 1)) and the target vector y ∈ ℝ^m:

    | -- (x^(1))^T -- |     | 1   x_1^(1)   ...   x_d^(1) |
X = | -- (x^(2))^T -- |  =  | 1   x_1^(2)   ...   x_d^(2) |
    |       ...       |     | ...   ...     ...     ...   |
    | -- (x^(m))^T -- |     | 1   x_1^(m)   ...   x_d^(m) |

y = [ y^(1), y^(2), ..., y^(m) ]^T

The vectorized predictions across all m samples are given compactly by:

y_hat = X θ ∈ ℝ^m

The Mean Squared Error (MSE) Cost Function#

Under Ordinary Least Squares, the loss function is the squared Euclidean distance between predicted and actual values. The objective cost function J(θ), scaled by a standard factor of 1 / (2m) for convenient differentiation, is:

J(θ) = (1 / (2m)) · ∑ [i = 1 to m] (h_θ(x^(i)) - y^(i))^2
     = (1 / (2m)) · ||X θ - y||^2
     = (1 / (2m)) · (X θ - y)^T (X θ - y)

Expanding the matrix quadratic form:

(X θ - y)^T (X θ - y) = ((X θ)^T - y^T) (X θ - y)
                      = (θ^T X^T - y^T) (X θ - y)
                      = θ^T X^T X θ - θ^T X^T y - y^T X θ + y^T y

Because θ^T X^T y is a scalar (dimension 1 × 1), it equals its own transpose: (θ^T X^T y)^T = y^T X θ. Thus:

J(θ) = (1 / (2m)) · [ θ^T X^T X θ - 2 · y^T X θ + y^T y ]

Closed-Form Solution: The Normal Equations#

Because J(θ) is a convex quadratic function in θ, its global minimum occurs where the gradient vector ∇_θ J(θ) equals the zero vector 0 ∈ ℝ^(d + 1).

Matrix Calculus Derivation#

Recalling standard matrix calculus identities for symmetric matrices and vectors:

1. ∇_θ (a^T θ) = a
2. ∇_θ (θ^T A θ) = 2 A θ (when A is symmetric; here A = X^T X is inherently symmetric because (X^T X)^T = X^T X)

Differentiating J(θ) with respect to the parameter vector θ:

∇_θ J(θ) = ∇_θ [ (1 / (2m)) · (θ^T X^T X θ - 2 · y^T X θ + y^T y) ]
         = (1 / (2m)) · [ 2 · X^T X θ - 2 · X^T y + 0 ]
         = (1 / m) · [ X^T X θ - X^T y ]

Setting the gradient to zero:

(1 / m) · [ X^T X θ - X^T y ] = 0
X^T X θ = X^T y

Assuming that the matrix product X^T X ∈ ℝ^((d + 1) × (d + 1)) is non-singular (invertible), we multiply both sides by (X^T X)^(-1) to obtain the famous Normal Equation:

θ* = (X^T X)^(-1) X^T y

Invertibility & Geometric Interpretation#

The matrix X^T X fails to be invertible (is singular) under two specific circumstances:

1. Redundant Features (Multicollinearity): Two or more features are linearly dependent (e.g., x_1 = 2 · x_2, or measuring distance in both meters and feet).
2. High-Dimensional Regime (m < d + 1): The number of features exceeds the number of training observations, rendering the rank of X at most m, which is strictly less than d + 1.

Geometrically, the prediction vector y_hat = X θ* represents the orthogonal projection of the observation target vector y ∈ ℝ^m onto the subspace spanned by the column vectors of the design matrix X (Col(X)):

Residual vector e = y - X θ* is strictly orthogonal to Col(X):
X^T e = X^T (y - X θ*) = 0  ===>  X^T X θ* = X^T y

         ^ y (Actual target vector in ℝ^m)
         |  .
         |    .
         |      .  Residual vector e = y - X θ* (Orthogonal to subspace)
         |        .
         +----------> X θ* = y_hat (Orthogonal projection onto Col(X))
       /             /
      /             /  Subspace spanned by columns of design matrix X
     +-------------+

Gradient Descent Optimization#

While the Normal Equation provides an exact analytical solution, computing the matrix inversion (X^T X)^(-1) requires O(d^3) floating-point operations. When the feature dimension d is large (e.g., d > 10,000 in genomic or natural language embeddings), direct inversion becomes computationally prohibitive and memory-exhausting. Iterative first-order optimization algorithms—predominantly Gradient Descent—solve for θ* with O(m · d) per iteration.

1. Batch Gradient Descent (BGD)#

Batch Gradient Descent updates the parameter vector iteratively in the direction of steepest descent, defined by the negative gradient:

θ := θ - α · ∇_θ J(θ)

Substituting the gradient of the MSE cost function:

θ := θ - α · (1 / m) · X^T (X θ - y)

For each individual coordinate j ∈ {0, 1, ..., d}:

θ_j := θ_j - α · (1 / m) · ∑ [i = 1 to m] (h_θ(x^(i)) - y^(i)) · x_j^(i)

where α > 0 represents the learning rate (step size).

• Convergence Conditions: If J(θ) has an L-Lipschitz continuous gradient (||∇J(u) - ∇J(v)|| ≤ L ||u - v||), Batch Gradient Descent is guaranteed to converge monotonically to the global minimum provided 0 < α < 2 / L.
• Limitations: Evaluating ∇J(θ) requires summing over all m training instances at every single iteration. If m = 10,000,000, a single parameter update requires an entire pass through 10 million data points.

2. Stochastic Gradient Descent (SGD)#

Instead of waiting to process all m instances, Stochastic Gradient Descent approximates the full gradient using a single randomly sampled observation (x^(i), y^(i)):

θ := θ - α · (h_θ(x^(i)) - y^(i)) · x^(i)

• Advantages: Immediate updates; O(d) per step; capable of online streaming learning; stochastic noise helps escape shallow local minima in non-convex landscapes.
• Disadvantages: The gradient estimate is noisy and high-variance; parameters oscillate around the optimum rather than converging smoothly, requiring learning rate decay schedules (e.g., α_t = α_0 / (1 + γ · t)).

3. Mini-Batch Gradient Descent#

Mini-batch Gradient Descent achieves the optimal engineering compromise by computing the gradient over small, vectorized subsets of size B (typically B ∈ {32, 64, 128, 256}):

θ := θ - α · (1 / B) · ∑ [i ∈ Batch] (h_θ(x^(i)) - y^(i)) · x^(i)

This exploits SIMD and GPU matrix multiplication parallelization while maintaining low gradient variance.

Feature Scaling: Standardization vs. Normalization#

Gradient descent convergence rates are heavily influenced by the condition number of the Hessian matrix H = ∇^2 J(θ) = (1 / m) X^T X. When features exist on vastly disparate numerical scales (e.g., house size in sq ft: [500, 5000] vs. bedrooms: [1, 5]), the cost contours become highly elongated, eccentric ellipses. The gradient vector points almost perpendicular to the direction of the minimum, causing violent oscillations and requiring very small learning rates.

Unscaled Features (Elongated Contours):         Feature Standardized (Spherical Contours):
      θ_2 ^                                            θ_2 ^
          |  ( / / / / )                                   |    ( ( o ) )
          | ( / / o / / )  <-- Oscillates                  |       o     <-- Direct descent
          +--------------> θ_1                             +--------------> θ_1

Scaling Techniques#

1. Z-Score Standardization:

Transforms each feature to have zero empirical mean and unit variance:

```

μ_j = (1 / m) · ∑ [i = 1 to m] x_j^(i)

σ_j = √[ (1 / m) · ∑ [i = 1 to m] (x_j^(i) - μ_j)^2 ]

x_scaled = (x_j - μ_j) / σ_j

```

1. Min-Max Normalization:

Rescales features into a bounded interval, typically [0, 1]:

```

x_scaled = (x_j - x_min) / (x_max - x_min)

```

The Bias-Variance Decomposition#

To understand why simple models underperform and complex models overfit, we analyze the expected generalization error through the formal Bias-Variance Decomposition.

Assume the true data-generating process is:

y = f(x) + ε

where f(x) = E[y | x] is the true regression function, and ε is zero-mean irreducible observation noise with variance Var(ε) = σ^2:

E[ε] = 0,   E[ε^2] = σ^2,   Cov(f(x), ε) = 0

Let h_S(x) denote the hypothesis trained on a specific training set S sampled from D. The expected prediction error at a fixed test point x over all possible realizations of the training set S decomposes into three mutually orthogonal terms:

E_S,ε [ (y - h_S(x))^2 ] = Bias[h(x)]^2 + Var_S[h(x)] + σ^2

Derivation Breakdown#

1. Add and subtract E_S[h_S(x)] inside the error term:

```

y - h_S(x) = (f(x) + ε) - h_S(x) = (f(x) - E_S[h_S(x)]) + (E_S[h_S(x)] - h_S(x)) + ε

```

1. Squaring and taking expectations eliminates cross-terms because E[ε] = 0 and training sets S are independent of noise ε at test time:

```

E_S,ε [ (y - h_S(x))^2 ] = (f(x) - E_S[h_S(x)])^2 + E_S [ (h_S(x) - E_S[h_S(x)])^2 ] + E[ε^2]

```

1. Definitions:

Bias^2*: (f(x) - E_S[h_S(x)])^2 - Measures the structural inability of the hypothesis class to capture the true underlying function (Underfitting). High bias models (e.g., linear models fitting quadratic data) have rigid assumptions.

Variance*: E_S [ (h_S(x) - E_S[h_S(x)])^2 ] - Measures the sensitivity of the learned hypothesis to fluctuations in the training dataset S (Overfitting). High variance models (e.g., high-degree polynomials) fit noise in S, yielding erratic predictions.

Irreducible Error*: σ^2 - The inherent noise floor of the physical measurement process, impossible to eliminate regardless of model complexity.

Total Error ^
            |      \                          /  Total Error = Bias^2 + Variance + σ^2
            |       \       Optimal          /
            |        \     Tradeoff         /  Variance
            |   Bias^2\       v            /
            |          \       .          /
            |           __   .   _______/
            |              ____./
            | - - - - - - - - - - - - - - - - Irreducible Error (σ^2)
            +---------------------------------------------> Model Complexity
                (Underfitting)             (Overfitting)

Regularization: Ridge (L2), Lasso (L1), and Elastic Net#

When feature dimension d is large or features exhibit multicollinearity, unconstrained Empirical Risk Minimization yields parameter vectors θ with massive magnitudes. Small perturbations in input x produce wild swings in y_hat. Regularization penalizes hypothesis complexity by augmenting the objective cost with a parameter norm penalty.

1. Ridge Regression (L2 / Tikhonov Regularization)#

Ridge regression adds the squared L2 Euclidean norm of the weight vector (excluding the intercept θ_0) to the cost function:

J_Ridge(θ) = (1 / (2m)) · ||X θ - y||^2 + (λ / (2m)) · ||θ||_2^2
           = (1 / (2m)) · [ (X θ - y)^T (X θ - y) + λ · θ^T θ ]

where λ ≥ 0 is the regularization hyperparameter controlling the penalty strength.

Analytical Derivation of the Ridge Estimator#

Differentiating J_Ridge(θ) with respect to θ:

∇_θ J_Ridge(θ) = (1 / m) · [ X^T (X θ - y) + λ · θ ]
               = (1 / m) · [ (X^T X + λ I) θ - X^T y ]

Setting the gradient to zero:

(X^T X + λ I) θ = X^T y
θ_Ridge = (X^T X + λ I)^(-1) X^T y

• Invertibility Guarantee: For any λ > 0, the matrix (X^T X + λ I) is strictly positive definite and guaranteed to be invertible, even if X^T X is singular (e.g., when m < d).
• Shrinkage Mechanics: Ridge shrinks all parameter coefficients proportionally toward zero, but never forces coefficients to be exactly zero (unless λ = ∞).

2. Lasso Regression (L1 / Sparse Regularization)#

Lasso (Least Absolute Shrinkage and Selection Operator) penalizes the L1 norm of the parameter vector:

J_Lasso(θ) = (1 / (2m)) · ||X θ - y||^2 + (λ / m) · ||θ||_1
           = (1 / (2m)) · ∑ [i = 1 to m] (h_θ(x^(i)) - y^(i))^2 + (λ / m) · ∑ [j = 1 to d] |θ_j|

Because the absolute value function |θ_j| is non-differentiable at θ_j = 0, Lasso lacks a closed-form matrix solution and must be optimized via coordinate descent or subgradient methods.

Why Lasso Induces Sparsity: The Geometric Perspective#

Formulating regularization as a constrained optimization problem (via Karush-Kuhn-Tucker duality):

• Minimize: ||X θ - y||^2
• Subject to: ||θ||_1 ≤ t (for Lasso) or ||θ||_2^2 ≤ t (for Ridge)

        Lasso (L1 Constraint: Diamond)             Ridge (L2 Constraint: Circle)
                  θ_2 ^                                      θ_2 ^
                      |  /                                       |  /
                    / | /                                        | /
              <---( * )---> θ_1                            +---( * )---+ θ_1
                   /  | \                                   \   /|   /
                  /   |  \                                   _/ | _/

• The L1 constraint region |θ_1| + |θ_2| ≤ t forms a diamond with sharp vertices situated directly on the coordinate axes (θ_1 = 0 or θ_2 = 0).
• As the elliptical quadratic contours of the OLS loss expand from the unconstrained minimum, they intersect the diamond boundary at a sharp corner with high mathematical probability. At a corner, one or more parameters are driven to identically zero.
• Thus, Lasso performs automated feature selection, eliminating irrelevant features entirely.

3. Elastic Net#

Elastic Net combines both L1 and L2 penalties, resolving Lasso's limitation when dealing with groups of highly correlated features (where Lasso arbitrarily selects one and discards the rest):

J_Elastic(θ) = (1 / (2m)) · ||X θ - y||^2 + (λ / m) · [ α · ||θ||_1 + ((1 - α) / 2) · ||θ||_2^2 ]

where α ∈ [0, 1] balances L1 sparsity against L2 grouped shrinkage.

Evaluation Metrics for Regression#

Evaluating regression models requires quantifying prediction discrepancies across test sets:

1. Mean Absolute Error (MAE):

```

MAE = (1 / m) · ∑ [i = 1 to m] |y^(i) - y_hat^(i)|

```

Robust to outliers; measures average linear magnitude of errors.

1. Mean Squared Error (MSE):

```

MSE = (1 / m) · ∑ [i = 1 to m] (y^(i) - y_hat^(i))^2

```

Penalizes large errors quadratically.

1. Root Mean Squared Error (RMSE):

```

RMSE = √[ (1 / m) · ∑ [i = 1 to m] (y^(i) - y_hat^(i))^2 ]

```

Restores metric to original units of target variable y.

1. Coefficient of Determination (R^2 Score):

```

SS_res = ∑ [i = 1 to m] (y^(i) - y_hat^(i))^2

SS_tot = ∑ [i = 1 to m] (y^(i) - y_mean)^2

R^2 = 1 - (SS_res / SS_tot)

```

Measures the proportion of variance in y explained by features X:

* R^2 = 1.0: Perfect predictions (SS_res = 0).

* R^2 = 0.0: Model performs no better than predicting the mean y_mean for all inputs.

* R^2 < 0.0: Model performs worse than the simple horizontal line y_mean.

Checkpoint Practice Problems#

Problem 1 (Closed-Form Normal Equation Calculation): Given a small training dataset with m = 3 observations and d = 1 feature:

• Observation 1: x^(1) = 1, y^(1) = 2
• Observation 2: x^(2) = 2, y^(2) = 5
• Observation 3: x^(3) = 3, y^(3) = 7

Using the augmented design matrix X (including the intercept column of 1s), compute:

1. The matrix product X^T X and vector X^T y.
2. The exact inverse matrix (X^T X)^(-1).
3. The optimal parameter vector θ* = [θ_0, θ_1]^T.
4. The predicted target y_hat^(4) for a new input x^(4) = 4.

Solution:

1. Construct design matrix X and vector y:

```

X = [ [1, 1],

[1, 2],

[1, 3] ]

y = [ 2, 5, 7 ]^T

```

1. Compute X^T X:

```

X^T X = [ [1+1+1, 1+2+3],

[1+2+3, 1^2 + 2^2 + 3^2] ]

= [ [3, 6],

[6, 14] ]

```

1. Compute X^T y:

```

X^T y = [ 2 + 5 + 7,

1(2) + 2(5) + 3(7) ]

= [ 14,

2 + 10 + 21 ]

= [ 14, 33 ]^T

```

1. Compute (X^T X)^(-1):

Determinant det = (3)(14) - (6)(6) = 42 - 36 = 6.

```

(X^T X)^(-1) = (1 / 6) · [ [14, -6],

[-6, 3] ]

= [ [7/3, -1],

[-1, 1/2] ]

```

1. Multiply to find θ*:

```

θ_0 = (7/3)(14) + (-1)(33) = (98 / 3) - (99 / 3) = -1/3 ≈ -0.3333

θ_1 = (-1)(14) + (1/2)(33) = -14 + 16.5 = 2.5 = 5/2

θ* = [ -1/3, 2.5 ]^T

```

1. Prediction for x = 4:

```

y_hat^(4) = θ_0 + θ_1 · (4) = -1/3 + 2.5(4) = -0.3333 + 10 = 9.6667 (or 29/3)

```

Problem 2 (Single-Step Batch Gradient Descent Update): For the linear hypothesis h_θ(x) = θ_0 + θ_1 · x, suppose current parameters are θ = [0, 0]^T. The learning rate is α = 0.1. Using the same dataset from Problem 1:

• (x^(1), y^(1)) = (1, 2)
• (x^(2), y^(2)) = (2, 5)
• (x^(3), y^(3)) = (3, 7)

Compute:

1. The prediction error e^(i) = h_θ(x^(i)) - y^(i) for all three samples.
2. The partial derivatives ∂J / ∂θ_0 and ∂J / ∂θ_1.
3. The updated parameter vector θ_new after one gradient descent step.

Solution:

1. Compute predictions and errors with θ = [0, 0]^T:

For all i, h_θ(x^(i)) = 0 + 0(x^(i)) = 0.

```

e^(1) = 0 - 2 = -2

e^(2) = 0 - 5 = -5

e^(3) = 0 - 7 = -7

```

1. Compute gradient components:

```

∂J / ∂θ_0 = (1 / m) · ∑ [i = 1 to 3] e^(i)

= (1 / 3) · (-2 - 5 - 7) = -14 / 3 ≈ -4.6667

∂J / ∂θ_1 = (1 / m) · ∑ [i = 1 to 3] (e^(i) · x^(i))

= (1 / 3) · [ (-2)(1) + (-5)(2) + (-7)(3) ]

= (1 / 3) · [ -2 - 10 - 21 ] = -33 / 3 = -11.0

```

1. Update parameters with α = 0.1:

```

θ_0_new = θ_0 - α · (∂J / ∂θ_0) = 0 - 0.1 · (-4.6667) = +0.4667

θ_1_new = θ_1 - α · (∂J / ∂θ_1) = 0 - 0.1 · (-11.0) = +1.1000

θ_new = [ 0.4667, 1.1000 ]^T

```

Problem 3 (Ridge Regression Shrinkage Factor): In a centered orthogonal design where X^T X = m · I (features are uncorrelated and have sample variance 1), prove that the Ridge regression parameter vector θ_Ridge is a scalar shrinkage of the unconstrained OLS parameter vector θ_OLS:

θ_Ridge = γ · θ_OLS

Find the exact formula for shrinkage factor γ in terms of sample size m and regularization parameter λ. If m = 100 and λ = 25, compute the percentage shrinkage applied to the coefficients.

Solution:

1. Recall the estimators:

```

θ_OLS = (X^T X)^(-1) X^T y

θ_Ridge = (X^T X + λ I)^(-1) X^T y

```

1. Substitute X^T X = m · I:

```

θ_OLS = (m · I)^(-1) X^T y = (1 / m) · X^T y ===> X^T y = m · θ_OLS

```

Now substitute into Ridge:

```

X^T X + λ I = m · I + λ · I = (m + λ) · I

(X^T X + λ I)^(-1) = (1 / (m + λ)) · I

```

1. Express θ_Ridge in terms of θ_OLS:

```

θ_Ridge = (1 / (m + λ)) · I · (m · θ_OLS)

= (m / (m + λ)) · θ_OLS

```

Thus, γ = m / (m + λ).

1. Evaluate with m = 100 and λ = 25:

```

γ = 100 / (100 + 25) = 100 / 125 = 0.80

```

Each parameter coefficient is shrunk to 80% of its OLS magnitude. The percentage shrinkage is:

```

Shrinkage = (1 - 0.80) · 100% = 20.0%

```

Problem 4 (Comprehensive Regression Metrics Evaluation): A test dataset contains N = 5 observations with true targets y and model predictions y_hat:

• Instance 1: y = 10, y_hat = 12
• Instance 2: y = 20, y_hat = 19
• Instance 3: y = 30, y_hat = 32
• Instance 4: y = 40, y_hat = 37
• Instance 5: y = 50, y_hat = 52

Calculate:

1. The Mean Absolute Error (MAE).
2. The Mean Squared Error (MSE) and Root Mean Squared Error (RMSE).
3. The sample mean y_mean and total sum of squares SS_tot.
4. The Coefficient of Determination (R^2 Score).

Solution:

1. Residuals e_i = y_i - y_hat_i:

* e_1 = 10 - 12 = -2

* e_2 = 20 - 19 = +1

* e_3 = 30 - 32 = -2

* e_4 = 40 - 37 = +3

* e_5 = 50 - 52 = -2

1. Compute MAE:

```

MAE = (1 / 5) · [ |-2| + |1| + |-2| + |3| + |-2| ]

= (1 / 5) · [ 2 + 1 + 2 + 3 + 2 ] = 10 / 5 = 2.0

```

1. Compute MSE and RMSE:

```

SS_res = (-2)^2 + (1)^2 + (-2)^2 + (3)^2 + (-2)^2

= 4 + 1 + 4 + 9 + 4 = 22

MSE = SS_res / 5 = 22 / 5 = 4.4

RMSE = √(4.4) ≈ 2.0976

```

1. Compute SS_tot:

Target mean:

```

y_mean = (10 + 20 + 30 + 40 + 50) / 5 = 150 / 5 = 30

```

Deviations from mean:

* (10 - 30)^2 = (-20)^2 = 400

* (20 - 30)^2 = (-10)^2 = 100

* (30 - 30)^2 = 0^2 = 0

* (40 - 30)^2 = 10^2 = 100

* (50 - 30)^2 = 20^2 = 400

```

SS_tot = 400 + 100 + 0 + 100 + 400 = 1000

```

1. Compute R^2:

```

R^2 = 1 - (SS_res / SS_tot) = 1 - (22 / 1000) = 1 - 0.022 = 0.978

```

The model explains 97.8% of the variance in the target variable with an RMSE of 2.10.