Classification Algorithms & Model Evaluation

While regression models predict continuous quantities, classification algorithms resolve the problem of categorical decision making: partitioning feature space into discrete class regions or estimating class posterior probabilities. From automated medical diagnostic screenings and fraud detection to spam filtering and autonomous object recognition, classification models form the operational core of applied machine learning systems.

This comprehensive guide delivers a rigorous, university-grade examination of classification mathematics and theory. We explore generative versus discriminative models, derive Maximum Likelihood Estimation for logistic and softmax regression, formalize Support Vector Machine (SVM) margin maximization and kernel duality, analyze recursive decision tree partitioning, and construct formal evaluation metrics including ROC-AUC and Precision-Recall dynamics.

Classification Foundations & The Bayes Optimal Classifier#

Let the input feature space be X ⊆ ℝ^d, and let the label space be a finite discrete set of classes:

• Binary Classification: Y = {0, 1} (or conventionally Y = {-1, +1} in margin-based methods like SVMs).
• Multi-Class Classification: Y = {1, 2, ..., K} where K ≥ 3 mutually exclusive classes exist.

We assume training observations S = {(x^(1), y^(1)), ..., (x^(m), y^(m))} are sampled i.i.d. from a joint probability distribution P(X, Y).

The Bayes Optimal Classifier#

Under 0-1 loss (where loss is 0 for correct classification and 1 for misclassification):

L(y_hat, y) = 1  if y_hat ≠ y,   0  if y_hat = y

The theoretically optimal prediction that minimizes expected risk is the Bayes Optimal Classifier, which assigns an observation x to the class with the maximum posterior probability:

h_Bayes(x) = argmax [k ∈ Y] P(Y = k | X = x)

The error rate achieved by this classifier—termed the Bayes Error Rate—represents the minimum achievable error floor, caused by irreducible overlap in class-conditional feature distributions. No machine learning algorithm can outperform the Bayes Error Rate on the same feature representation.

Generative vs. Discriminative Classifiers#

Machine learning algorithms approximate the Bayes optimal decision boundary via two distinct philosophical paradigms:

+---------------------------------------------------------------------------------------+
|                        Generative vs. Discriminative Paradigms                        |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. Discriminative Models (Direct Mapping / Boundary Estimation)                     |
|      Input x -------------------------> Model estimates P(Y | X) directly            |
|      Algorithms: Logistic Regression, Support Vector Machines, Neural Networks       |
|                                                                                       |
|   2. Generative Models (Joint Distribution Modeling via Bayes' Theorem)               |
|      Estimates class prior P(Y) & class likelihood P(X | Y)                           |
|      Computes posterior: P(Y=k | X=x) = [ P(X=x | Y=k) · P(Y=k) ] / P(X=x)            |
|      Algorithms: Naive Bayes, Linear Discriminant Analysis (LDA), Gaussian Mixtures   |
|                                                                                       |
+---------------------------------------------------------------------------------------+

• Discriminative models focus mathematical capacity strictly on modeling the decision boundary between classes, often achieving superior asymptotic accuracy when large training corpora are available.
• Generative models model how data points are generated within each class; they handle missing data naturally and converge to their asymptotic error with fewer training instances.

Logistic Regression & Maximum Likelihood Estimation#

Despite its historical name, Logistic Regression is a foundational linear discriminative classification algorithm. Rather than predicting raw class labels directly, it models the posterior probability P(Y = 1 | X = x) as a continuous, differentiable function bounded in the open interval (0, 1).

The Sigmoid (Logistic) Function#

To map an unconstrained real-valued linear score z = θ^T x ∈ (-∞, +∞) into a valid probability p ∈ (0, 1), we apply the standard logistic sigmoid activation function σ(z):

p = σ(z) = 1 / (1 + e^(-z)) = e^z / (1 + e^z)

The sigmoid function exhibits critical mathematical properties:

1. Symmetry: σ(-z) = 1 - σ(z)
2. Derivative: The first derivative can be expressed entirely in terms of the function value itself:

```

dσ/dz = (d/dz) (1 + e^(-z))^(-1)

= - (1 + e^(-z))^(-2) · (-e^(-z))

= [ 1 / (1 + e^(-z)) ] · [ e^(-z) / (1 + e^(-z)) ]

= σ(z) · (1 - σ(z))

```

Odds Ratio & The Logit Link Function#

The odds of an event occurring is defined as the ratio of probability of occurrence to probability of non-occurrence:

Odds = P(Y = 1 | x) / P(Y = 0 | x) = p / (1 - p)

Substituting the sigmoid formulation for p:

p / (1 - p) = [ 1 / (1 + e^(-θ^T x)) ] / [ e^(-θ^T x) / (1 + e^(-θ^T x)) ] = 1 / e^(-θ^T x) = e^(θ^T x)

Taking the natural logarithm yields the logit (log-odds):

logit(p) = ln( p / (1 - p) ) = θ^T x = θ_0 + θ_1 · x_1 + ... + θ_d · x_d

Thus, Logistic Regression assumes that the log-odds of the positive class is a strictly linear function of the input features. The decision boundary separating class 1 from class 0 is defined by the hyperparameter plane where p = 0.5 (log-odds = 0):

θ^T x = 0

Maximum Likelihood Estimation (MLE) & Binary Cross-Entropy#

Because target labels y^(i) ∈ {0, 1} are binary, each observation is modeled as a Bernoulli random variable with parameter p^(i) = h_θ(x^(i)) = σ(θ^T x^(i)):

P(y^(i) | x^(i); θ) = (h_θ(x^(i)))^(y^(i)) · (1 - h_θ(x^(i)))^(1 - y^(i))

Assuming i.i.d. training samples, the joint Likelihood L(θ) across the dataset S is:

L(θ) = ∏ [i = 1 to m] (h_θ(x^(i)))^(y^(i)) · (1 - h_θ(x^(i)))^(1 - y^(i))

To facilitate numerical stability and continuous differentiation, we take the natural logarithm to obtain the Log-Likelihood ℓ(θ):

ℓ(θ) = ln(L(θ)) = ∑ [i = 1 to m] [ y^(i) · ln(h_θ(x^(i))) + (1 - y^(i)) · ln(1 - h_θ(x^(i))) ]

Maximizing log-likelihood is mathematically equivalent to minimizing the negative log-likelihood scaled by 1 / m, which defines the Binary Cross-Entropy (Log Loss) cost function J(θ):

J(θ) = - (1 / m) · ∑ [i = 1 to m] [ y^(i) · ln(h_θ(x^(i))) + (1 - y^(i)) · ln(1 - h_θ(x^(i))) ]

Gradient Vector Derivation#

Differentiating J(θ) with respect to parameter weight θ_j using the chain rule:

∂J/∂θ_j = - (1 / m) · ∑ [i = 1 to m] [ (y^(i) / h_θ(x^(i))) · (∂h_θ/∂θ_j) - ((1 - y^(i)) / (1 - h_θ(x^(i)))) · (∂h_θ/∂θ_j) ]

Since h_θ(x) = σ(θ^T x), its derivative with respect to θ_j is:

∂h_θ/∂θ_j = σ(θ^T x) · (1 - σ(θ^T x)) · x_j = h_θ(x) · (1 - h_θ(x)) · x_j

Substituting this derivative into the summation:

∂J/∂θ_j = - (1 / m) · ∑ [i = 1 to m] [ y^(i) · (1 - h_θ(x^(i))) - (1 - y^(i)) · h_θ(x^(i)) ] · x_j^(i)
        = - (1 / m) · ∑ [i = 1 to m] [ y^(i) - y^(i)·h_θ - h_θ + y^(i)·h_θ ] · x_j^(i)
        = (1 / m) · ∑ [i = 1 to m] (h_θ(x^(i)) - y^(i)) · x_j^(i)

In vectorized matrix notation:

∇_θ J(θ) = (1 / m) · X^T (h_θ(X) - y)

Remarkably, the gradient of the Binary Cross-Entropy loss for Logistic Regression exhibits the exact same algebraic functional form as Ordinary Least Squares linear regression. However, unlike OLS, h_θ(x) is non-linear (the sigmoid function), which precludes a closed-form analytical solution. Optimization must proceed via iterative algorithms: Gradient Descent, Newton-Raphson, or L-BFGS.

Support Vector Machines (SVM) & The Maximum Margin Hyperplane#

Support Vector Machines approach classification through structural risk minimization: finding a separating hyperplane that maximizes the geometric distance (margin) to the nearest training instances of either class.

Consider a linearly separable dataset with labels y^(i) ∈ {-1, +1}. A separating hyperplane is parametrized by normal vector w ∈ ℝ^d and intercept scalar b ∈ ℝ:

w^T x + b = 0

               x_2 ^
                   |         + (Positive Class y = +1)
                   |       +   +
                   |     +   [+] <--- Support Vector (w^T x + b = +1)
                   |  - - - - - - - - - - - - - - -
                   |         /     ^
                   |        /      | Geometric Margin γ = 2 / ||w||
                   |       /       v
                   |  - - - - - - - - - - - - - - -
                   |   [-] <--- Support Vector (w^T x + b = -1)
                   |     -   -
                   |       -   (Negative Class y = -1)
                   +----------------------------------> x_1

Functional vs. Geometric Margin#

• Functional Margin: For a single instance (x^(i), y^(i)), the functional margin is γ_hat^(i) = y^(i) · (w^T x^(i) + b). If y^(i) = +1, we require w^T x^(i) + b > 0; if y^(i) = -1, we require w^T x^(i) + b < 0. Thus, a correct classification always yields γ_hat^(i) > 0.
• Geometric Margin: The actual Euclidean distance from point x^(i) to the hyperplane is:

```

γ^(i) = y^(i) · [ (w^T x^(i) + b) / ||w|| ] = γ_hat^(i) / ||w||

```

To construct a unique, canonical scale, we fix the functional margin of the closest points to 1:

min [i = 1 to m] y^(i) · (w^T x^(i) + b) = 1

The total geometric margin between the two bounding hyperplanes (w^T x + b = +1 and w^T x + b = -1) is:

Margin = 2 / ||w||

The Primal Optimization Problem#

Maximizing the margin 2 / ||w|| is mathematically equivalent to minimizing (1/2) ||w||^2. The Hard-Margin SVM optimization problem is formulated as a constrained quadratic program:

Minimize:   (1 / 2) · ||w||^2
Subject to: y^(i) · (w^T x^(i) + b) ≥ 1    for all i = 1, ..., m

Soft-Margin SVM & Slack Variables#

Real-world datasets are rarely perfectly linearly separable; noise, measurement error, or intrinsic class overlap make strict linear separation impossible. To handle non-separable data, Cortes and Vapnik (1995) introduced non-negative slack variables ξ_i ≥ 0, which quantify the margin violation of instance x^(i):

Minimize:   (1 / 2) · ||w||^2 + C · ∑ [i = 1 to m] ξ_i
Subject to: y^(i) · (w^T x^(i) + b) ≥ 1 - ξ_i,    ξ_i ≥ 0    for all i = 1, ..., m

• The C Hyperparameter: Controls the trade-off between maximizing the margin (small ||w||) and penalizing training classification errors:

Large C*: Imposes heavy penalties on margin violations. The optimizer prioritizes zero training errors, producing narrow margins that risk overfitting.

Small C*: Tolerates more misclassifications. The optimizer prioritizes finding a wide margin, yielding greater regularization and generalization robustness.

The Dual Formulation & The Kernel Trick#

Formulating the Lagrangian of the primal problem with multipliers α_i ≥ 0 and r_i ≥ 0, setting partial derivatives with respect to primal variables (w, b, ξ) to zero, and substituting back yields the Wolfe Dual Formulation:

Maximize:   ∑ [i = 1 to m] α_i - (1 / 2) · ∑ [i = 1 to m] ∑ [j = 1 to m] α_i · α_j · y^(i) · y^(j) · (x^(i)^T x^(j))
Subject to: 0 ≤ α_i ≤ C    for all i = 1, ..., m
            ∑ [i = 1 to m] α_i · y^(i) = 0

By the Karush-Kuhn-Tucker (KKT) complementary slackness conditions, α_i · [y^(i)(w^T x^(i) + b) - 1 + ξ_i] = 0.

• For all points lying strictly outside the margin, α_i = 0. These points exert zero influence on the final decision boundary.
• Points with α_i > 0 are the Support Vectors; the optimal weight vector w* is a linear combination exclusively of support vectors:

```

w* = ∑ [i ∈ Support Vectors] α_i · y^(i) · x^(i)

```

The Kernel Trick (Mercer's Theorem)#

Notice that the dual optimization problem depends on the data only through inner products x^(i)^T x^(j). If data is not linearly separable in ℝ^d, we can project features into a higher-dimensional (potentially infinite-dimensional) Hilbert space H via a mapping function Φ(x).

Computing Φ(x)^T Φ(z) directly in high dimensions is computationally impossible. A Kernel Function K(x, z) computes this inner product directly in low-dimensional input space without ever explicitly evaluating the high-dimensional coordinates:

K(x, z) = ⟨Φ(x), Φ(z)⟩_H

Common standard kernels:

1. Linear Kernel: K(x, z) = x^T z
2. Polynomial Kernel: K(x, z) = (x^T z + c)^d
3. Radial Basis Function (RBF / Gaussian) Kernel:

```

K(x, z) = exp( - γ · ||x - z||^2 )

```

The RBF kernel corresponds to an inner product in an infinite-dimensional feature space, allowing SVMs to synthesize highly complex, smooth non-linear decision boundaries.

Decision Trees & Ensemble Architectures#

Decision trees partition feature space into a set of axis-aligned hyper-rectangles using recursive greedy binary splitting.

Impurity Functions & Split Criteria#

At any internal node containing dataset S, the tree evaluates all possible features j and split thresholds t to maximize the reduction in impurity, termed Information Gain (IG):

IG(S, j, t) = Impurity(S) - [ (|S_left| / |S|) · Impurity(S_left) + (|S_right| / |S|) · Impurity(S_right) ]

Let p_k denote the empirical proportion of class k instances in node S:

1. Shannon Entropy:

```

H(S) = - ∑ [k = 1 to K] p_k · log2(p_k)

```

Measures information disorder; maximizes at uniform distribution (H = log2(K)) and vanishes when a node is pure (p_k = 1 for a single class).

1. Gini Impurity:

```

G(S) = 1 - ∑ [k = 1 to K] p_k^2 = ∑ [k = 1 to K] p_k · (1 - p_k)

```

Measures the probability that a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the class distribution. Gini impurity is computationally faster than entropy because it avoids logarithmic evaluations.

Ensemble Architectures: Random Forests & Boosting#

Single decision trees suffer from notoriously high variance: small perturbations in the training data produce radically different tree structures. Ensemble methods aggregate multiple base learners to reduce variance or bias:

• Bootstrap Aggregation (Random Forests): Trains T deep, unpruned decision trees in parallel. Each tree is trained on a distinct bootstrap sample S* (sampled uniformly with replacement from S). To decorrelate trees, at each split node, the algorithm considers only a random subset of m_try features (typically m_try = √d). Predictions are aggregated via majority voting:

```

Var(Forest) = ρ · σ^2 + ((1 - ρ) / T) · σ^2

```

As T → ∞, the second term vanishes, reducing total model variance proportionally to the decorrelation coefficient ρ.

• Gradient Boosting (GBDT / XGBoost / LightGBM): Trains shallow, high-bias decision trees sequentially. Each subsequent tree h_t(x) is fit to the negative gradient (pseudo-residuals) of the loss function with respect to the ensemble's previous predictions:

```

r_i^(t) = - [ ∂L(y^(i), y_hat^(i)) / ∂y_hat^(i) ]

```

Boosting systematically drives down bias while maintaining low variance through shrinkage learning rates.

Classification Evaluation Metrics & ROC Analysis#

Evaluating classification systems using raw accuracy (Accuracy = (TP + TN) / (TP + TN + FP + FN)) is profoundly deceptive in class-imbalanced contexts (e.g., fraud detection where only 0.1% of transactions are fraudulent; a naive model predicting 0 for all instances achieves 99.9% accuracy while detecting zero fraud).

The Confusion Matrix#

For binary classification:

                       Actual Class Positive (1)    Actual Class Negative (0)
Predicted Positive (1)    True Positive (TP)           False Positive (FP) (Type I Error)
Predicted Negative (0)    False Negative (FN)          True Negative (TN)  (Type II Error)
                          (Type II Error)

Derived Performance Metrics#

1. Precision (Positive Predictive Value):

```

Precision = TP / (TP + FP)

```

Answers: "Of all instances the model labeled as positive, what proportion was actually positive?" Crucial when false positives carry high cost (e.g., spam filtering).

1. Recall (Sensitivity / True Positive Rate):

```

Recall (TPR) = TP / (TP + FN)

```

Answers: "Of all actual positive instances, what proportion did the model correctly identify?" Crucial when false negatives carry severe consequences (e.g., tumor detection).

1. Specificity (True Negative Rate):

```

Specificity (TNR) = TN / (TN + FP)

```

1. False Positive Rate (Fall-out):

```

FPR = 1 - Specificity = FP / (TN + FP)

```

1. F1-Score (Harmonic Mean):

```

F1 = 2 · [ (Precision · Recall) / (Precision + Recall) ] = 2 · TP / (2 · TP + FP + FN)

```

The harmonic mean heavily penalizes extreme imbalances between Precision and Recall.

1. F_β Score:

```

F_β = (1 + β^2) · [ (Precision · Recall) / (β^2 · Precision + Recall) ]

```

Weights recall β times as heavily as precision (e.g., F_2 score prioritizes recall; F_0.5 score prioritizes precision).

ROC Curves & Precision-Recall Curves#

Most classification models output a continuous probability score p = P(Y = 1 | x). Converting p into a binary decision requires choosing a decision threshold τ ∈ [0, 1] (y_hat = 1 if p ≥ τ else 0).

• Receiver Operating Characteristic (ROC) Curve: Plots True Positive Rate (Recall) on the y-axis against False Positive Rate (1 - Specificity) on the x-axis across all continuous threshold values τ ∈ [0, 1].

* An uninformative random baseline produces a diagonal line from (0,0) to (1,1) with Area Under the Curve (AUC) = 0.50.

* A perfect classifier passes through the top-left coordinate (0, 1) with AUC = 1.00.

• Precision-Recall (PR) Curve: Plots Precision (y-axis) against Recall (x-axis). Unlike ROC curves, which can present an overly optimistic picture on severely skewed datasets because TN dominates the denominator of FPR, PR curves focus strictly on the minority positive class.

Checkpoint Practice Problems#

Problem 1 (Logistic Regression Odds & Posterior Probability Calculation): A binary logistic regression model for loan default prediction has learned weights:

• Intercept: θ_0 = -2.5
• Feature 1 (Debt-to-Income Ratio): θ_1 = +3.0
• Feature 2 (Delinquency Count): θ_2 = +0.8

A loan applicant presents features x_1 = 0.6 and x_2 = 2.

1. Compute the linear logit score z = θ^T x.
2. Compute the odds of loan default.
3. Compute the model's predicted probability of default P(Y = 1 | x).
4. If the decision threshold is τ = 0.35, will this loan be flagged for rejection?

Solution:

1. Calculate the linear combination z:

```

z = θ_0 + θ_1 · x_1 + θ_2 · x_2

z = -2.5 + 3.0(0.6) + 0.8(2)

z = -2.5 + 1.8 + 1.6 = 0.90

```

1. Compute the odds:

```

Odds = e^z = e^(0.90) ≈ 2.4596

```

The odds of default are approximately 2.46 to 1.

1. Compute posterior probability using sigmoid:

```

P(Y = 1 | x) = 1 / (1 + e^(-z)) = 1 / (1 + e^(-0.90))

e^(-0.90) ≈ 0.40657

P(Y = 1 | x) = 1 / (1 + 0.40657) = 1 / 1.40657 ≈ 0.71095 (71.1%)

```

1. Threshold comparison:

Since P(Y = 1 | x) = 0.711 ≥ 0.35, the application is flagged for rejection.

Problem 2 (Soft-Margin SVM Geometric Margin & Slack Evaluation): A 2D linear soft-margin SVM has hyperparameter C = 10. The optimal parameters are found to be:

w = [3.0, 4.0]^T,   b = -5.0

Consider three training points:

• Point A: x^(A) = [2, 1]^T, with ground-truth y^(A) = +1
• Point B: x^(B) = [1, 1]^T, with ground-truth y^(B) = +1
• Point C: x^(C) = [0, 0]^T, with ground-truth y^(C) = -1

For each point:

1. Compute the functional margin γ_hat = y · (w^T x + b).
2. Compute the slack variable ξ (where ξ = max(0, 1 - γ_hat)).
3. State whether the point is correctly classified, whether it lies on the margin boundary, and whether it acts as a Support Vector.

Solution:

First, compute Euclidean norm of w:

||w|| = √(3^2 + 4^2) = √(9 + 16) = √25 = 5.0
The total margin width is 2 / ||w|| = 2 / 5.0 = 0.40.

• Point A (y = +1):

```

w^T x^(A) + b = 3(2) + 4(1) - 5 = 6 + 4 - 5 = +5

γ_hat^(A) = (+1) · (+5) = +5

Slack ξ_A = max(0, 1 - 5) = 0

```

Point A is correctly classified, lies far outside the margin (5 > 1), and is not a support vector (α_A = 0).

• Point B (y = +1):

```

w^T x^(B) + b = 3(1) + 4(1) - 5 = 3 + 4 - 5 = +2

γ_hat^(B) = (+1) · (+2) = +2

Slack ξ_B = max(0, 1 - 2) = 0

```

Point B is correctly classified, lies safely on the correct side of the margin boundary (2 > 1), and is not a support vector.

• Point C (y = -1):

```

w^T x^(C) + b = 3(0) + 4(0) - 5 = -5

γ_hat^(C) = (-1) · (-5) = +5

Slack ξ_C = max(0, 1 - 5) = 0

```

Point C is correctly classified with zero slack violation, lying far on the negative side.

Suppose a fourth point D exists with x^(D) = [1, 0.5]^T and y^(D) = +1:

w^T x^(D) + b = 3(1) + 4(0.5) - 5 = 3 + 2 - 5 = 0.
γ_hat^(D) = (+1) · (0) = 0.
Slack ξ_D = max(0, 1 - 0) = 1.0.

Point D lies directly on the decision boundary (w^T x + b = 0). It has slack ξ_D = 1.0, is a margin violator, and is an active Support Vector.

Problem 3 (Decision Tree Information Gain vs. Gini Impurity): A parent dataset S contains 40 instances: 24 of Class Positive (+) and 16 of Class Negative (-). A candidate split on feature A partitions S into two child nodes:

• Left Child (S_L): 20 instances (18 Positive, 2 Negative)
• Right Child (S_R): 20 instances (6 Positive, 14 Negative)

1. Compute the parent node Gini impurity G(S).
2. Compute the weighted Gini impurity of the split G(S, A) and the resulting Gini Impurity Reduction.
3. Compute the Shannon Entropy of the parent node H(S) in bits.
4. Compute the Information Gain IG(S, A) in bits.

Solution:

1. Parent Gini Impurity:

Class proportions: p_+ = 24 / 40 = 0.60, p_- = 16 / 40 = 0.40.

```

G(S) = 1 - (p_+^2 + p_-^2) = 1 - (0.60^2 + 0.40^2) = 1 - (0.36 + 0.16) = 1 - 0.52 = 0.480

```

1. Child Nodes Gini Impurity:

* Left Child (S_L, n = 20):

p_+ = 18 / 20 = 0.90, p_- = 2 / 20 = 0.10.

```

G(S_L) = 1 - (0.90^2 + 0.10^2) = 1 - (0.81 + 0.01) = 1 - 0.82 = 0.180

```

* Right Child (S_R, n = 20):

p_+ = 6 / 20 = 0.30, p_- = 14 / 20 = 0.70.

```

G(S_R) = 1 - (0.30^2 + 0.70^2) = 1 - (0.09 + 0.49) = 1 - 0.58 = 0.420

```

* Weighted Split Gini:

```

G(S, A) = (20 / 40) · G(S_L) + (20 / 40) · G(S_R)

= 0.50 · 0.180 + 0.50 · 0.420 = 0.090 + 0.210 = 0.300

```

Gini Reduction = G(S) - G(S, A) = 0.480 - 0.300 = 0.180*.

1. Parent Shannon Entropy:

```

H(S) = - [ 0.60 · log2(0.60) + 0.40 · log2(0.40) ]

log2(0.60) ≈ -0.73697, log2(0.40) ≈ -1.32193

H(S) = - [ 0.60(-0.73697) + 0.40(-1.32193) ] = - [ -0.44218 - 0.52877 ] = 0.97095 bits

```

1. Information Gain IG(S, A):

* Left Child Entropy:

```

H(S_L) = - [ 0.90 · log2(0.90) + 0.10 · log2(0.10) ]

log2(0.90) ≈ -0.15200, log2(0.10) ≈ -3.32193

H(S_L) = - [ 0.90(-0.15200) + 0.10(-3.32193) ] = - [ -0.13680 - 0.33219 ] = 0.46899 bits

```

* Right Child Entropy:

```

H(S_R) = - [ 0.30 · log2(0.30) + 0.70 · log2(0.70) ]

log2(0.30) ≈ -1.73697, log2(0.70) ≈ -0.51457

H(S_R) = - [ 0.30(-1.73697) + 0.70(-0.51457) ] = - [ -0.52109 - 0.36020 ] = 0.88129 bits

```

* Weighted Child Entropy:

```

H(S, A) = 0.50 · (0.46899) + 0.50 · (0.88129) = 0.23450 + 0.44065 = 0.67515 bits

```

* Information Gain:

```

IG(S, A) = H(S) - H(S, A) = 0.97095 - 0.67515 = 0.2958 bits

```

Problem 4 (Comprehensive Classification Evaluation Metrics): A cancer diagnostic model evaluates N = 1,000 patient biopsy samples. The resulting confusion matrix is:

• True Positives (TP): 85
• False Positives (FP): 15
• False Negatives (FN): 15
• True Negatives (TN): 885

Calculate:

1. Classification Accuracy and Error Rate.
2. Precision and Recall (Sensitivity).
3. Specificity and False Positive Rate (FPR).
4. The balanced F1-Score and the F_2 Score (which weights recall twice as heavily as precision).

Solution:

1. Accuracy & Error Rate:

```

Accuracy = (TP + TN) / Total = (85 + 885) / 1000 = 970 / 1000 = 0.970 (97.0%)

Error Rate = 1 - Accuracy = 30 / 1000 = 0.030 (3.0%)

```

1. Precision & Recall:

```

Precision = TP / (TP + FP) = 85 / (85 + 15) = 85 / 100 = 0.850 (85.0%)

Recall = TP / (TP + FN) = 85 / (85 + 15) = 85 / 100 = 0.850 (85.0%)

```

1. Specificity & FPR:

```

Specificity = TN / (TN + FP) = 885 / (885 + 15) = 885 / 900 ≈ 0.9833 (98.33%)

FPR = 1 - Specificity = 15 / 900 ≈ 0.0167 (1.67%)

```

1. F-Scores:

Because Precision = Recall = 0.850:

```

F1 = 2 · (0.850 · 0.850) / (0.850 + 0.850) = 0.850 (85.0%)

```

For F_2 score with β = 2:

```

F_2 = (1 + 2^2) · (P · R) / (2^2 · P + R)

= 5 · (0.850 · 0.850) / (4 · 0.850 + 0.850)

= 5 · (0.7225) / (5 · 0.850) = 0.7225 / 0.850 = 0.850 (85.0%)

```

The model achieves 97.0% accuracy, 85.0% precision, 85.0% recall, and an F1-score of 0.850.