Neural Network Fundamentals & Backpropagation

Artificial Neural Networks (ANNs) represent one of the most powerful and transformative computational paradigms in computer science. Drawing initial inspiration from biological neurobiology, modern deep learning formalizes artificial neural networks as parameterized directed acyclic graphs of affine matrix transformations composed with non-linear activation functions. When trained at scale via reverse-mode automatic differentiation (backpropagation), neural networks learn hierarchical representations of data—automatically discovering edge filters, phoneme detectors, syntactic parse trees, and semantic abstractions directly from raw inputs.

This guide provides a rigorous, university-grade mathematical foundation for artificial neural networks. We trace the computational progression from the early Rosenblatt Perceptron to multi-layer architectures, establish the Universal Approximation Theorem, derive the multivariate chain-rule equations of Backpropagation, evaluate modern activation functions, and analyze advanced adaptive optimizers including Adam and RMSProp.

The Artificial Neuron & The Perceptron#

The theoretical origins of artificial neural networks date back to the threshold logic model introduced by Warren McCulloch and Walter Pitts (1943) and Frank Rosenblatt's Perceptron (1958).

The Mathematical Perceptron Model#

A single perceptron takes an input vector x = [x_1, x_2, ..., x_d]^T ∈ ℝ^d, computes an affine linear combination parameterized by weight vector w = [w_1, w_2, ..., w_d]^T ∈ ℝ^d and scalar bias b ∈ ℝ, and applies a non-linear Heavyside step function φ(z):

z = w^T x + b = ∑ [j = 1 to d] w_j · x_j + b

y_hat = φ(z) = { +1  if z ≥ 0,
               { -1  if z < 0

   x_1 ---> [ w_1 ] \
   x_2 ---> [ w_2 ] ---> ( Sum: z = w^T x + b ) ---> [ Activation: φ(z) ] ---> y_hat
   x_d ---> [ w_d ] /
   b   ---> [  1  ] /

The Perceptron Learning Rule#

For linearly separable binary classification with targets y ∈ {-1, +1}, the perceptron learning rule updates weights iteratively when a misclassification occurs:

w := w + η · (y^(i) - y_hat^(i)) · x^(i)
b := b + η · (y^(i) - y_hat^(i))

where η > 0 is the learning rate.

• Novikoff's Perceptron Convergence Theorem (1962): If the training dataset S is linearly separable with geometric margin γ > 0, and all instances satisfy ||x^(i)||_2 ≤ R, the perceptron learning algorithm is mathematically guaranteed to converge to a separating hyperplane in at most k ≤ (R / γ)^2 updates.

The Linear Separability Collapse: The XOR Problem#

In their seminal 1969 monograph Perceptrons, Marvin Minsky and Seymour Papert proved that a single-layer perceptron is fundamentally incapable of solving functions that are not linearly separable. The canonical counterexample is the Exclusive-OR (XOR) boolean logic gate:

x_1    x_2    x_1 XOR x_2
 0      0          0
 0      1          1
 1      0          1
 1      1          0

Plotting the four points on a 2D Cartesian plane reveals that no single linear boundary (w_1 · x_1 + w_2 · x_2 + b = 0) can simultaneously isolate {(0, 1), (1, 0)} from {(0, 0), (1, 1)}. This mathematical limitation halted funding for neural network research for over a decade—an era known as the "AI Winter"—until the widespread formalization of multi-layer architectures trained with non-linear activations and backpropagation.

Multi-Layer Perceptrons (MLP) & Representation Capacity#

A Multi-Layer Perceptron overcomes the linear separability barrier by cascading multiple layers of neurons. Intermediate layers—situated between the external input and the final output—are termed Hidden Layers because their values are not directly observed in the training data.

+---------------------------------------------------------------------------------------+
|                          Multi-Layer Perceptron Architecture                          |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   Input Layer (a^[0] ∈ ℝ^d)    Hidden Layer 1 (a^[1])      Output Layer (a^[2] ∈ ℝ^K) |
|          ( x_1 ) -----------------> ( h_1^[1] ) ---------------> ( y_hat_1 )          |
|                                  /                  \                                 |
|          ( x_2 ) -------------> ( h_2^[1] ) ----------> ( y_hat_2 )                   |
|                                  \                  /                                 |
|          ( x_d ) -----------------> ( h_n^[1] ) ---------------> ( y_hat_K )          |
|                                                                                       |
|   Weights: W^[1] ∈ ℝ^(n1 × d)                Weights: W^[2] ∈ ℝ^(K × n1)              |
|   Biases:  b^[1] ∈ ℝ^(n1 × 1)                Biases:  b^[2] ∈ ℝ^(K × 1)               |
|                                                                                       |
+---------------------------------------------------------------------------------------+

Why Non-Linear Activation Functions Are Mandatory#

Suppose a two-layer neural network uses purely linear activation functions: g(z) = z. The forward propagation equations are:

z^[1] = W^[1] x + b^[1]
a^[1] = z^[1]
z^[2] = W^[2] a^[1] + b^[2]
a^[2] = z^[2] = W^[2] (W^[1] x + b^[1]) + b^[2]
              = (W^[2] W^[1]) x + (W^[2] b^[1] + b^[2])
              = W_effective · x + b_effective

Because the matrix product of two linear transformations W_effective = W^[2] W^[1] is itself simply a linear transformation, an arbitrary N-layer deep linear network collapses mathematically to a single-layer linear model. Non-linear activation functions are strictly required to bend and fold coordinate space, enabling the network to construct complex non-linear decision boundaries.

The Universal Approximation Theorem#

The theoretical justification for deep neural networks is anchored by the Universal Approximation Theorem (proven by George Cybenko in 1989 for sigmoid activations, and generalized by Kurt Hornik in 1991 for arbitrary non-constant, bounded continuous activations):

> Let σ: ℝ → ℝ be any non-constant, bounded, and continuous activation function. Then, for any continuous function f: [0, 1]^d → ℝ and any precision tolerance ε > 0, there exists an integer N and real constants v_i, b_i, and weight vectors w_i such that the single-hidden-layer network:

> ```

> F(x) = ∑ [i = 1 to N] v_i · σ(w_i^T x + b_i)

> ```

> satisfies |F(x) - f(x)| < ε for all x in the compact domain [0, 1]^d.

While the theorem guarantees that a single hidden layer with sufficiently large width N can approximate any function, shallow networks require exponential width N = O(2^d). Modern deep architectures cascade multiple hidden layers of moderate width, achieving hierarchical feature composition with exponentially fewer parameters.

Non-Linear Activation Functions#

The choice of activation function directly dictates gradient propagation dynamics, optimization convergence speed, and susceptibility to numerical saturation.

+--------------------------------------------------------------------------------------------------+
|                                Activation Function Comparison                                    |
+---------------+-----------------------------+-------------------------------+--------------------+
| Activation    | Formula                     | Derivative Range              | Zero-Centered?     |
+---------------+-----------------------------+-------------------------------+--------------------+
| Sigmoid (σ)   | 1 / (1 + e^(-z))            | (0, 0.25]                     | No (Output > 0)    |
| Tanh          | (e^z - e^(-z))/(e^z + e^(-z))| (0, 1.0]                     | Yes [-1, +1]       |
| ReLU          | max(0, z)                   | {0 if z < 0, 1 if z > 0}      | No [0, +∞)         |
| Leaky ReLU    | max(α z, z) where α ≈ 0.01  | {α if z < 0, 1 if z > 0}      | Approximately      |
| GELU          | z · Φ(z)                    | Continuous smooth curve       | Approximately      |
+---------------+-----------------------------+-------------------------------+--------------------+

1. The Sigmoid Function & The Vanishing Gradient Problem#

σ(z) = 1 / (1 + e^(-z)),   σ'(z) = σ(z) · (1 - σ(z))

The maximum value of the sigmoid derivative occurs at z = 0, where σ(0) = 0.5:

σ'(0) = 0.5 · (1 - 0.5) = 0.25

For inputs |z| > 4, σ(z) saturates near 0 or 1, and the derivative σ'(z) approaches 0. When backpropagating through an L-layer network, gradients are multiplied by the local derivatives at each layer:

Gradient ∝ ∏ [l = 1 to L] σ'(z^[l]) ≤ (0.25)^L

For a 10-layer network, (0.25)^10 ≈ 9.5 × 10^(-7). Gradients effectively vanish before reaching the earliest layers, completely halting weight updates. Furthermore, because σ(z) > 0 for all z, all gradients for weights entering a neuron share the same algebraic sign, causing inefficient 'zig-zagging' gradient descent dynamics.

2. Hyperbolic Tangent (Tanh)#

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z)) = 2 · σ(2z) - 1
d/dz tanh(z) = 1 - tanh^2(z)

Tanh is strictly zero-centered, with output range (-1, +1). Its maximum derivative is 1.0 (at z = 0). While it converges faster than sigmoid, it still saturates for large |z|, suffering from vanishing gradients in deep architectures.

3. Rectified Linear Unit (ReLU)#

f(z) = max(0, z) = { z  if z ≥ 0,
                   { 0  if z < 0

Introduced to deep networks by Nair and Hinton (2010), ReLU is the default activation function in modern feedforward and convolutional networks.

• Advantages: Computationally trivial (a simple branch or clamp operation); non-saturating for z > 0 (derivative is strictly 1.0), which completely eliminates vanishing gradients in the positive domain; induces sparse representations (inactive neurons output exactly 0).
• The Dying ReLU Problem: If a large gradient updates weights such that a neuron's pre-activation z = w^T x + b is negative across the entire training dataset, the neuron outputs 0 and its derivative is permanently 0. The neuron becomes mathematically dead and never updates again.
• Mitigations: Leaky ReLU (f(z) = max(0.01 z, z)) and Parametric ReLU (PReLU) assign a tiny non-zero slope to the negative domain, ensuring gradients propagate even when z < 0.

Vectorized Forward Propagation#

Let a neural network possess L layers, where layer l = 0 represents the input, and layer l = L represents the output. Let n_[l] denote the number of neurons in layer l.

• Weight Matrix W^[l] ∈ ℝ^(n_[l] × n_[l-1])
• Bias Vector b^[l] ∈ ℝ^(n_[l] × 1)

For a mini-batch of m training instances, we stack feature vectors column-wise into matrix A^[0] = X^T ∈ ℝ^(n_[0] × m). Forward propagation executes iteratively across layers l = 1, 2, ..., L:

Z^[l] = W^[l] · A^[l-1] + b^[l]    (Linear Pre-Activation, Z^[l] ∈ ℝ^(n_[l] × m))
A^[l] = g^[l](Z^[l])               (Non-Linear Post-Activation, A^[l] ∈ ℝ^(n_[l] × m))

where g^[l] is the activation function of layer l, applied element-wise, and addition with b^[l] utilizes broadcasting across all m columns.

The Backpropagation Algorithm: Multivariate Chain Rule Derivation#

Backpropagation (Rumelhart, Hinton, and Williams, 1986) is an efficient implementation of reverse-mode automatic differentiation that computes the gradient of a scalar cost function J with respect to all parameter weights and biases throughout the network.

Let the total empirical cost function across the mini-batch of m instances be:

J(W, b) = (1 / m) · ∑ [i = 1 to m] L(a^[L](x^(i)), y^(i))

To derive the backpropagation equations, we define the intermediate error term δ^[l] for layer l as the partial derivative of the scalar cost J with respect to the linear pre-activation Z^[l]:

δ^[l] = ∂J / ∂Z^[l] ∈ ℝ^(n_[l] × m)

Step 1: Output Layer Error (Layer L)#

Using the chain rule on the final layer L:

δ^[L] = (∂J / ∂A^[L]) ⊙ g'^[L](Z^[L])

where ⊙ denotes the Hadamard (element-wise) product.

Case Study: Softmax Output with Cross-Entropy Loss#

For a K-class classification task, the output activation is the Softmax function:

a_k^[L] = e^(z_k^[L]) / [ ∑ [j = 1 to K] e^(z_j^[L]) ]

The categorical cross-entropy loss for an observation with one-hot target vector y ∈ ℝ^K is:

L = - ∑ [k = 1 to K] y_k · ln(a_k^[L])

Computing the derivative with respect to pre-activation z_i^[L]:

• If k = i: ∂a_i / ∂z_i = a_i · (1 - a_i)
• If k ≠ i: ∂a_k / ∂z_i = - a_k · a_i

Applying the chain rule:

∂L / ∂z_i^[L] = ∑ [k = 1 to K] (∂L / ∂a_k^[L]) · (∂a_k^[L] / ∂z_i^[L])
              = - (y_i / a_i) · a_i · (1 - a_i) - ∑ [k ≠ i] (y_k / a_k) · (- a_k · a_i)
              = - y_i · (1 - a_i) + a_i · ∑ [k ≠ i] y_k
              = - y_i + y_i · a_i + a_i · (1 - y_i)
              = a_i^[L] - y_i

In vectorized notation for the entire mini-batch:

δ^[L] = A^[L] - Y

The complex non-linear derivatives of the softmax denominator cancel out completely, leaving an extraordinarily elegant and numerically stable error signal: predictions minus ground truth.

Step 2: Backpropagating the Error (Hidden Layers l = L-1 down to 1)#

To compute δ^[l] from the subsequent layer's error δ^[l+1], we apply the multivariate chain rule:

∂J / ∂Z^[l] = (∂Z^[l+1] / ∂Z^[l])^T · (∂J / ∂Z^[l+1])

Since Z^[l+1] = W^[l+1] A^[l] + b^[l+1] and A^[l] = g^[l](Z^[l]):

∂Z^[l+1] / ∂A^[l] = W^[l+1]
∂A^[l] / ∂Z^[l] = g'^[l](Z^[l])

Multiplying and transposing appropriately:

δ^[l] = ((W^[l+1])^T · δ^[l+1]) ⊙ g'^[l](Z^[l])

The error vector from layer l + 1 is projected backward through the transpose of the weight matrix (W^[l+1])^T and gated element-wise by the local derivative of the activation function g'^[l](Z^[l]).

Step 3: Computing Weight and Bias Gradients#

Having computed δ^[l], the partial derivatives with respect to parameters in layer l are obtained directly:

∂J / ∂W^[l] = (1 / m) · δ^[l] · (A^[l-1])^T
∂J / ∂b^[l] = (1 / m) · ∑ [columns of δ^[l]]

+---------------------------------------------------------------------------------------+
|                             The Backpropagation Control Loop                          |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   Forward Pass:                                                                       |
|   A^[0] = X^T ---> [ Z^[1] = W^[1] A^[0] + b^[1] ] ---> A^[1] = g(Z^[1]) ---> ...     |
|                                                                                       |
|   Loss Computation: J(A^[L], Y)                                                       |
|                                                                                       |
|   Backward Pass:                                                                      |
|   δ^[L] = A^[L] - Y                                                                   |
|     |                                                                                 |
|     +---> ∂J/∂W^[L] = (1/m) δ^[L] (A^[L-1])^T,   ∂J/∂b^[L] = (1/m) sum(δ^[L])         |
|     |                                                                                 |
|     v                                                                                 |
|   δ^[L-1] = ((W^[L])^T δ^[L]) ⊙ g'(Z^[L-1])                                           |
|     |                                                                                 |
|     +---> ∂J/∂W^[L-1] = (1/m) δ^[L-1] (A^[L-2])^T, ...                                |
|                                                                                       |
+---------------------------------------------------------------------------------------+

Advanced First-Order Optimization Algorithms#

Standard Stochastic Gradient Descent (SGD) struggles on cost surfaces with pathological curvature—ravines where the surface curves much more steeply in one dimension than in another, causing wild oscillations.

1. SGD with Momentum#

Momentum accelerates gradient descent in the direction of persistent gradients while dampening orthogonal oscillations by maintaining an exponentially decaying moving average of past gradients:

v_t = β · v_(t-1) + (1 - β) · ∇_θ J(θ)
θ := θ - α · v_t

where β ∈ [0.8, 0.99] is the momentum coefficient (typically β = 0.9).

2. RMSProp (Root Mean Square Propagation)#

Geoffrey Hinton introduced RMSProp to resolve oscillation by keeping an exponentially decaying average of squared gradients, dynamically scaling the step size for each parameter inversely proportional to its recent gradient magnitude:

s_t = β_2 · s_(t-1) + (1 - β_2) · (∇_θ J(θ))^2
θ := θ - [ α / (√(s_t) + ε) ] ⊙ ∇_θ J(θ)

Parameters receiving large, erratic gradients receive smaller effective updates, while parameters with infrequent or small gradients receive larger steps.

3. Adam (Adaptive Moment Estimation)#

Adam (Kingma and Ba, 2014) combines the advantages of Momentum (first moment estimation) and RMSProp (second raw moment estimation), incorporating analytical bias-correction terms to account for initialization at the origin (v_0 = 0, s_0 = 0):

1. Update Biased First Moment:

```

m_t = β_1 · m_(t-1) + (1 - β_1) · ∇_θ J(θ)

```

1. Update Biased Second Moment:

```

v_t = β_2 · v_(t-1) + (1 - β_2) · (∇_θ J(θ))^2

```

1. Compute Bias-Corrected Moments:

```

m_hat_t = m_t / (1 - β_1^t)

v_hat_t = v_t / (1 - β_2^t)

```

1. Update Parameters:

```

θ := θ - [ α / (√(v_hat_t) + ε) ] · m_hat_t

```

Standard hyperparameter defaults: α = 0.001, β_1 = 0.9, β_2 = 0.999, ε = 10^(-8).

Weight Initialization & Regularization#

1. Why Zero Initialization Fails (Symmetry Breaking)#

If all weights in a neural network are initialized to zero (W = 0):

• Every neuron in a hidden layer computes the identical pre-activation: z_i^[1] = 0 + b_i.
• Every neuron outputs the identical post-activation: a_i^[1] = g(z_i^[1]).
• During backpropagation, all neurons receive the identical error δ_i^[1] and gradient ∂J/∂w.
• Even after thousands of iterations, all hidden units remain identical clones of one another, failing to learn distinct features.

Weights must be initialized with random perturbations to break symmetry.

2. Xavier (Glorot) & He (Kaiming) Initialization#

To prevent activations and gradients from exploding or vanishing across deep layers, the variance of the outputs of each layer must equal the variance of its inputs: Var(a^[l]) = Var(a^[l-1]).

• Xavier (Glorot) Initialization (for Tanh / Sigmoid activations):

```

W^[l] ~ Normal( 0, σ^2 = 2 / (n_[l-1] + n_[l]) )

```

• He (Kaiming) Initialization (for ReLU activations):

Because ReLU zeroes out approximately half of all activations (z < 0), the input variance is halved. Compensating for this requires doubling the initialization variance:

```

W^[l] ~ Normal( 0, σ^2 = 2 / n_[l-1] )

```

3. Regularization: Dropout & Batch Normalization#

• Dropout (Srivastava et al., 2014): During each training forward pass, each neuron is independently zeroed out with probability p_drop (e.g., p = 0.5). This prevents neurons from developing fragile co-adaptations; a neuron cannot rely on the presence of specific neighboring units and must learn robust, generalized representations. Modern implementations use Inverted Dropout (scaling surviving activations by 1 / (1 - p_drop) during training so no modification is needed at test time).
• Batch Normalization (Ioffe and Szegedy, 2015): Normalizes the pre-activations Z^[l] across the mini-batch to have zero mean and unit variance, followed by a learned affine transformation (γ · z_norm + β). This dramatically stabilizes gradient flow, reduces sensitivity to weight initialization, and acts as a mild regularizer.

Checkpoint Practice Problems#

Problem 1 (Two-Layer MLP Forward Propagation): A tiny neural network has 2 inputs, a hidden layer with 2 neurons (using ReLU activation), and an output layer with 1 neuron (using Sigmoid activation):

W^[1] = [ [ 2.0, -1.0 ],
          [ 1.0,  3.0 ] ]
b^[1] = [ -1.0, 2.0 ]^T

W^[2] = [ [ 1.5, -2.0 ] ]
b^[2] = [ 0.5 ]

For an input observation x = [ 1.0, 2.0 ]^T:

1. Compute the linear pre-activation vector z^[1].
2. Compute the post-activation vector a^[1] = ReLU(z^[1]).
3. Compute the output pre-activation scalar z^[2].
4. Compute the final network prediction y_hat = a^[2] = σ(z^[2]).

Solution:

1. Compute z^[1] = W^[1] x + b^[1]:

```

z_1^[1] = (2.0)(1.0) + (-1.0)(2.0) + (-1.0) = 2.0 - 2.0 - 1.0 = -1.0

z_2^[1] = (1.0)(1.0) + (3.0)(2.0) + (2.0) = 1.0 + 6.0 + 2.0 = 9.0

z^[1] = [ -1.0, 9.0 ]^T

```

1. Apply ReLU activation a^[1] = max(0, z^[1]):

```

a_1^[1] = max(0, -1.0) = 0.0

a_2^[1] = max(0, 9.0) = 9.0

a^[1] = [ 0.0, 9.0 ]^T

```

1. Compute z^[2] = W^[2] a^[1] + b^[2]:

```

z^[2] = (1.5)(0.0) + (-2.0)(9.0) + 0.5 = 0 - 18.0 + 0.5 = -17.5

```

1. Compute final output y_hat = σ(-17.5):

```

y_hat = 1 / (1 + e^(17.5))

e^(17.5) ≈ 39,824,784

y_hat ≈ 1 / 39,824,785 ≈ 2.51 × 10^(-8)

```

The network outputs approximately 0.000000025.

Problem 2 (Exact Backpropagation Gradient Pass): Continuing with the network from Problem 1, suppose the true ground truth label for this input is y = 1.0. The loss is binary cross-entropy:

L = - [ y · ln(a^[2]) + (1 - y) · ln(1 - a^[2]) ].

1. Compute the output error scalar δ^[2] = ∂L / ∂z^[2].
2. Compute the gradient of the loss with respect to second-layer weights ∂L / ∂W^[2] and bias ∂L / ∂b^[2].
3. Compute the hidden-layer error vector δ^[1] = ((W^[2])^T · δ^[2]) ⊙ ReLU'(z^[1]).
4. Compute the gradient ∂L / ∂W^[1].

Solution:

1. Output error δ^[2]:

For sigmoid with binary cross-entropy, δ^[2] = a^[2] - y.

Since a^[2] ≈ 0.0 and y = 1.0:

```

δ^[2] = a^[2] - y = 0.0 - 1.0 = -1.0

```

1. Gradients for Layer 2:

```

∂L / ∂W^[2] = δ^[2] · (a^[1])^T = (-1.0) · [ 0.0, 9.0 ] = [ 0.0, -9.0 ]

∂L / ∂b^[2] = δ^[2] = -1.0

```

1. Hidden Layer error δ^[1]:

Recall z^[1] = [ -1.0, 9.0 ]^T.

The derivative of ReLU is:

```

ReLU'(-1.0) = 0.0

ReLU'(9.0) = 1.0

ReLU'(z^[1]) = [ 0.0, 1.0 ]^T

```

Compute (W^[2])^T · δ^[2]:

```

(W^[2])^T = [ 1.5, -2.0 ]^T

(W^[2])^T · δ^[2] = [ 1.5, -2.0 ]^T · (-1.0) = [ -1.5, 2.0 ]^T

```

Apply element-wise product:

```

δ^[1] = [ -1.5, 2.0 ]^T ⊙ [ 0.0, 1.0 ]^T = [ (-1.5)(0.0), (2.0)(1.0) ]^T = [ 0.0, 2.0 ]^T

```

1. Gradients for Layer 1:

The input was x = [ 1.0, 2.0 ]^T:

```

∂L / ∂W^[1] = δ^[1] · (x)^T = [ [ 0.0 ], · [ 1.0, 2.0 ]

[ 2.0 ] ]

= [ [ 0.0(1.0), 0.0(2.0) ],

[ 2.0(1.0), 2.0(2.0) ] ]

= [ [ 0.0, 0.0 ],

[ 2.0, 4.0 ] ]

```

Notice that the first neuron's weights receive zero gradient because it was inactive in the forward pass (ReLU deactivated).

Problem 3 (Softmax Probability & Multi-Class Cross-Entropy): An output layer produces pre-activation logits for K = 3 classes:

z = [ 2.0, 1.0, -1.0 ]^T

The ground-truth one-hot label vector is:

y = [ 0, 1, 0 ]^T (Class 2 is correct).

1. Compute the Softmax probability vector a = [ a_1, a_2, a_3 ]^T.
2. Compute the Categorical Cross-Entropy Loss L.
3. Compute the output layer error gradient vector δ = a - y.

Solution:

1. Compute exponentials:

* e^(z_1) = e^(2.0) ≈ 7.38906

* e^(z_2) = e^(1.0) ≈ 2.71828

* e^(z_3) = e^(-1.0) ≈ 0.36788

Sum = 7.38906 + 2.71828 + 0.36788 = 10.47522

1. Compute Softmax probabilities:

```

a_1 = 7.38906 / 10.47522 ≈ 0.70538 (70.5%)

a_2 = 2.71828 / 10.47522 ≈ 0.25950 (26.0%)

a_3 = 0.36788 / 10.47522 ≈ 0.03512 (3.5%)

a = [ 0.7054, 0.2595, 0.0351 ]^T

```

1. Compute Cross-Entropy Loss:

Since only y_2 = 1:

```

L = - ln(a_2) = - ln(0.25950) ≈ - (-1.3490) = 1.3490

```

1. Compute error gradient δ:

```

δ_1 = a_1 - y_1 = 0.70538 - 0 = +0.70538

δ_2 = a_2 - y_2 = 0.25950 - 1 = -0.74050

δ_3 = a_3 - y_3 = 0.03512 - 0 = +0.03512

δ = [ 0.7054, -0.7405, 0.0351 ]^T

```

Problem 4 (He/Kaiming Weight Initialization Variance): A deep convolutional neural network uses ReLU activation functions. Layer l receives input from n_[l-1] = 512 incoming channels.

1. Compute the optimal variance Var(W) mandated by He (Kaiming) normal initialization.
2. Compute the corresponding standard deviation σ_He.
3. If Xavier (Glorot) initialization had been erroneously used instead (assuming n_[l] = 512 outgoing channels), compute the resulting variance and state what happens to activations across deep layers.

Solution:

1. He Initialization Variance:

```

Var(W_He) = 2 / n_[l-1] = 2 / 512 = 1 / 256 = 0.00390625

```

1. Standard Deviation:

```

σ_He = √(2 / 512) = √(1 / 256) = 1 / 16 = 0.0625

```

Weights should be sampled from Normal(μ = 0, σ = 0.0625).

1. Comparison with Xavier Initialization:

```

Var(W_Xavier) = 2 / (n_[l-1] + n_[l]) = 2 / (512 + 512) = 2 / 1024 = 1 / 512 ≈ 0.001953

```

Notice that Var(W_Xavier) is exactly half of Var(W_He).

If Xavier initialization is used with ReLU, the variance of activations drops by approximately 50% at every single layer:

```

Var(a^[L]) ≈ (0.5)^L · Var(a^[0])

```

In a 30-layer deep network, (0.5)^30 ≈ 9.3 × 10^(-10). Activation signals decay to zero exponentially, severely impairing the network's capacity to propagate information.