Word Embeddings & Recurrent Neural Networks

A fundamental turning point in artificial intelligence was the transition from sparse, orthogonal lexical spaces (such as One-Hot encodings and Bag-of-Words matrices) to dense, continuous, low-dimensional Distributed Representations. In a One-Hot representation, every word is an isolated basis vector in R^V whose inner product with any other word is identically zero: w_{cat} · w_{dog} = 0. Consequently, statistical models could not generalize knowledge learned about "cat" to semantically adjacent entities like "kitten" or "feline."

Guided by linguist J.R. Firth's classic 1957 aphorism—"You shall know a word by the company it keeps"—modern representation learning maps words to dense continuous vectors (R^D, where D ≪ V, typically 100 ≤ D ≤ 300) such that geometric proximity directly reflects semantic and syntactic similarity. When paired with Recurrent Neural Networks (RNNs) and gated variants (LSTMs and GRUs), these continuous vector spaces unlocked the ability to model dynamic temporal dependencies, syntactic hierarchies, and variable-length sequences.

1. Theoretical & Mathematical Foundations of Word Embeddings#

The Distributional Hypothesis and Continuous Vector Spaces#

Let V represent a fixed vocabulary of |V| discrete word types. A distributed word embedding is an injective mapping:

ϕ: V → R^D (D ≪ |V|)

where the geometric distance (Euclidean distance or Cosine distance) between vectors v_i and v_j reflects their contextual substitutability across natural language corpora.

+-----------------------------------------------------------------------------------+
|                        THE CONTINUOUS EMBEDDING GEOMETRY                          |
+-----------------------------------------------------------------------------------+
|  High-Dimensional Semantic Manifold (D = 300):                                    |
|                                                                                   |
|         v_king o-----------------------------> o v_queen                          |
|                \                              /                                   |
|                 \   Vector Difference:       /                                    |
|                  \  v_king - v_man          /  v_queen - v_woman                  |
|                   \  ≈ Gender Direction    /   ≈ Gender Direction                 |
|                    v                      v                                       |
|          v_man o-----------------------------> o v_woman                          |
|                                                                                   |
|  Analogical Vector Arithmetic:                                                    |
|  v_king - v_man + v_woman ≈ v_queen                                               |
|  v_paris - v_france + v_italy ≈ v_rome                                            |
+-----------------------------------------------------------------------------------+

Word2Vec: CBOW and Continuous Skip-Gram#

Mikolov et al. (2013) introduced Word2Vec, replacing deep non-linear neural language models with simplified, highly parallelizable single-layer feedforward architectures:

1. Continuous Bag-of-Words (CBOW): Predicts a central target word w_t given the surrounding context words within a sliding window of radius c:

Context(w_t) = \{w_{t-c}, …, w_{t-1}, w_{t+1}, …, w_{t+c}\}

The input vectors of the context words are averaged, projecting the sum into hidden layer h = (1 / 2c) ∑_{-c ≤ j ≤ c, j ≠ 0} v_{w_{t+j}}. CBOW smooths over distributional noise and trains rapidly on large corpora.

1. Continuous Skip-Gram: Inverts the CBOW objective: given a single central target word w_t, the network predicts the probability distribution of surrounding context words w_{t+j}. Skip-gram gives equal weight to rare context words, excelling at capturing fine-grained semantics on infrequent words.

CBOW ARCHITECTURE:                           SKIP-GRAM ARCHITECTURE:
[Context Word w_{t-2}] ---\                  [Context Word w_{t-2}] <---
[Context Word w_{t-1}] ----\  Projection                                \
                            +-> [SUM/AVG] -> [Target w_t] -> [Target w_t] -+-> [Output Softmax]
[Context Word w_{t+1}] ----/                                            /
[Context Word w_{t+2}] ---/                  [Context Word w_{t+2}] <---

The Softmax Bottleneck & Negative Sampling (SGNS)#

In the standard formulation, the conditional probability of predicting context word w_o given target word w_i is governed by the full categorical Softmax function:

P(w_o | w_i) = (exp(u_{w_o}^T v_{w_i}) / ∑_{w=1}^{|V|} exp(u_w^T v_{w_i}))

where v_w is the input embedding and u_w is the output context embedding.

• The Computational Bottleneck: Evaluating the denominator requires calculating the inner product across all |V| words in the vocabulary for every single training word. If |V| = 1,000,000, computing the normalization factor for a 1-billion-token corpus requires 10^{15} floating-point operations, rendering training computationally intractable.

Skip-Gram with Negative Sampling (SGNS):

Mikolov et al. resolved this by replacing the multi-class categorical classification problem with a set of independent Binary Logistic Regressions. For each observed positive word-context pair (w, c), the model generates K negative samples (w, n_k) drawn at random from a noise distribution:

L_{SGNS} = ln σ(u_c^T v_w) + ∑_{k=1}^K E_{n_k ~ P_n(w)} [ ln σ(-u_{n_k}^T v_w) ]

where σ(z) = (1 / (1 + e^(-z))) is the logistic sigmoid function.

• Complexity Reduction: Evaluating the loss and computing parameter gradients now scales as O(K), where typically 5 ≤ K ≤ 20. Training speed increases by thousands of times.
• The Unigram 3/4 Distribution: Negative samples are drawn from the smoothed unigram distribution:

P_n(w) = (U(w)^(3/4) / ∑_{w'} U(w')^(3/4))

Raising word frequency U(w) to the power of 0.75 dampens the probability of ubiquitous stop words (like the, of) while significantly boosting the probability of rare words, ensuring that negative samples challenge the model with diverse vocabulary.

Global Vectors for Word Representation (GloVe)#

While Word2Vec relies on local window sampling, it fails to leverage global co-occurrence statistics across the entire corpus. GloVe (Pennington, Socher, and Manning, 2014) established a log-bilinear model that explicitly fits global co-occurrence matrix counts X_{i, j}:

J_{GloVe} = ∑_{i, j=1}^{|V|} f(X_{i, j}) ( w_i^T w̃_j + b_i + b̃_j - ln X_{i, j} )^2

where f(X) = min(1, (X / x_{max})^α) (with x_{max} = 100, α = 0.75) is a weighting function that prevents very frequent co-occurrences from dominating the loss. GloVe provides an elegant mathematical bridge between global matrix factorization (like LSA) and local context window learning.

FastText: Subword Character N-Grams#

Standard Word2Vec and GloVe assign a distinct vector to each full word string. If an inflected or misspelled word (e.g., un-preprocessing) was absent during training, it receives zero vector representation.

FastText (Bojanowski et al., 2017) decomposes every word into a bag of character n-grams bounded by angle brackets. For example, with n = 3, the word where yields:

n-grams = \{<wh, whe, her, ere, re>\} ∪ \{<where>\}

The final embedding of a word is the linear vector sum of its constituent n-gram embeddings:

v_{word} = ∑_{g ∈ G_{word}} z_g

FastText captures morphological prefixes and suffixes (-tion, un-, anti-), allowing the model to generate robust embeddings for completely unseen out-of-vocabulary words.

2. Recurrent Neural Networks (RNNs) and Sequence Modeling#

Word embeddings project individual symbols into continuous vectors, but language is inherently sequential: syntactic structure and semantic disambiguation depend on temporal context.

The Standard Recurrent Neural Network (Vanilla RNN)#

A standard Recurrent Neural Network processes an ordered sequence of input vectors (x_1, x_2, …, x_T) by maintaining an internal hidden state vector h_t ∈ R^H acting as working temporal memory:

h_t = tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)

ŷ_t = Softmax(W_{hy} h_t + b_y)

UNROLLED COMPUTATIONAL GRAPH OF A RECURRENT NEURAL NETWORK:
      y_1             y_2             y_3             y_T
       ^               ^               ^               ^
       | W_hy          | W_hy          | W_hy          | W_hy
    [ h_1 ] --W_hh-> [ h_2 ] --W_hh-> [ h_3 ] ...-> [ h_T ]
       ^               ^               ^               ^
       | W_xh          | W_xh          | W_xh          | W_xh
      x_1             x_2             x_3             x_T

• Parameter Sharing: The weight matrices W_{hh}, W_{xh}, and W_{hy} are tied and shared identically across every time step t. This allows the model to process sequences of arbitrary, variable length.

The Vanishing and Exploding Gradient Problem#

Training recurrent networks requires Backpropagation Through Time (BPTT). Let total sequence loss be L = ∑_{t=1}^T L_t. When evaluating the gradient with respect to hidden state weight matrix W_{hh}, the chain rule propagates error backwards through time:

(∂ L_T / ∂ h_1) = (∂ L_T / ∂ h_T) ∏_{k=2}^T (∂ h_k / ∂ h_{k-1})

The Jacobian matrix of the hidden state transition is:

(∂ h_k / ∂ h_{k-1}) = diag(1 - tanh^2(a_k)) W_{hh}^T

Mathematical Consequence:

The term ∏_{k=2}^T W_{hh}^T represents repeated matrix multiplication across T - 1 steps.

• Let the eigendecomposition of W_{hh} have largest absolute eigenvalue (spectral radius) λ_{max}.
• If λ_{max} > 1, the gradient magnitude grows exponentially as λ_{max}^{T-1}, causing Exploding Gradients (numeric NaN overflows and destructive weight updates).
• If λ_{max} < 1 (or because 1 - tanh^2(z) ≤ 1), the gradient magnitude decays exponentially to zero as λ_{max}^{T-1}, causing Vanishing Gradients. By time step t = 20, the gradient signal from time step t = 1 is completely extinguished, preventing the network from learning long-term semantic dependencies.

Mitigation for Exploding Gradients (Gradient Clipping):

If the Euclidean norm of the parameter gradient vector exceeds a pre-defined threshold C:

g arrow {

g if ||g||_2 ≤ C

C * (g / ||g||_2) if ||g||_2 > C

}

Gradient clipping bounds the step size while preserving the true descent direction.

3. Gated Recurrent Architectures: LSTM and GRU#

To resolve the vanishing gradient pathology, modern architectures introduce Gated Linear Memory Channels that establish highway connections for gradient flow across hundreds of time steps.

Long Short-Term Memory (LSTM)#

Invented by Hochreiter & Schmidhuber (1997), the LSTM decouples working memory into two distinct states:

• c_t ∈ R^H: The Cell State (long-term memory channel).
• h_t ∈ R^H: The Hidden State (short-term working memory).

+-----------------------------------------------------------------------------------+
|                            LSTM CELL ARCHITECTURE                                 |
+-----------------------------------------------------------------------------------+
|  Cell State Highway: c_{t-1} -----------------( * )----------------------> c_t    |
|                                                ^              ^                   |
|                                                | f_t          | (i_t * c~_t)      |
|                                                |              |                   |
|         +-----------------------+--------------+              |                   |
|         |                       |                             |                   |
|         |                       |     +-----------------------+                   |
|         |                       |     |                                           |
|         v                       v     v                       v                   |
|       [ σ ]                   [ σ ] [tanh]                  [ σ ]                 |
|      Forget                   Input Candidate               Output                |
|       Gate                    Gate  State                    Gate                 |
|        f_t                     i_t   c~_t                     o_t                 |
|         ^                       ^     ^                        ^                  |
|         |                       |     |                        |                  |
|  [ x_t, h_{t-1} ] --------------+-----+------------------------+                  |
|                                                                |                  |
|  Hidden State Output: h_t <----------------------------------( * ) <--- [tanh(c_t)]
+-----------------------------------------------------------------------------------+

The Governing LSTM Equations:

1. Forget Gate (f_t): Determines what fraction of the past cell state to discard:

f_t = σ(W_f [h_{t-1}, x_t] + b_f)

1. Input Gate (i_t): Regulates which dimensions of the candidate state to admit:

i_t = σ(W_i [h_{t-1}, x_t] + b_i)

1. Candidate State (c̃_t): Generates new candidate information:

c̃_t = tanh(W_c [h_{t-1}, x_t] + b_c)

1. Cell State Update (c_t): The fundamental core of the LSTM:

c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t

where ⊙ denotes the element-wise Hadamard product.

1. Output Gate (o_t) & Hidden State (h_t):

o_t = σ(W_o [h_{t-1}, x_t] + b_o)

h_t = o_t ⊙ tanh(c_t)

Why LSTMs Do Not Vanish:

In the cell state update equation, the gradient (∂ c_t / ∂ c_{t-1}) = f_t. If the network learns to saturate the forget gate close to 1 (f_t ≈ 1), the gradient flows backwards across arbitrary time intervals without decaying:

(∂ L / ∂ c_{t-k}) ≈ (∂ L / ∂ c_t) × 1.0

This is termed the Constant Error Carousel (CEC).

The Gated Recurrent Unit (GRU)#

Cho et al. (2014) introduced the GRU, compressing the LSTM into a computationally leaner architecture without a separate cell state:

1. Update Gate (z_t): Combines the functions of the input and forget gates:

z_t = σ(W_z [h_{t-1}, x_t] + b_z)

1. Reset Gate (r_t): Determines how much of the previous state to ignore when computing candidate activations:

r_t = σ(W_r [h_{t-1}, x_t] + b_r)

1. Candidate Hidden State (h̃_t):

h̃_t = tanh(W [r_t ⊙ h_{t-1}, x_t] + b)

1. Hidden State Interpolation (h_t):

h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t

• Advantages: Has 25\% fewer parameters than an LSTM (3 gate matrices instead of 4), trains faster, and achieves comparable empirical performance on small-to-medium sequence tasks.

4. Sequence-to-Sequence (Seq2Seq) and Bahdanau Attention#

For tasks where input and output lengths differ (such as Machine Translation, Summarization, and Speech-to-Text), Sutskever et al. (2014) established the Seq2Seq Encoder-Decoder architecture:

ENCODER (BiLSTM)                                      DECODER (Autoregressive LSTM)
"The"      "cat"      "sat"                             "Le"       "chat"     "était"
  |          |          |                                 |          |          |
[ h_1 ] -> [ h_2 ] -> [ h_3 ] === (Context Vector c) ===> [ s_1 ] -> [ s_2 ] -> [ s_3 ]

• The Information Bottleneck: The Encoder compresses an arbitrary-length input sentence into a single, fixed-dimensional context vector c = h_T. For sentences exceeding 20 words, compressing all semantic nuance into a fixed vector causes catastrophic forgetting of early sentence tokens.

Bahdanau Additive Attention (1994 / 2014)#

Bahdanau, Cho, and Bengio (2014) eliminated this bottleneck by introducing Attention: allowing the decoder to look back at all encoder hidden states (h_1, …, h_{T_x}) dynamically at each decoding step t:

c_t = ∑_{i=1}^{T_x} α_{t, i} h_i

where the attention weight α_{t, i} measures how much the decoder at state s_{t-1} should focus on encoder state h_i:

e_{t, i} = v_a^T tanh(W_a s_{t-1} + U_a h_i)

α_{t, i} = (exp(e_{t, i}) / ∑_{j=1}^{T_x} exp(e_{t, j}))

By calculating dynamic soft-alignment weights across the input sequence, attention provided the mathematical bridge from recurrent sequential processing to the fully attentional Transformer architecture.

5. University-Grade Checkpoint Practice Problems#

Problem 1: Skip-Gram Negative Sampling Mathematical Gradient Derivation#

Context: In a Word2Vec model, the Skip-Gram with Negative Sampling (SGNS) objective function for a single target word w and an observed context word c with K negative samples \{n_1, n_2, …, n_K\} is defined as:

L(v_w, u_c, u_{n_1}, …, u_{n_K}) = ln σ(u_c^T v_w) + ∑_{k=1}^K ln σ(-u_{n_k}^T v_w)

where:

• v_w ∈ R^D is the embedding of target word w.
• u_c ∈ R^D is the output context embedding of true context word c.
• u_{n_k} ∈ R^D is the output context embedding of negative sample n_k.
• σ(z) = (1 / (1 + e^(-z))) is the standard logistic sigmoid function.

Mathematical Identity:

(d / dz) σ(z) = σ(z)(1 - σ(z))

σ(-z) = 1 - σ(z)

Task:

1. Derive the exact derivative of ln σ(z) with respect to z.
2. Derive the exact derivative of ln σ(-z) with respect to z.
3. Derive the gradient of the objective function with respect to the target word embedding vector:

∇_{v_w} L

1. Provide the physical and intuitive interpretation of this gradient update during stochastic gradient ascent:

v_w arrow v_w + η ∇_{v_w} L

Solution:

1. Derivative of ln σ(z):

(d / dz) ln σ(z) = (1 / σ(z)) × (d / dz) σ(z) = (1 / σ(z)) × σ(z)(1 - σ(z)) = 1 - σ(z)

1. Derivative of ln σ(-z):

* Using the chain rule with u = -z:

(d / dz) ln σ(-z) = (1 / σ(-z)) × (d / dz) σ(-z) = (1 / 1 - σ(z)) × (-σ(-z)(1 - σ(-z)))

(d / dz) ln σ(-z) = (1 / 1 - σ(z)) × (-(1 - σ(z)) σ(z)) = -σ(z)

1. Gradient with Respect to Target Embedding v_w:

* Evaluating the positive term:

Let z_c = u_c^T v_w.

∇_{v_w} ln σ(u_c^T v_w) = (d ln σ(z_c) / d z_c) ∇_{v_w}(u_c^T v_w) = (1 - σ(u_c^T v_w)) u_c

* Evaluating the negative terms:

For each k, let z_{n_k} = u_{n_k}^T v_w.

∇_{v_w} ln σ(-u_{n_k}^T v_w) = -σ(u_{n_k}^T v_w) u_{n_k}

* Summing over all terms:

∇_{v_w} L = (1 - σ(u_c^T v_w)) u_c - ∑_{k=1}^K σ(u_{n_k}^T v_w) u_{n_k}

1. Intuitive Physical Interpretation:

* The gradient vector ∇_{v_w} L consists of two competing physical forces:

1. Attractive Force: The positive context vector u_c is added with scalar weight (1 - σ(u_c^T v_w)). If the model already assigns high probability to the true context (σ ≈ 1), the scalar is ≈ 0 (no update needed). If the model fails to predict the context (σ ≈ 0), the scalar is ≈ 1, pulling v_w strongly in the direction of u_c.

2. Repulsive Force: Each negative sample vector u_{n_k} is subtracted with scalar weight σ(u_{n_k}^T v_w). If the model mistakenly places target w close to negative sample n_k (σ ≈ 1), a strong repulsive force pushes v_w directly away from u_{n_k}.

Problem 2: LSTM Constant Error Carousel Mathematical Proof#

Context: In a Long Short-Term Memory (LSTM) network, the cell state update equation is governed by:

c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t

where:

• f_t = σ(W_f [h_{t-1}, x_t] + b_f) (Forget gate)
• i_t = σ(W_i [h_{t-1}, x_t] + b_i) (Input gate)
• c̃_t = tanh(W_c [h_{t-1}, x_t] + b_c) (Candidate cell state)
• h_t = o_t ⊙ tanh(c_t) (Hidden state)

Suppose a network processes a sequence of length T = 100. A loss L is computed at the final time step T.

We examine the backpropagation of error from cell state c_T back to cell state c_1:

(∂ L / ∂ c_1) = (∂ L / ∂ c_T) ∏_{t=2}^T (∂ c_t / ∂ c_{t-1})

Task:

1. Derive the exact Jacobian matrix (∂ c_t / ∂ c_{t-1}) for an arbitrary time step t.
2. Identify the primary additive term in the Jacobian that remains independent of hidden state activation derivatives.
3. Prove that if the forget gates are initialized with positive biases such that f_t ≈ 1, the gradient magnitude is preserved across all 100 time steps without exponential decay.
4. Contrast this mathematical mechanism with the Vanilla RNN hidden state transition matrix (∂ h_t / ∂ h_{t-1}) = diag(1 - h_t^2) W_{hh}^T.

Solution:

1. Jacobian Matrix Derivation (∂ c_t / ∂ c_{t-1}):

* Differentiating the update equation c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t with respect to c_{t-1}:

(∂ c_t / ∂ c_{t-1}) = diag(f_t) + (∂ f_t / ∂ c_{t-1}) ⊙ c_{t-1} + (∂ i_t / ∂ c_{t-1}) ⊙ c̃_t + (∂ c̃_t / ∂ c_{t-1}) ⊙ i_t

* Note that f_t, i_t, c̃_t depend on c_{t-1} only indirectly through h_{t-1}:

(∂ f_t / ∂ c_{t-1}) = (∂ f_t / ∂ h_{t-1}) (∂ h_{t-1} / ∂ c_{t-1})

where (∂ h_{t-1} / ∂ c_{t-1}) = diag(o_{t-1} ⊙ (1 - tanh^2(c_{t-1}))).

1. The Dominant Additive Term:

* The Jacobian separates into two distinct components:

(∂ c_t / ∂ c_{t-1}) = diag(f_t) + Ω_t

where Ω_t contains all indirect derivative paths through h_{t-1}.

The crucial element is the direct additive linear highway*:

Direct Term = diag(f_t)

1. Proof of Gradient Preservation:

* Expanding the product across T - 1 time steps:

∏_{t=2}^T (∂ c_t / ∂ c_{t-1}) = ∏_{t=2}^T (diag(f_t) + Ω_t) = ( ∏_{t=2}^T diag(f_t) ) + Cross Terms

* When forget gate biases are initialized positively (e.g., b_f = +1.0 or +2.0, recommended by Jozefowicz et al., 2015), the sigmoid saturates near 1:

f_t ≈ 1 ⟹ diag(f_t) ≈ I (Identity Matrix)

* Evaluating the product:

∏_{t=2}^{100} diag(f_t) ≈ ∏_{t=2}^{100} I = I

* Therefore:

(∂ L / ∂ c_1) ≈ (∂ L / ∂ c_T) I = (∂ L / ∂ c_T)

Conclusion*: The gradient flows completely unimpeded across 100 time steps with zero exponential shrinkage.

1. Contrast with Vanilla RNN:

* In a Vanilla RNN, the gradient is strictly multiplicative:

(∂ h_T / ∂ h_1) = ∏_{t=2}^T diag(1 - h_t^2) W_{hh}^T

* Because tanh'(z) ∈ (0, 1], |1 - h_t^2| < 1. Even if W_{hh} = I, the product of 100 numbers less than 0.5 decays to 0.5^{99} ≈ 1.57 × 10^{-30} ≈ 0.

The LSTM solves vanishing gradients not by clever parameter weights, but through additive architectural topology* (the Constant Error Carousel).

Problem 3: FastText Character N-Gram Vector Composition & OOV Similarity#

Context: A FastText subword embedding model is trained with embedding dimensionality D = 4.

The model encounters an out-of-vocabulary (OOV) compound word:

Word W = "subtask"

With character n-gram bounds set to n = 3, the model decomposes <subtask> into character trigrams and the full bound word token:

• Trigrams: <su, sub, ubt, bta, tas, ask, sk>
• Full word: <subtask>

The model's trained subword embedding table contains the following learned 4D vectors for these components:

A known vocabulary word "task" has pre-computed composite embedding:

v_{task} = [0.85, 0.60, -0.45, 0.80]^T

Task:

1. Calculate the composite FastText vector v_{subtask} by summing all constituent subword embeddings.
2. Calculate the Euclidean norm \|v_{subtask}\|_2 and \|v_{task}\|_2.
3. Compute the Cosine Similarity between the unseen word "subtask" and the known word "task".
4. Explain how FastText succeeded in representing "subtask" where standard Word2Vec would fail.

Solution:

1. Composite Vector Summation for v_{subtask}:

v_{subtask} = ∑_{g} z_g

* Coordinate 1:

v_1 = 0.20 + 0.80 - 0.10 + 0.10 + 0.40 + 0.30 + 0.10 + 0.00 = 1.80

* Coordinate 2:

v_2 = -0.10 + 0.40 + 0.30 - 0.20 + 0.50 + 0.20 - 0.10 + 0.00 = 1.00

* Coordinate 3:

v_3 = 0.40 - 0.20 + 0.10 + 0.30 - 0.10 - 0.30 + 0.20 + 0.00 = 0.40

* Coordinate 4:

v_4 = 0.10 + 0.60 - 0.20 + 0.00 + 0.30 + 0.40 + 0.20 + 0.00 = 1.40

* Resulting vector:

v_{subtask} = [1.80, 1.00, 0.40, 1.40]^T

1. Euclidean Norm Calculations:

* For v_{subtask}:

\|v_{subtask}\|_2 = √(1.80^2 + 1.00^2 + 0.40^2 + 1.40^2) = √(3.24 + 1.00 + 0.16 + 1.96) = √(6.36) ≈ 2.5219

* For v_{task} = [0.85, 0.60, -0.45, 0.80]^T:

\|v_{task}\|_2 = √(0.85^2 + 0.60^2 + (-0.45)^2 + 0.80^2) = √(0.7225 + 0.3600 + 0.2025 + 0.6400) = √(1.9250) ≈ 1.3874

1. Cosine Similarity Calculation:

* Dot product:

v_{subtask} · v_{task} = (1.80 × 0.85) + (1.00 × 0.60) + (0.40 × -0.45) + (1.40 × 0.80)

v_{subtask} · v_{task} = 1.530 + 0.600 - 0.180 + 1.120 = 3.070

* Cosine Similarity:

CosineSimilarity = (3.070 / (||v_subtask||_2 * ||v_task||_2)) = (3.070 / 2.5219 × 1.3874) = (3.070 / 3.4989) ≈ 0.8774 (87.74\%)

1. Architectural Conclusion:

* In standard Word2Vec, "subtask" would be absent from the vocabulary dictionary, forcing the system to return an all-zero or randomly initialized [UNK] vector with 0\% semantic alignment.

* FastText synthesized a high-fidelity semantic representation (87.74\% cosine similarity to "task") purely by composing subword character trigrams (tas, ask), proving how morphological subword representations bridge vocabulary gaps.

Problem 4: Recurrent Network Gradient Clipping & BPTT Activation Memory#

Context: A multi-layer Bidirectional LSTM language model processes batches of sequential text.

Architecture & Hyperparameters:

• Sequence length: T = 256 time steps.
• Batch size: B = 32 sequences.
• Number of bidirectional LSTM layers: L = 2 layers.
• Hidden state dimensionality per direction: H = 512 (total bidirectional hidden state = 2 × 512 = 1,024).
• Data precision: 32-bit IEEE-754 floating point (4 bytes per value).

During the forward pass of Backpropagation Through Time (BPTT), the engine must retain all intermediate cell states c_t, hidden states h_t, and gate pre-activations in GPU VRAM to compute backward gradients.

• Each LSTM direction at each time step maintains 6 state/gate vectors: f_t, i_t, o_t, c̃_t, c_t, h_t, each of dimension H = 512.

Gradient Norm & Optimization:

During backward pass execution, the global gradient vector g across all network parameters exhibits a large norm:

\|g\|_2 = 24.8

The training configuration enforces gradient clipping with maximum norm threshold:

C = 6.0

Task:

1. Calculate the total number of floating-point values stored in GPU memory to retain forward activations for BPTT across all time steps, batch items, layers, and directions.
2. Calculate the total physical GPU VRAM footprint (in Megabytes) consumed by these activations alone.
3. Calculate the scaling factor α applied to the parameter gradient vector under gradient clipping.
4. Calculate the new Euclidean norm of the clipped gradient vector \|g_{clipped}\|_2.

Solution:

1. Total Floating-Point Activations Stored:

* Number of vectors per cell step = 6 vectors (f, i, o, c̃, c, h).

* Elements per vector = H = 512 floats.

* Floats per cell step = 6 × 512 = 3,072 floats.

* Each bidirectional layer has 2 directions (forward and backward):

3,072 × 2 = 6,144 floats per layer per time step per batch item

* Multiplying across all L = 2 layers:

6,144 × 2 = 12,288 floats per time step per batch item

* Multiplying across sequence length T = 256 and batch size B = 32:

N_{floats} = 12,288 × 256 × 32 = 100,663,296 floating-point values

1. GPU Activation VRAM Footprint:

* Data precision: 4 bytes per float.

* Total activation memory:

Mem_{act} = 100,663,296 × 4 bytes = 402,653,184 bytes

* Converting to Megabytes (base 2):

Mem_{act} = (402,653,184 / 1,024 × 1,024) = 384.0 MB of GPU VRAM

1. Gradient Clipping Scaling Factor:

* Unclipped gradient norm: \|g\|_2 = 24.8.

* Clipping threshold: C = 6.0.

* Because \|g\|_2 > C, the scaling factor is:

α = (C / ||g||_2) = (6.0 / 24.8) = (3 / 12.4) ≈ 0.2419355

* The update rule scales every parameter gradient by α:

g_{clipped} = 0.2419355 × g

1. Clipped Gradient Norm:

\|g_{clipped}\|_2 = \| (C / ||g||_2) g \|_2 = (C / ||g||_2) \|g\|_2 = C = 6.0

Result*: The gradient norm is strictly capped at 6.0, preventing gradient explosion and numerical overflow while keeping the multidimensional descent direction mathematically unaltered.