Large Language Models (LLMs) & Fine-Tuning

The scaling of the Transformer architecture to tens and hundreds of billions of parameters catalyzed the transition from task-specific natural language processing systems (trained on narrow datasets for sentiment analysis or named entity recognition) to general-purpose, multimodal Large Language Models (LLMs). When pre-trained at scale across trillions of tokens of diverse text, code, and scientific literature, autoregressive decoder models (such as GPT-4, LLaMA-3, Claude, and Gemini) exhibit remarkable capabilities—including zero-shot generalization, multi-step chain-of-thought mathematical reasoning, and autonomous code generation.

However, raw pre-trained foundation models are fundamentally next-token probability predictors: they lack conversational discipline, hallucinate false premises with high confidence, and can generate unsafe or toxic outputs. Transforming a raw base model into a reliable, helpful, and instruction-aligned assistant requires a rigorous post-pretraining stack: Supervised Fine-Tuning (SFT), preference alignment via Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), and efficient parameter-efficient adaptation (LoRA / QLoRA), backed by high-throughput inference serving engines (PagedAttention / vLLM).

1. Theoretical Foundations of Autoregressive Language Modeling & Scaling Laws#

The Autoregressive Objective & Perplexity#

Autoregressive language models decompose the joint probability of an arbitrary token sequence x = (x_1, x_2, …, x_T) into an ordered product of conditional probabilities using the probability chain rule:

P(x) = ∏_{t=1}^T P(x_t | x_1, x_2, …, x_{t-1}; θ)

During pre-training, parameters θ are optimized by minimizing the empirical Cross-Entropy / Negative Log-Likelihood (NLL) loss:

L_{NLL}(θ) = -(1 / T) ∑_{t=1}^T ln P(x_t | x_{<t}; θ)

The primary intrinsic evaluation metric for language modeling quality is Perplexity (PPL), representing the exponential of the cross-entropy loss:

PPL = exp(L_{NLL}) = exp( -(1 / T) ∑_{t=1}^T ln P(x_t | x_{<t}) )

• Information-Theoretic Meaning: Perplexity measures the average branch factor of the model—the effective number of equally likely tokens the model is choosing between at each step. A lower perplexity indicates higher certainty and superior generative fidelity.

+-----------------------------------------------------------------------------------+
|                        AUTOREGRESSIVE PRE-TRAINING PIPELINE                       |
+-----------------------------------------------------------------------------------+
|  Input Tokens:   [ "The", " distributed", " consensus", " algorithm" ]            |
|  Forward Pass:   Project through L Transformer Decoder Layers (Causal Attention)   |
|  Target Tokens:  [ " distributed", " consensus", " algorithm", " is" ]            |
|  Loss:           Cross-Entropy across |V| Softmax Vocabulary                      |
+------------------------------------------+----------------------------------------+
                                           |
                                           v
+-----------------------------------------------------------------------------------+
|                          THE POST-PRETRAINING ALIGNMENT STACK                     |
+-----------------------------------------------------------------------------------+
|  1. FOUNDATION PRE-TRAINING: Massive web crawl (Trillions of tokens, 10^24 FLOPs)  |
|  2. SUPERVISED FINE-TUNING (SFT): High-quality (User, Assistant) dialogues        |
|  3. PREFERENCE ALIGNMENT: RLHF (PPO) or Direct Preference Optimization (DPO)      |
|  4. DEPLOYMENT & SERVING: 4-bit Quantization (AWQ/GPTQ) + PagedAttention (vLLM)   |
+-----------------------------------------------------------------------------------+

Empirical Scaling Laws: Kaplan vs. Chinchilla#

How should an engineering team allocate a fixed compute budget C between model size (number of parameters N) and data volume (number of training tokens D)?

In 2020, OpenAI (Kaplan et al.) proposed that cross-entropy loss follows a power law, arguing that scaling parameter count N was substantially more effective than scaling token volume D (N ∝ C^{0.73}, D ∝ C^{0.27}). Consequently, early models were vastly under-trained relative to their parameter capacity (e.g., GPT-3 had 175 billion parameters but was trained on only 300 billion tokens).

In 2022, DeepMind (Hoffmann et al., "Chinchilla") identified statistical flaws in Kaplan's learning rate schedule and established Compute-Optimal Scaling Laws:

1. Equal Proportional Scaling: For compute-optimal training, model parameter count N and token count D must scale in equal proportions:

N ∝ C^{0.5}, D ∝ C^{0.5}

1. The Chinchilla Ratio: The optimal ratio of training tokens to model parameters is approximately:

D_{optimal} ≈ 20 × N

1. Compute Budget Formulation: The total floating-point operations (FLOPs) required for training an autoregressive transformer is given by:

C ≈ 6 N D

(where 2 FLOPs per parameter are consumed in the forward pass, and 4 FLOPs in the backward pass).

Chinchilla Empirical Verification: A 70-billion parameter model trained on 1.4 trillion tokens (1:20 ratio) consistently outperforms a 175-billion parameter model trained on 300 billion tokens, while requiring over 50\% less VRAM at inference time.

2. Post-Pretraining Alignment: SFT, RLHF, and DPO#

A raw base model pre-trained on the internet is an uncensored document completer. If prompted with "How do I configure an Nginx reverse proxy?", it might complete the prompt with a second question: "And how do I configure Apache?".

Supervised Fine-Tuning (SFT)#

SFT conditions the base model on conversational dialogue structures using instruction-response pairs formatted with standardized role tokens (<|im_start|>user ... <|im_end|> and <|im_start|>assistant ... <|im_end|>).

• During SFT, backpropagation loss is masked: gradients are calculated strictly over the assistant's tokens, preventing the model from unlearning how to process user prompts.
• While SFT teaches formatting and tool-use syntax, SFT alone suffers from Distributional Shift: the model is trained purely on ground-truth demonstration tokens, making it fragile when generating long, unguided autoregressive chains during live deployment.

Reinforcement Learning from Human Feedback (RLHF)#

Pioneered by Christiano et al. (2017) and Stiennon et al. (2020), RLHF aligns language models with human values (Helpfulness, Honesty, Harmlessness - the 3Hs):

+-----------------------------------------------------------------------------------+
|                        CLASSIC RLHF (PPO) WORKFLOW                                |
+-----------------------------------------------------------------------------------+
|  Step 1: Preference Data Collection: Human annotators label y_w > y_l for prompt x|
|  Step 2: Train Reward Model r_ψ(x, y) using Bradley-Terry preference loss         |
|  Step 3: RL Fine-Tuning via PPO:                                                  |
|          Maximize r_ψ(x, y) - β * D_KL( π_θ(y|x) || π_ref(y|x) )                  |
|          (Actor-Critic loop with Reference Model, Value Model, and Reward Model)  |
+-----------------------------------------------------------------------------------+

The Bradley-Terry Preference Model:

Given a prompt x and two candidate responses (y_w, y_l) where human raters prefer y_w over y_l (y_w ≻ y_l), the probability of preference is modeled as:

P(y_w ≻ y_l | x) = σ(r_ψ(x, y_w) - r_ψ(x, y_l))

where r_ψ(x, y) is a learned scalar reward model.

The PPO Objective with KL-Divergence Penalty:

The generative policy π_θ is fine-tuned to maximize expected reward while penalizing divergence from the frozen reference SFT model π_ref:

max_θ E_{x ~ D, y ~ π_θ} [ r_ψ(x, y) - β D_KL(π_θ(y | x) || π_ref(y | x)) ]

The Kullback-Leibler (KL) divergence penalty (β D_KL) prevents Reward Hacking—a failure mode where the policy exploits flaws in the reward model to generate nonsensical text that receives high scores.

Direct Preference Optimization (DPO)#

While RLHF via PPO works effectively, it is notoriously unstable: training requires maintaining four separate LLMs simultaneously in GPU VRAM (the Active Policy π_θ, the Reference Policy π_ref, the Reward Model r_ψ, and the PPO Critic Value Network V_ϕ), leading to memory exhaustion and brittle hyperparameter tuning.

Rafailov et al. (2023) revolutionized alignment by deriving Direct Preference Optimization (DPO). By mathematically reparameterizing the ground-truth reward function in terms of the optimal policy:

r(x, y) = β ln (π*(y | x) / π_ref(y | x)) + β ln Z(x)

DPO substitutes this identity directly into the Bradley-Terry preference objective, completely eliminating the need for a separate reward model or RL reinforcement loop!

The DPO Closed-Form Loss Function:

L_{DPO}(θ; π_ref) = -E_{(x, y_w, y_l) ~ D} [ ln σ ( β ln (π_θ(y_w | x) / π_ref(y_w | x)) - β ln (π_θ(y_l | x) / π_ref(y_l | x)) ) ]

• Mechanics: DPO increases the log-likelihood of the preferred response y_w while decreasing the log-likelihood of the dispreferred response y_l, weighted dynamically by how confident the reference model was. DPO is stable, computationally lightweight, and requires only standard cross-entropy-style gradient descent.

3. Parameter-Efficient Fine-Tuning (PEFT): LoRA and QLoRA#

Full parameter fine-tuning of modern LLMs is computationally prohibitive: updating all 70 billion parameters of a LLaMA model in 16-bit precision requires over 1.1 TB of GPU VRAM when accounting for AdamW optimizer states (16 bytes per parameter: 2 bytes weight, 2 bytes gradient, 4 bytes first momentum, 4 bytes second momentum, 4 bytes master copy).

Low-Rank Adaptation (LoRA)#

Aghajanyan et al. (2020) proved that the weight updates during task adaptation have a very low intrinsic dimension. Hu et al. (2021) leveraged this insight in Low-Rank Adaptation (LoRA):

+-----------------------------------------------------------------------------------+
|                        LOW-RANK ADAPTATION (LoRA) ARCHITECTURE                    |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|                   Input Vector: x in R^{1 x d}                                    |
|                         /            \                                            |
|                        /              \                                           |
|                       v                v                                          |
|            [ FROZEN BASE WEIGHT ]    [ LORA MATRIX A ] in R^{d x r}               |
|               W_0 in R^{d x k}                 |                                  |
|               (Requires zero                  v                                   |
|                optimizer states!)    [ LORA MATRIX B ] in R^{r x k}               |
|                       \                        |                                  |
|                        \                       v                                  |
|                         \             Scaled by (alpha / r)                       |
|                          \            Delta_W = (alpha / r) * A * B               |
|                           \                    /                                  |
|                            v                  v                                   |
|                             (+) <------------+                                    |
|                              |                                                    |
|                              v                                                    |
|                   Output Vector: h in R^{1 x k}                                   |
|                   h = x * W_0 + (alpha / r) * x * A * B                           |
+-----------------------------------------------------------------------------------+

Instead of updating the full parameter matrix W_0 ∈ R^{d × k}, LoRA decomposes the weight update Δ W into the product of two low-rank matrices:

W = W_0 + Δ W = W_0 + (α / r) B A

where:

• A ∈ R^{r × d} is initialized from a Gaussian distribution N(0, σ^2).
• B ∈ R^{k × r} is initialized to zero (B = 0), ensuring Δ W = 0 at step zero.
• r ≪ min(d, k) is the low-rank dimension (typically 4 ≤ r ≤ 64).
• α is a constant scaling hyperparameter.

VRAM Savings: By freezing W_0, optimizer states are stored only for A and B, reducing trainable parameter count by > 99.8\% and slashing GPU memory requirements from 8 GPUs down to a single consumer GPU. At inference time, the adapter weights can be permanently folded back into the base matrix (W_{final} = W_0 + (α / r) B A), incurring zero additional latency.

QLoRA: 4-Bit NormalFloat Quantization#

Dettmers et al. (2023) established QLoRA, enabling the fine-tuning of a 65B parameter model on a single 48 GB GPU through three innovations:

1. 4-bit NormalFloat (NF4): An information-theoretically optimal quantile quantization data type for normally distributed neural network weights.
2. Double Quantization (DQ): Quantizes the quantization constants themselves, saving 0.37 bits per parameter.
3. Paged Optimizers: Leverages CUDA unified memory to dynamically page optimizer states between GPU VRAM and CPU system RAM during memory spikes.

4. Inference Serving: KV Cache & PagedAttention#

During autoregressive generation, generating each new token requires computing self-attention against all preceding tokens in the context window. Recomputing query, key, and value vectors for all previous tokens at every step would result in quadratic O(T^2) computational complexity per token.

The KV Cache#

To eliminate redundant computation, inference engines maintain a Key-Value Cache (KV Cache):

• For all previous tokens t' < t, the key vectors k_{t'} and value vectors v_{t'} are stored permanently in GPU memory.
• At step t, the model projects only the single newly generated token into q_t, k_t, v_t.
• k_t and v_t are appended to the KV cache.
• Attention is computed by taking the dot product of q_t against all cached keys:

Attention_t = Softmax( (q_t * K_cache^T / √d_k) ) V_{cache}

• Computational Complexity: Reduced from O(T^2) to O(T) operations per generated token.

PagedAttention (vLLM)#

While the KV cache accelerates inference, it creates a massive memory bottleneck:

• Memory per token:

Memory_{KV} = 2 × B × L × h × d_k × bytes per element

• Fragmentation Pathology: Traditional serving systems allocate contiguous virtual memory blocks sized to the maximum possible context length (e.g., 8,192 tokens). Because user requests vary in length, over 60\% to 80\% of GPU memory is wasted on internal and external memory fragmentation.

PagedAttention (Kwon et al., 2023, vLLM):

Inspired by virtual memory paging in operating systems, PagedAttention stores the KV cache in non-contiguous physical memory blocks:

• Memory is partitioned into fixed-size physical blocks (e.g., 16 tokens per block).
• A logical-to-physical Block Table translates sequence offsets to physical memory addresses on the fly.
• Memory waste is reduced to < 4\%, allowing serving systems to increase concurrent request batching capacity by 2× to 4×, drastically slashing inference serving costs.

5. University-Grade Checkpoint Practice Problems#

Problem 1: Chinchilla Compute-Optimal Training Allocation Math#

Context: An AI enterprise receives a dedicated computational compute budget of:

C = 1.80 × 10^{23} FLOPs (180 zettaFLOPs)

to pre-train a modern autoregressive transformer decoder.

According to DeepMind's Chinchilla empirical scaling laws (Hoffmann et al., 2022):

1. The floating-point operations required to train an autoregressive transformer are parameterized as:

C ≈ 6 N D

where N is the number of non-embedding model parameters, and D is the total volume of training tokens.

1. Compute-optimal efficiency mandates the empirical ratio:

D_{opt} ≈ 20 × N_{opt}

Task:

1. Substitute the Chinchilla token-to-parameter ratio into the compute equation and derive the formula for the optimal parameter count N_{opt} and optimal training tokens D_{opt} as a function of compute budget C.
2. Calculate the exact compute-optimal parameter count N_{opt} (in billions of parameters) and token count D_{opt} (in trillions of tokens) for the allocated budget C = 1.80 × 10^{23} FLOPs.
3. Suppose an uninformed engineering team instead builds a model with N_{sub} = 70 billion parameters using the identical C = 1.80 × 10^{23} FLOPs budget. Calculate the number of tokens D_{sub} this 70B model would be trained on, and calculate its token-to-parameter ratio.
4. Explain why the compute-optimal model will achieve lower cross-entropy validation loss and provide superior inference economics compared to the 70B model.

Solution:

1. Derivation of Compute-Optimal Formulation:

C = 6 N_{opt} D_{opt}

Substituting D_{opt} = 20 N_{opt}:

C = 6 N_{opt} (20 N_{opt}) = 120 N_{opt}^2

Solving for N_{opt}:

N_{opt}^2 = (C / 120) ⟹ N_{opt} = √((C / 120))

Solving for D_{opt}:

D_{opt} = 20 √((C / 120)) = √(400 × (C / 120)) = √((10C / 3))

1. Numerical Compute-Optimal Values:

* Evaluating N_{opt}:

N_{opt} = √( (1.80 × 10^23) / 120 ) = √(1.50 × 10^{21)} = √(15 × 10^{20)} ≈ 3.87298 × 10^{10} parameters

N_{opt} ≈ 38.73 billion parameters

* Evaluating D_{opt}:

D_{opt} = 20 × N_{opt} = 20 × 3.87298 × 10^{10} ≈ 7.74597 × 10^{11} tokens

D_{opt} ≈ 774.6 billion tokens (0.775 trillion tokens)

1. Suboptimal 70B Model Training Allocation:

* Parameters N_{sub} = 70 × 10^9.

* Given C = 6 N D:

D_{sub} = (C / (6 * N_sub)) = (1.80 × 10^23 / (6 × 70 × 10^9)) = (1.80 × 10^23 / (4.20 × 10^11)) ≈ 4.2857 × 10^{11} tokens

D_{sub} ≈ 428.6 billion tokens

* Token-to-parameter ratio:

Ratio = (428.6 10^9 / (70 10^9)) ≈ 6.12 tokens per parameter

1. Performance & Economic Comparison:

Validation Loss*: The 70B model trained on only 6.12 tokens per parameter is severely under-trained (undertrained by over 3.2× relative to the optimal 20:1 ratio). Chinchilla scaling proofs establish that the 38.7B model trained on 774.6B tokens achieves a lower validation loss L for the identical compute budget.

Inference Economics*: Serving the 70B model requires at least two 80 GB GPUs in FP16 (140 GB weights alone), whereas the 38.7B model requires only one 80 GB GPU (77.4 GB weights), cutting operational serving costs in half while delivering lower latency per token.

Problem 2: LoRA Parameter Reduction & Optimizer Memory Footprint Sizing#

Context: An infrastructure engineer evaluates fine-tuning a 70-billion parameter foundation model (N = 70 × 10^9).

Architecture Parameters:

• Hidden dimension: d_{model} = 8,192.
• Number of transformer decoder layers: L = 80 layers.
• Each layer contains 4 standard self-attention projection matrices: W_Q, W_K, W_V, W_O, each of size d_{model} × d_{model} = 8,192 × 8,192.
• The Feed-Forward Network (FFN) layers are left unmodified.

We compare two training regimes:

• Regime A (Full Parameter Fine-Tuning across Attention Projections):

* All attention weights are updated using the standard AdamW optimizer in FP16/FP32 mixed precision:

* Model parameter weights: 16-bit FP16 (2 bytes/param).

* Gradients: 16-bit FP16 (2 bytes/param).

* AdamW First Momentum (m_t): 32-bit FP32 (4 bytes/param).

* AdamW Second Momentum (v_t): 32-bit FP32 (4 bytes/param).

* FP32 Master Copy of Weights: 32-bit FP32 (4 bytes/param).

* Total memory per trainable parameter: M_{full} = 2 + 2 + 4 + 4 + 4 = 16 bytes per parameter.

• Regime B (Low-Rank Adaptation - LoRA):

* Every attention projection matrix W ∈ R^{d × d} is augmented with LoRA adapters:

Δ W = (α / r) B A

where A ∈ R^{r × d} and B ∈ R^{d × r} with rank r = 16 and scaling α = 32.

* Base model weights W_0 are frozen in 16-bit precision (2 bytes/param) and require zero optimizer memory.

* Only adapter matrices A and B are trained using AdamW (16 bytes per trainable parameter).

Task:

1. Calculate the total number of parameters in the attention projection matrices across all 80 layers.
2. Calculate the total physical GPU VRAM footprint (in Gigabytes) required for weights and optimizer states under Regime A (Full Fine-Tuning).
3. Calculate the total number of trainable adapter parameters under Regime B (LoRA with r = 16).
4. Calculate the total physical GPU VRAM footprint (in Gigabytes) under Regime B.
5. Compute the percentage reduction in trainable parameters and VRAM achieved by LoRA.

Solution:

1. Total Attention Parameter Count:

* Parameters per projection matrix:

P_{matrix} = 8,192 × 8,192 = 67,108,864 parameters

* Matrices per layer = 4 (W_Q, W_K, W_V, W_O).

* Parameters per layer:

P_{layer} = 4 × 67,108,864 = 268,435,456 parameters

* Across L = 80 layers:

N_{attn} = 80 × 268,435,456 = 21,474,836,480 parameters (≈ 21.47 billion)

1. Regime A (Full Fine-Tuning) VRAM Footprint:

* Memory multiplier: 16 bytes per parameter.

* Total memory:

Mem_{full} = 21,474,836,480 × 16 bytes = 343,597,383,680 bytes

* Converting to Gigabytes (1024^3):

Mem_{full} = (343,597,383,680 / 1,073,741,824) = 320.0 GiB

* Just fine-tuning the attention layers requires 320 GB of VRAM (requiring an 8-GPU cluster of 80 GB A100s with ZeRO-3 sharding).

1. Regime B (LoRA) Trainable Parameters:

* For each matrix W ∈ R^{d × d}, LoRA introduces:

* Matrix A ∈ R^{16 × 8,192} ⟹ 16 × 8,192 = 131,072 params.

* Matrix B ∈ R^{8,192 × 16} ⟹ 8,192 × 16 = 131,072 params.

* Total per matrix:

P_{lora, matrix} = 131,072 + 131,072 = 262,144 parameters

* Across 4 matrices per layer and 80 layers:

N_{lora} = 80 × 4 × 262,144 = 320 × 262,144 = 83,886,080 parameters (≈ 83.89 million)

1. Regime B (LoRA) VRAM Footprint:

Base Model Weights (Frozen)*:

21,474,836,480 × 2 bytes = 42,949,672,960 bytes = 40.0 GiB

LoRA Adapter Weights + AdamW States*:

83,886,080 × 16 bytes = 1,342,177,280 bytes = 1.25 GiB

* Total VRAM under LoRA:

Mem_{LoRA} = 40.0 GiB + 1.25 GiB = 41.25 GiB

1. Comparative Analysis:

* Trainable parameter reduction:

(N_lora / N_attn) = (83,886,080 / 21,474,836,480) = (262,144 / 67,108,864) = (2r / d_model) = (32 / 8,192) = (1 / 256) ≈ 0.003906

Trainable parameters are reduced by 99.61\%.

* VRAM reduction:

(1 - (41.25 GiB / 320.0 GiB)) × 100\% = (1 - 0.1289) × 100\% ≈ 87.11\%

* LoRA slashes memory by 87.1\%, enabling adaptation on a single 48 GB workstation GPU rather than a distributed 8-GPU cluster.

Problem 3: DPO Loss Gradient & Implicit Reward Margin Derivation#

Context: In Direct Preference Optimization (DPO), the loss for a single prompt x with preferred completion y_w and dispreferred completion y_l is defined as:

L_{DPO}(θ; π_ref) = -ln σ ( β ln (π_θ(y_w | x) / π_ref(y_w | x)) - β ln (π_θ(y_l | x) / π_ref(y_l | x)) )

Let the Implicit Reward of an arbitrary completion y be defined as:

r̂_θ(x, y) = β ln (π_θ(y | x) / π_ref(y | x))

Let the Implicit Reward Margin be defined as:

M̂_θ(x, y_w, y_l) = r̂_θ(x, y_w) - r̂_θ(x, y_l) = β ( ln (π_θ(y_w | x) / π_ref(y_w | x)) - ln (π_θ(y_l | x) / π_ref(y_l | x)) )

Thus, the DPO loss simplifies to:

L_{DPO} = -ln σ(M̂_θ)

Task:

1. Using the chain rule and the identity (d / dz) ln σ(z) = 1 - σ(z), derive the exact gradient of L_{DPO} with respect to model parameters θ:

∇_θ L_{DPO}

1. Factor the gradient into a scalar weighting term W(θ) and a directional vector term.
2. Explain how the scalar weight W(θ) dynamically modulates gradient descent:

* What happens to the gradient when the model strongly prefers the correct answer (r̂_θ(x, y_w) ≫ r̂_θ(x, y_l))?

* What happens to the gradient when the model incorrectly assigns higher reward to the dispreferred answer (r̂_θ(x, y_l) ≫ r̂_θ(x, y_w))?

Solution:

1. DPO Gradient Derivation:

* Applying the chain rule:

∇_θ L_{DPO} = (d(-ln σ(M̂_θ)) / d M̂_θ) ∇_θ M̂_θ

* Since (d(-ln σ(z)) / dz) = -(1 - σ(z)) = σ(-z) - 1:

∇_θ L_{DPO} = -(1 - σ(M̂_θ)) ∇_θ M̂_θ = -σ(-M̂_θ) ∇_θ M̂_θ

* Evaluating ∇_θ M̂_θ:

M̂_θ = β ( ln π_θ(y_w | x) - ln π_ref(y_w | x) - ln π_θ(y_l | x) + ln π_ref(y_l | x) )

Since π_ref is frozen with respect to θ:

∇_θ M̂_θ = β ( ∇_θ ln π_θ(y_w | x) - ∇_θ ln π_θ(y_l | x) )

* Combining terms:

∇_θ L_{DPO} = -β σ(-M̂_θ) [ ∇_θ ln π_θ(y_w | x) - ∇_θ ln π_θ(y_l | x) ]

1. Factorization into Scalar Weight and Direction:

∇_θ L_{DPO} = -W(θ) ( ∇_θ ln π_θ(y_w | x) - ∇_θ ln π_θ(y_l | x) )

where the scalar weighting factor is:

W(θ) = β σ(-M̂_θ) = β ( 1 - σ(M̂_θ) )

* The directional term increases the likelihood of preferred completion y_w while decreasing the likelihood of dispreferred completion y_l.

1. Dynamic Error-Weighted Modulation:

Case 1: Well-Aligned Model (r̂_θ(x, y_w) ≫ r̂_θ(x, y_l))*:

* Margin M̂_θ → +∞.

* σ(-M̂_θ) = σ(-∞) → 0.

* Scalar weight W(θ) → 0.

Behavior*: The gradient drops to zero. The model does not waste capacity updating parameters on examples it already classifies correctly.

Case 2: Misaligned Model (r̂_θ(x, y_l) ≫ r̂_θ(x, y_w))*:

* Margin M̂_θ → -∞ (the model strongly prefers the wrong answer).

* σ(-M̂_θ) = σ(+∞) → 1.

* Scalar weight W(θ) → β (maximum magnitude).

Behavior*: The gradient delivers a maximum corrective push, rapidly penalizing the bad generation and boosting the preferred completion.

Problem 4: KV Cache Memory Sizing & Maximum Concurrent Serving Concurrency#

Context: An AI deployment engineer architects an enterprise LLM inference service hosting a 13-billion parameter model (N = 13 × 10^9) in half-precision (FP16, 2 bytes per value).

Hardware Deployment:

• The service runs on a server equipped with two NVIDIA H100 SXM GPUs, each with 80 GB of High-Bandwidth Memory (total combined physical VRAM M_{server} = 160 GB).
• Operating system and CUDA runtime buffers reserve M_{cuda} = 6.0 GB.

Model Architecture Constants:

• Number of transformer layers: L = 40 layers.
• Number of attention heads: h = 32 heads.
• Head dimension: d_k = 128.
• Maximum supported sequence context window: T = 4,096 tokens.

Serving Architecture Constraints:

• The serving engine employs PagedAttention (vLLM), reducing memory fragmentation to practically zero (0\%).
• Each user request generates an average response that utilizes the full context window (T = 4,096 tokens).
• During generation, both Key and Value vectors for all past tokens must be preserved in the KV cache across all 40 layers.

Task:

1. Calculate the exact VRAM memory footprint (in GB) required to hold the static model weights in FP16 precision.
2. Calculate the exact KV cache memory footprint (in Megabytes) consumed by a single user request at maximum context length T = 4,096.
3. Calculate the total remaining VRAM memory budget available strictly for KV cache allocation across the two GPUs.
4. Determine the Maximum Concurrent User Concurrency (B_{max})—the maximum number of active user requests the server can process simultaneously in parallel without running OutOfMemory.

Solution:

1. Static Model Weight Footprint:

* Parameters N = 13 × 10^9.

* Precision = 2 bytes per parameter.

* Weight memory:

M_{weights} = 13 × 10^9 × 2 bytes = 26,000,000,000 bytes ≈ 26.0 GB (decimal) (24.21 GiB)

1. KV Cache Footprint Per User Request:

* At each token position, for each layer, for each head, the model stores:

* 1 Key vector of dimension d_k = 128 in FP16 (128 × 2 bytes = 256 bytes).

* 1 Value vector of dimension d_k = 128 in FP16 (128 × 2 bytes = 256 bytes).

* Total per token per head per layer: 256 + 256 = 512 bytes.

* Multiplying by h = 32 heads:

512 bytes × 32 = 16,384 bytes per token per layer = 16 KB

* Multiplying by L = 40 layers:

16,384 bytes × 40 = 655,360 bytes per token across model = 640 KB

* Multiplying by context length T = 4,096 tokens:

M_{KV, user} = 4,096 × 655,360 bytes = 2,684,354,560 bytes

* Converting to Megabytes and Gigabytes:

M_{KV, user} = (2,684,354,560 / 1,024 × 1,024) = 2,560.0 MiB = 2.50 GiB per user

1. Available KV Cache Memory Budget:

* Total server physical VRAM: M_{server} = 160.0 GB = 149.01 GiB (160 × 10^9 bytes).

* Using base-10 gigabytes:

M_{available} = 160.0 GB - M_{weights} - M_{cuda} = 160.0 GB - 26.0 GB - 6.0 GB = 128.0 GB

M_{available} = 128,000,000,000 bytes

1. Maximum Concurrent Concurrency B_{max}:

B_{max} = ⌊ (M_available / M_KV_user) ⌋ = ⌊ (128,000,000,000 bytes / 2,684,354,560 bytes/user) ⌋ = ⌊ 47.68 ⌋ = 47 concurrent requests

Result: The dual-H100 server can sustain exactly 47 concurrent users* generating full 4,096-token responses simultaneously.

Architectural Optimization*: If the team adopts Grouped-Query Attention (GQA) with 8 key-value heads instead of 32 (a 4× KV reduction, standard in LLaMA-3), KV cache size drops to 0.625 GiB per user, quadrupling maximum serving concurrency to 47 × 4 = 188 concurrent users on the identical hardware.