Continuous Densities, Moment Generating Functions & the Central Limit Theorem

In computing systems, financial modeling, machine learning, and physical science, many random variables cannot be categorized into discrete integer buckets. Execution latencies, temperatures, stock price returns, and sensor voltages operate along continuous intervals of the real number line. Transitioning from discrete probability masses to continuous probability densities requires the analytical machinery of integration, moment generating functions, and asymptotic limit theorems.

1. Continuous Random Variables & Probability Density Geometry#

For a discrete random variable, probability is assigned to specific individual values: P(X = x) > 0. For a continuous random variable X, the probability of observing any exact, infinitely precise real number is identically zero:

P(X = x) = 0 for any specific x in R.

(Just as a point on a line has zero length, an exact point in a continuous distribution has zero probability mass). Instead, probability is distributed continuously across intervals via a Probability Density Function (PDF).

The Axioms of a Probability Density Function#

A function f_X: R → [0, ∞) is a valid Probability Density Function for continuous random variable X if it satisfies two criteria:

1. Non-Negativity: f_X(x) >= 0 for all x in R.
2. Normalization (Total Area = 1):

integral_{-∞}^∞ f_X(x) dx = 1.

The probability that X falls within an interval [a, b] is given by the definite area integral beneath the density curve:

P(a <= X <= b) = integral_a^b f_X(x) dx.

Note: Because exact individual points have zero probability, boundary endpoints do not alter the probability:

P(a <= X <= b) = P(a < X <= b) = P(a <= X < b) = P(a < X < b).

+-----------------------------------------------------------------------------+
|                  CONTINUOUS PROBABILITY DENSITY INTEGRAL                    |
+-----------------------------------------------------------------------------+
|    Density f(x)                                                             |
|         ^                                                                   |
|         |                     P(a <= X <= b)                                |
|         |                        /=======\                                  |
|         |                       /=========\                                 |
|         |                      /===========\                                |
|         |                     /=============\                               |
|         +--------------------+---------------+------------------------> x   |
|                              a               b                              |
|                              <--- Area ------>                              |
|                                                                             |
|    Total Area under entire curve: integral_{-infinity}^{+infinity} f(x) dx = 1|
+-----------------------------------------------------------------------------+

The Cumulative Distribution Function (CDF)#

The Cumulative Distribution Function (CDF), denoted F_X(x), represents the accumulated probability that random variable X takes a value less than or equal to x:

F_X(x) = P(X <= x) = integral_{-∞}^x f_X(t) dt.

Fundamental Properties of the CDF:

1. Monotonicity: F_X(x) is non-decreasing; if x1 < x2, then F_X(x1) <= F_X(x2).
2. Boundary Limits: lim_{x → -∞} F_X(x) = 0 and lim_{x → +∞} F_X(x) = 1.
3. Interval Evaluation: P(a < X <= b) = F_X(b) - F_X(a).
4. Fundamental Link via Calculus:

By the Fundamental Theorem of Calculus, the PDF is the exact derivative of the CDF:

f_X(x) = d/dx [ F_X(x) ].

Three Canonical Continuous Distributions#

1. The Uniform Distribution U(a, b):

f(x) = 1 / (b - a) for x in [a, b], and 0 elsewhere.

Mean: E[X] = (a + b) / 2.

Variance: Var(X) = (b - a)^2 / 12.

1. The Exponential Distribution Exp(λ):

Models the waiting time between Poisson event arrivals with rate parameter λ > 0:

f(x) = λ · e^(-λ x) for x >= 0, and 0 for x < 0.

CDF: F(x) = 1 - e^(-λ x).

Mean: E[X] = 1 / λ.

Variance: Var(X) = 1 / λ^2.

The Memoryless Property:

P(X > s + t | X > s) = P(X > t).

Given that a server has operated for s hours without failure, the probability that it survives an additional t hours is identical to a brand new server!

1. The Normal (Gaussian) Distribution N(μ, σ^2):

The universal bell curve parameterized by mean μ and standard deviation σ > 0:

f(x) = [ 1 / (σ · sqrt(2π)) ] · e^(-(x - μ)^2 / (2 σ^2)).

Standard Normal Distribution Z ~ N(0, 1):

φ(z) = (1 / sqrt(2π)) · e^(-z^2 / 2).

CDF denoted Φ(z).

The 68-95-99.7 Empirical Rule:

- P(μ - σ <= X <= μ + σ) ≈ 68.27%

- P(μ - 2σ <= X <= μ + 2σ) ≈ 95.45%

- P(μ - 3σ <= X <= μ + 3σ) ≈ 99.73%

2. Mathematical Moments & Moment Generating Functions (MGF)#

To describe the shape of an arbitrary probability distribution beyond its mean, statistics analyzes higher-order moments.

Raw Moments vs. Central Moments#

• The k-th Raw Moment:

α_k = E[X^k] = integral_{-∞}^∞ x^k · f_X(x) dx.

The 1st raw moment is the mean: μ = E[X].

• The k-th Central Moment:

μ_k = E[(X - μ)^k] = integral_{-∞}^∞ (x - μ)^k · f_X(x) dx.

- 2nd Central Moment: Variance σ^2 = E[(X - μ)^2] = E[X^2] - (E[X])^2.

- 3rd Standardized Moment: Skewness γ1 = E[(X - μ)^3] / σ^3. Measures distribution asymmetry. (Positive skew = long right tail; Negative skew = long left tail).

- 4th Standardized Moment: Kurtosis γ2 = E[(X - μ)^4] / σ^4. Measures tail heaviness. (Excess kurtosis = γ2 - 3; Normal distribution has excess kurtosis = 0).

The Moment Generating Function (MGF)#

Evaluating moments directly through high-order integrals is frequently analytically intractable. The Moment Generating Function (MGF) packages all moments into a single analytical function.

Definition: For random variable X, the Moment Generating Function M_X(t) is defined as:

M_X(t) = E[e^(t X)] = integral_{-∞}^∞ e^(t x) · f_X(x) dx

provided this expectation is finite in an open interval around t = 0 (for |t| < h).

Generating Moments via Derivatives#

Using the Maclaurin series expansion of the exponential function e^(t X):

e^(t X) = 1 + t · X + (t^2 · X^2) / 2! + (t^3 · X^3) / 3! + ... = sum(k=0 to ∞, (t^k · X^k) / k!).

Applying linearity of expectation:

M_X(t) = E[ sum(k=0 to ∞, (t^k · X^k) / k!) ] = sum(k=0 to ∞, (t^k / k!) · E[X^k]).

Taking the k-th derivative of M_X(t) with respect to t and setting t = 0:

E[X^k] = M_X^(k)(0) = d^k M_X(t) / dt^k |_{t=0}.

• First Moment (Mean): E[X] = M'_X(0).
• Second Moment: E[X^2] = M''_X(0).
• Variance: Var(X) = M''_X(0) - (M'_X(0))^2.

Properties of Moment Generating Functions#

1. MGF of Linear Transformations:

For Y = a · X + b:

M_Y(t) = E[e^(t(a X + b))] = e^(b t) · E[e^((a t) X)] = e^(b t) · M_X(a · t).

1. MGF of Independent Sums (Multiplication Property):

If X and Y are statistically independent random variables:

M_{X + Y}(t) = E[e^(t(X + Y))] = E[e^(t X) · e^(t Y)] = E[e^(t X)] · E[e^(t Y)] = M_X(t) · M_Y(t).

The MGF of a sum of independent random variables is simply the product of their individual MGFs!

1. The MGF Uniqueness Theorem:

If two random variables X and Y have MGFs that agree on an open neighborhood around t = 0 (M_X(t) = M_Y(t) for all |t| < h), then X and Y possess the EXACT same cumulative probability distribution (F_X(x) = F_Y(x) everywhere).

3. Tail Concentration Inequalities: Markov, Chebyshev & Chernoff#

When the exact probability distribution of a random variable is unknown, concentration inequalities provide rigorous upper bounds on the probability that a variable deviates far from its expected value.

1. Markov's Inequality#

Theorem: Let X be a non-negative random variable (X >= 0). For any positive constant a > 0:

P(X >= a) <= E[X] / a.

Rigorous Analytical Proof:

Let I_{X >= a} be the indicator random variable that equals 1 if X >= a, and 0 otherwise.

Notice that for all X >= 0:

a · I_{X >= a} <= X.

(If X < a, left side is 0 <= X. If X >= a, left side is a <= X).

Taking the expectation on both sides:

E[a · I_{X >= a}] <= E[X]

a · E[I_{X >= a}] <= E[X]

a · P(X >= a) <= E[X]

P(X >= a) <= E[X] / a.

2. Chebyshev's Inequality#

Theorem: Let X be any random variable with finite mean μ = E[X] and finite variance σ^2 = Var(X). For any k > 0:

P(|X - μ| >= k · σ) <= 1 / k^2.

Proof:

Apply Markov's inequality to the non-negative random variable Y = (X - μ)^2 with constant a = (k · σ)^2:

P(|X - μ| >= k · σ) = P((X - μ)^2 >= k^2 · σ^2) <= E[(X - μ)^2] / (k^2 · σ^2) = σ^2 / (k^2 · σ^2) = 1 / k^2.

Implications:

• At least 75% of all data points must lie within 2 standard deviations of the mean (k = 2: 1 - 1/4 = 75%).
• At least 88.89% must lie within 3 standard deviations (k = 3: 1 - 1/9 = 88.89%).

This holds for ANY distribution, no matter how asymmetric or non-normal!

3. Chernoff Bounds#

Chernoff bounds apply Markov's inequality exponentially to the Moment Generating Function:

For any t > 0:

P(X >= a) = P(e^(t X) >= e^(t a)) <= e^(-t a) · E[e^(t X)] = e^(-t a) · M_X(t).

Minimizing over t > 0 yields the sharpest exponential tail bound:

P(X >= a) <= inf_{t > 0} [ e^(-t a) · M_X(t) ].

Chernoff bounds provide exponential concentration decay (e^(-c · n)), forming the mathematical foundation of randomized algorithm analysis (e.g., QuickSort depth, Bloom filters).

4. The Law of Large Numbers (LLN)#

The Law of Large Numbers establishes that empirical sample averages converge deterministically to the true theoretical population expectation as sample size grows.

The Sample Mean Random Variable#

Let X1, X2, ..., Xn be independent and identically distributed (i.i.d.) random variables drawn from a population with mean μ = E[X_i] and variance σ^2 = Var(X_i).

The sample mean is defined as:

X_bar_n = (1 / n) · sum(i=1 to n, X_i).

Moments of the Sample Mean:

• Expectation: E[X_bar_n] = E[(1/n) sum X_i] = (1/n) sum E[X_i] = (1/n)(n · μ) = μ. (The sample mean is an unbiased estimator).
• Variance: Var(X_bar_n) = Var((1/n) sum X_i) = (1 / n^2) sum Var(X_i) = (1 / n^2)(n · σ^2) = σ^2 / n.

Notice that the variance of the sample mean shrinks proportionally to 1/n!

The Weak Law of Large Numbers (WLLN)#

Theorem: For any arbitrary tolerance margin ε > 0:

lim_{n → ∞} P(|X_bar_n - μ| >= ε) = 0.

The probability of the sample average deviating from the true mean by more than ε vanishes to zero as n → ∞. (Convergence in Probability: X_bar_n →^P μ).

Analytical Proof via Chebyshev's Inequality:

Apply Chebyshev's inequality to X_bar_n:

P(|X_bar_n - μ| >= ε) <= Var(X_bar_n) / ε^2 = (σ^2 / n) / ε^2 = σ^2 / (n · ε^2).

Taking the limit as n → ∞:

lim_{n → ∞} [ σ^2 / (n · ε^2) ] = 0.

This completes the proof.

The Strong Law of Large Numbers (SLLN)#

Theorem (Emil Borel & Andrey Kolmogorov):

The sample average converges to the population mean with probability 1 (Almost Sure Convergence):

P(lim_{n → ∞} X_bar_n = μ) = 1.

5. The Central Limit Theorem (CLT) & Analytical Proof via MGFs#

While the Law of Large Numbers states that the sample mean collapses onto a single point μ, the Central Limit Theorem reveals the shape of the fluctuation distribution around μ.

Formal Statement (Lindeberg-Lévy Central Limit Theorem)#

Theorem: Let X1, X2, ..., Xn be an i.i.d. sequence of random variables with finite mean μ and finite non-zero variance σ^2 < ∞.

Define the standardized sample mean Z_n:

Z_n = [ X_bar_n - μ ] / [ σ / sqrt(n) ] = [ sum(i=1 to n, (X_i - μ)) ] / [ σ · sqrt(n) ].

As sample size n approaches infinity, the cumulative distribution of Z_n converges point-wise to the standard normal distribution N(0, 1):

lim_{n → ∞} P(Z_n <= z) = Φ(z) = integral_{-∞}^z (1 / sqrt(2π)) · e^(-u^2 / 2) du.

In shorthand notation: Z_n →^d N(0, 1).

The Practical Meaning of the CLT:

The sum or average of a large number of independent random variables will be approximately normally distributed, REGARDLESS of the underlying distribution of the individual variables!

Whether the raw data is Uniform, Poisson, Exponential, Bimodal, or Bernoulli—averaging n >= 30 samples invariably transforms the distribution into a Gaussian bell curve.

+-----------------------------------------------------------------------------+
|                      THE CENTRAL LIMIT THEOREM EFFECT                       |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Raw Data: Uniform U(0, 1)                      Sample Means (n = 40)       |
|       ^                                                ^                    |
|       |  +--------------------+                        |         .---.      |
|       |  |                    |                        |        /     \     |
|       |  |   Flat Density     |                        |       /       \    |
|       |  |                    |                        |     .'         `.  |
|       +--+--------------------+---> x                  +----+-------------+>|
|          0                    1                             0.4   0.5   0.6 |
|                                                                             |
|  As n grows, any finite-variance distribution converges to Gaussian N(0, 1)!|
+-----------------------------------------------------------------------------+

Complete Analytical Proof of the CLT via MGFs#

1. Standardize individual components:

Define Y_i = (X_i - μ) / σ.

Then E[Y_i] = 0 and Var(Y_i) = E[Y_i^2] = 1.

The standardized sum Z_n is:

Z_n = (1 / sqrt(n)) · sum(i=1 to n, Y_i).

1. Express MGF of Z_n in terms of MGF of Y:

M_{Z_n}(t) = E[e^(t · Z_n)] = E[e^((t / sqrt(n)) · sum Y_i)].

Because the Y_i are independent:

M_{Z_n}(t) = prod_{i=1}^n E[e^((t / sqrt(n)) · Y_i)] = [ M_Y(t / sqrt(n)) ]^n.

1. Taylor expand M_Y(u) around u = 0:

M_Y(u) = M_Y(0) + M'_Y(0) · u + (1/2) M''_Y(0) · u^2 + o(u^2).

Recall:

- M_Y(0) = 1

- M'_Y(0) = E[Y] = 0

- M''_Y(0) = E[Y^2] = 1.

Therefore:

M_Y(u) = 1 + 0 + (1/2)(1) · u^2 + o(u^2) = 1 + u^2 / 2 + o(u^2).

1. Substitute u = t / sqrt(n):

M_{Z_n}(t) = [ 1 + (t / sqrt(n))^2 / 2 + o(t^2 / n) ]^n

= [ 1 + (t^2 / 2) / n + o(t^2 / n) ]^n.

1. Evaluate limit as n → ∞ using Euler's limit lim_{n → ∞} (1 + x/n)^n = e^x:

lim_{n → ∞} M_{Z_n}(t) = e^(t^2 / 2).

1. Inversion via Uniqueness Theorem:

The function M(t) = e^(t^2 / 2) is the unique Moment Generating Function of a standard normal random variable N(0, 1).

By the MGF Uniqueness Theorem, Z_n converges in distribution to N(0, 1).

This completes the proof.

6. Systems Performance, Simulation & Data Engineering Applications#

The Central Limit Theorem and continuous density models govern real-world infrastructure and data systems.

1. High-Fan-Out Microservices & The Tyranny of Tail Latency#

In distributed architectures (Google, Amazon, Netflix), a single front-end request triggers fan-out RPC queries to hundreds of backend microservices.

Suppose a front-end service queries 100 backend partitions in parallel, and user request completion requires waiting for the slowest backend service to respond:

Total_Latency = max(L1, L2, ..., L100).

Even if individual service latency has a 99th percentile of 200 ms (P(L_i <= 200 ms) = 0.99):

P(User completes <= 200 ms) = (0.99)^100 ≈ 0.366 (36.6%).

Over 63.4% of end users will experience a latency exceeding 200 ms!

Understanding extreme value distributions and tail percentiles (p95, p99, p99.9) is essential for configuring backup request hedging and RPC cancellation thresholds.

2. Monte Carlo Numerical Integration#

In statistical physics, financial risk modeling (Value-at-Risk), and machine learning, evaluating high-dimensional integrals:

I = integral_Ω g(x) dx

is analytically impossible when dimension d > 10.

Monte Carlo integration draws N uniform random samples x1, ..., xN ~ U(Ω):

I_approx = (Volume(Ω) / N) · sum_{i=1}^N g(x_i).

By the Central Limit Theorem:

The estimation error is asymptotically normally distributed with standard deviation:

Standard_Error = σ_g / sqrt(N).

Notice: The error rate O(1 / sqrt(N)) is completely independent of the dimension d! While deterministic grid integration suffers from the curse of dimensionality (requiring k^d points), Monte Carlo scaling remains invariant.

3. A/B Testing Sample Size Determination#

When running an online experimentation trial (e.g., measuring conversion rate lift Δ = p_treatment - p_control):

Using the Central Limit Theorem normal approximation for binomial proportions:

Sample size required per variant to achieve significance level α (e.g. 0.05) and statistical power 1 - β (e.g. 0.80):

n = [ 2 · (z_{α/2} + z_β)^2 · p · (1 - p) ] / Δ^2.

This formula guarantees that random sampling fluctuations will not obscure genuine feature improvements.

7. Common Architectural Pitfalls & Misconceptions#

• Pitfall 1: Assuming the Central Limit Theorem Works for Heavy-Tailed Distributions.

The Lindeberg-Lévy CLT strictly requires a FINITE variance (σ^2 < ∞). If observations are drawn from a Pareto distribution with shape parameter α <= 2 or a Cauchy distribution, the variance is infinite. The sample mean does NOT converge to a Gaussian; it remains heavy-tailed for all n!

• Pitfall 2: Confusing Probability Density f(x) with Probability P(X = x).

In continuous distributions, the density f(x) can exceed 1.0! For example, for Uniform distribution U(0, 0.2), f(x) = 1 / 0.2 = 5.0 for all x in [0, 0.2]. Density represents probability per unit length, NOT an absolute probability.

• Pitfall 3: The Gambler's Fallacy: Expecting Reversion via Anti-Correlated Compensation.

Believing that after a streak of 10 heads, tails is "due". The Law of Large Numbers operates through dilution (adding millions of new independent coin tosses dilutes the initial 10 heads into statistical insignificance), NOT through mystical self-correction.

• Pitfall 4: Blindly Using n = 30 for Heavily Skewed Data.

The textbook rule of thumb that "n >= 30 is sufficient for CLT" applies only to moderately symmetric distributions. For highly skewed financial losses or rare web latencies (where skewness γ1 > 5), sample sizes of n = 500 or 10,000 may be necessary before the sampling distribution approaches normal symmetry.

8. Summary & Checkpoint Exercises#

Solidify your mastery of continuous probability densities, moment generating functions, Chebyshev bounds, and the Central Limit Theorem with these four checkpoint problems. Full step-by-step mathematical solutions are provided.

Problem 1 (Normalizing a Continuous PDF, Computing CDF & Percentiles):

Consider the continuous random variable X with probability density function:

f(x) = c · (4 x - 2 x^2) for 0 <= x <= 2, and f(x) = 0 elsewhere.

1. Find the normalization constant c.
2. Derive the Cumulative Distribution Function F(x).
3. Compute the expected value E[X] and the probability P(0.5 <= X <= 1.5).
4. Find the median of the distribution.

Solution:

Part 1: Find Normalization Constant c.

By the total probability normalization axiom:

integral_{-∞}^∞ f(x) dx = integral_0^2 c · (4 x - 2 x^2) dx = 1.

Factor out c and integrate:

c · [ 2 x^2 - (2/3) x^3 ]_0^2 = 1.

Evaluate at x = 2:

c · [ 2(2^2) - (2/3)(2^3) ] = c · [ 2(4) - (2/3)(8) ] = c · [ 8 - 16/3 ] = c · [ 24/3 - 16/3 ] = c · [ 8/3 ].

c · (8/3) = 1 => c = 3/8.

The normalized PDF is:

f(x) = (3/8) · (4 x - 2 x^2) = (3/4) x - (3/4) x^2 for 0 <= x <= 2.

Part 2: Derive Cumulative Distribution Function F(x).

For x in [0, 2]:

F(x) = integral_0^x (3/8)(4 t - 2 t^2) dt = (3/8) · [ 2 t^2 - (2/3) t^3 ]_0^x

= (3/8)(2 x^2 - (2/3) x^3) = (3/4) x^2 - (1/4) x^3.

Summary of F(x):

• F(x) = 0 for x < 0.
• F(x) = (3/4) x^2 - (1/4) x^3 for 0 <= x <= 2.
• F(x) = 1 for x > 2.

Part 3: Compute E[X] and Interval Probability.

• Expected Value:

E[X] = integral_0^2 x · f(x) dx = integral_0^2 x · (3/8)(4 x - 2 x^2) dx

= (3/8) integral_0^2 (4 x^2 - 2 x^3) dx

= (3/8) · [ (4/3) x^3 - (1/2) x^4 ]_0^2

= (3/8) · [ (4/3)(8) - (1/2)(16) ] = (3/8) · [ 32/3 - 8 ] = (3/8) · [ 32/3 - 24/3 ] = (3/8) · (8/3) = 1.0.

(Notice: The distribution is symmetric about x = 1, so E[X] = 1.0).

• Interval Probability P(0.5 <= X <= 1.5):

P(0.5 <= X <= 1.5) = F(1.5) - F(0.5).

Evaluate F(1.5):

F(1.5) = (3/4)(1.5)^2 - (1/4)(1.5)^3 = (3/4)(2.25) - (1/4)(3.375) = 1.6875 - 0.84375 = 0.84375 = 27/32.

Evaluate F(0.5):

F(0.5) = (3/4)(0.5)^2 - (1/4)(0.5)^3 = (3/4)(0.25) - (1/4)(0.125) = 0.1875 - 0.03125 = 0.15625 = 5/32.

Difference:

P(0.5 <= X <= 1.5) = 0.84375 - 0.15625 = 0.6875 = 22/32 = 11/16 (68.75%).

Part 4: Find the Median.

The median m satisfies F(m) = 0.5.

Because f(x) is symmetric about x = 1.0:

F(1.0) = (3/4)(1)^2 - (1/4)(1)^3 = 3/4 - 1/4 = 2/4 = 0.5.

Therefore, the median is exactly m = 1.0.

Problem 2 (Deriving the MGF of an Exponential Distribution & Extracting Moments):

Let X ~ Exp(λ) with PDF f(x) = λ · e^(-λ x) for x >= 0.

1. Derive the Moment Generating Function M_X(t) from first principles, and state the domain of t for which it is defined.
2. Use M_X(t) to compute the first two raw moments E[X] and E[X^2].
3. Verify the variance formula Var(X) = 1 / λ^2.

Solution:

Part 1: Derive M_X(t).

M_X(t) = E[e^(t X)] = integral_0^∞ e^(t x) · (λ · e^(-λ x)) dx

= λ · integral_0^∞ e^((t - λ) x) dx.

For this improper integral to converge, the exponent coefficient must be strictly negative:

t - λ < 0 => t < λ.

Evaluate the integral for t < λ:

M_X(t) = λ · [ (1 / (t - λ)) · e^((t - λ) x) ]_0^∞

As x → ∞, e^((t - λ) x) → 0:

M_X(t) = λ · [ 0 - (1 / (t - λ)) · 1 ] = -λ / (t - λ) = λ / (λ - t).

Divide numerator and denominator by λ:

M_X(t) = (1 - t / λ)^(-1) for t < λ.

Part 2: Extract Moments via Derivatives.

Rewrite M_X(t) = λ · (λ - t)^(-1).

• First Derivative:

M'_X(t) = d/dt [ λ · (λ - t)^(-1) ] = λ · (-1)(λ - t)^(-2) · (-1) = λ / (λ - t)^2.

Evaluate at t = 0:

E[X] = M'_X(0) = λ / (λ - 0)^2 = λ / λ^2 = 1 / λ.

• Second Derivative:

M''_X(t) = d/dt [ λ · (λ - t)^(-2) ] = λ · (-2)(λ - t)^(-3) · (-1) = 2 λ / (λ - t)^3.

Evaluate at t = 0:

E[X^2] = M''_X(0) = 2 λ / (λ - 0)^3 = 2 λ / λ^3 = 2 / λ^2.

Part 3: Compute and Verify Variance.

Var(X) = E[X^2] - (E[X])^2

= (2 / λ^2) - (1 / λ)^2

= 2 / λ^2 - 1 / λ^2

= 1 / λ^2.

The derived variance matches the textbook formula exactly.

Problem 3 (Applying Chebyshev's and Markov's Inequalities to SLA Bounds):

A payment processing API has a mean response latency of E[X] = 120 ms with a standard deviation of σ = 30 ms. The exact distribution of latency is unknown.

1. Use Markov's inequality to obtain an upper bound on the probability that a transaction takes 300 ms or longer (P(X >= 300)).
2. Use Chebyshev's inequality to obtain a sharper upper bound on P(X >= 300).
3. If the true latency follows a Normal distribution N(120, 30^2), compute the exact Gaussian probability and compare with the bounds.

Solution:

Part 1: Markov's Inequality.

Since latency is strictly non-negative (X >= 0):

P(X >= 300) <= E[X] / 300 = 120 / 300 = 0.40 (40%).

Markov guarantees that at most 40% of requests exceed 300 ms.

Part 2: Chebyshev's Inequality.

Notice that 300 ms represents a deviation from the mean μ = 120 ms:

|X - 120| >= 300 - 120 = 180 ms.

Express deviation in terms of standard deviation σ = 30 ms:

k · σ = 180 => k · 30 = 180 => k = 6.

By Chebyshev's two-tailed inequality:

P(|X - 120| >= 180) <= 1 / k^2 = 1 / 6^2 = 1 / 36 ≈ 0.02778 (2.78%).

Since the event { X >= 300 } is a subset of { |X - 120| >= 180 }:

P(X >= 300) <= P(|X - 120| >= 180) <= 1 / 36 ≈ 2.78%.

Chebyshev slashes the upper bound from 40% down to under 2.8%!

Part 3: Normal Distribution Exact Probability.

If X ~ N(120, 30^2):

Standardize to Z:

Z = (300 - 120) / 30 = 180 / 30 = 6.0.

P(X >= 300) = P(Z >= 6.0) = 1 - Φ(6.0) ≈ 9.87 × 10^(-10).

Comparison:

• Markov Bound: <= 40% (coarse, uses only mean).
• Chebyshev Bound: <= 2.78% (much tighter, uses mean and variance).
• Normal Exact: ~1 in a billion (assumes parametric Gaussian shape).

Problem 4 (Central Limit Theorem Probability Calculation for a Sample Mean):

A cloud server's CPU utilization during background batch jobs is a random variable with mean μ = 65% and standard deviation σ = 15%. A random sample of n = 100 independent hours is recorded.

1. State the sampling distribution of the sample mean X_bar_100 under the Central Limit Theorem.
2. Calculate the probability that the sample mean CPU utilization exceeds 68% (P(X_bar_100 > 68)).
3. Find the symmetric 95% confidence bounds around the mean for X_bar_100.

Solution:

Part 1: Sampling Distribution under CLT.

Sample size n = 100 >= 30.

By the Central Limit Theorem:

The sample mean X_bar_100 is approximately normally distributed:

X_bar_100 ~ N(μ, σ^2 / n) = N(65, 15^2 / 100) = N(65, 225 / 100) = N(65, 2.25).

Standard Error (SE):

σ_{X_bar} = σ / sqrt(n) = 15 / sqrt(100) = 15 / 10 = 1.5%.

Part 2: Calculate P(X_bar_100 > 68).

Standardize the threshold 68%:

z = (68 - μ) / σ_{X_bar} = (68 - 65) / 1.5 = 3 / 1.5 = 2.0.

P(X_bar_100 > 68) ≈ P(Z > 2.0) = 1 - Φ(2.0).

From standard normal tables: Φ(2.0) ≈ 0.9772.

P(X_bar_100 > 68) = 1 - 0.9772 = 0.0228 (2.28%).

There is only a 2.28% chance that the average CPU utilization across 100 hours exceeds 68%.

Part 3: Symmetric 95% Confidence Bounds for X_bar_100.

For a standard normal distribution, 95% of probability mass lies within z = ± 1.96.

Margin of Error:

ME = 1.96 · σ_{X_bar} = 1.96 · 1.5% = 2.94%.

95% Bounds:

[ μ - ME, μ + ME ] = [ 65% - 2.94%, 65% + 2.94% ] = [ 62.06%, 67.94% ].

In 95% of random 100-hour audit samples, the sample mean CPU utilization will fall strictly between 62.06% and 67.94%.