Maximum Likelihood Estimation, Score Functions & the Cramér-Rao Lower Bound

In statistical inference and machine learning, we rarely have access to the complete population or ground-truth generative parameters. Instead, we observe a finite sample of empirical data and must deduce the hidden parameters that governed the data-generating process. Maximum Likelihood Estimation (MLE) provides the universal analytical framework for parameter estimation, underpinning neural network loss functions, logistic regression, signal detection, and distributed telemetry modeling.

1. Principles of Parametric Point Estimation & Evaluative Metrics#

Let X = { x1, x2, ..., xn } be an independent and identically distributed (i.i.d.) random sample drawn from a probability distribution characterized by a probability density (or mass) function f(x; θ), where θ in Θ ⊆ R^p represents an unknown parameter vector.

A point estimator θ_hat = g(X1, X2, ..., Xn) is a statistic—a measurable mathematical function of the observed data that generates a single numerical estimate of θ.

Evaluative Metrics for Estimator Quality#

Because an estimator θ_hat is a function of random variables, θ_hat is itself a random variable possessing its own sampling distribution. We evaluate estimators through four fundamental metrics:

1. Estimator Bias:

The bias of an estimator measures the systematic deviation of its expected value from the true parameter θ:

Bias(θ_hat) = E[θ_hat] - θ.

An estimator is defined to be unbiased if Bias(θ_hat) = 0 for all θ in Θ, meaning E[θ_hat] = θ. On average, an unbiased estimator hits the bullseye.

1. Estimator Variance:

The variance measures the dispersion or stability of the estimator across repeated experimental trials:

Var(θ_hat) = E[(θ_hat - E[θ_hat])^2].

1. Mean Squared Error (MSE) & The Bias-Variance Tradeoff:

The Mean Squared Error evaluates the total expected squared distance from the true parameter:

MSE(θ_hat) = E[(θ_hat - θ)^2].

Decomposition of MSE:

MSE(θ_hat) = E[( (θ_hat - E[θ_hat]) + (E[θ_hat] - θ) )^2]

= E[(θ_hat - E[θ_hat])^2] + 2 · (E[θ_hat] - θ) · E[θ_hat - E[θ_hat]] + (E[θ_hat] - θ)^2

Since E[θ_hat - E[θ_hat]] = 0:

MSE(θ_hat) = Var(θ_hat) + (Bias(θ_hat))^2.

The Fundamental Bias-Variance Tradeoff:

A biased estimator can sometimes achieve a substantially smaller total MSE than an unbiased estimator if its variance is drastically lower (e.g., Ridge Regularization in machine learning, or the James-Stein shrinkage estimator).

1. Consistency:

An estimator θ_hat_n is consistent if it converges in probability to the true parameter θ as sample size approaches infinity:

For every ε > 0, lim_{n → ∞} P(|θ_hat_n - θ| >= ε) = 0.

A sufficient condition for consistency is that Bias(θ_hat_n) → 0 and Var(θ_hat_n) → 0 as n → ∞.

2. The Maximum Likelihood Estimation (MLE) Framework#

To discover the optimal parameter value, we examine how plausible different candidate parameters are in light of observed data.

Probability vs. Likelihood: The Directional Inversion#

• Probability Function P(X | θ):

Evaluates the probability of observing various data outcomes X, treating parameter θ as a fixed, known constant.

• Likelihood Function L(θ | X):

Evaluates the plausibility of various candidate parameters θ, treating the observed sample data X as a fixed, known historical fact!

For an i.i.d. sample X = { x1, x2, ..., xn }, the joint Likelihood Function is the product of individual marginal densities:

L(θ; x) = prod_{i=1}^n f(x_i; θ).

The Log-Likelihood Function#

In practical computation, evaluating products of hundreds or thousands of probabilities leads to floating-point numerical underflow (e.g., (0.01)^1000 = 10^(-2000), which exceeds IEEE-754 double-precision limits).

Taking the natural logarithm transforms products into sums:

l(θ; x) = ln L(θ; x) = sum_{i=1}^n ln f(x_i; θ).

Because the natural logarithm is a strictly monotonically increasing function (ln(u) > ln(v) iff u > v):

The parameter value that maximizes the log-likelihood is guaranteed to be identical to the parameter value that maximizes the raw likelihood!

The Maximum Likelihood Estimator (MLE) is defined as:

θ_hat_MLE = argmax_{θ in Θ} l(θ; x).

+-----------------------------------------------------------------------------+
|                      LOG-LIKELIHOOD OPTIMIZATION CURVE                      |
+-----------------------------------------------------------------------------+
|    Log-Likelihood ell(theta)                                                |
|         ^                                                                   |
|         |                   theta_hat_MLE                                   |
|         |                        *   <-- Peak: Score S(theta) == 0          |
|         |                     .-----.                                       |
|         |                   .'       `.   Negative Curvature:               |
|         |                  /           \  d^2(ell)/d(theta)^2 < 0           |
|         |                 /             \ (Fisher Information)              |
|         +----------------+---------------+------------------------> theta   |
|                          0           theta_0                                |
+-----------------------------------------------------------------------------+

Analytical MLE Derivations#

To find the MLE analytically, differentiate the log-likelihood with respect to θ and set the derivative to zero.

Case 1: The Normal Distribution Mean & Variance:

Let X1, ..., Xn ~ N(μ, σ^2).

The density is f(x; μ, σ^2) = (1 / (sqrt(2π) σ)) · e^(-(x - μ)^2 / (2 σ^2)).

The log-likelihood is:

l(μ, σ^2) = - (n / 2) ln(2π) - (n / 2) ln(σ^2) - (1 / (2 σ^2)) · sum(i=1 to n, (x_i - μ)^2).

1. Estimating Mean μ:

∂l/∂μ = (1 / σ^2) · sum(i=1 to n, (x_i - μ)) = 0

sum(i=1 to n, x_i) - n · μ = 0 => μ_hat_MLE = (1 / n) · sum(i=1 to n, x_i) = X_bar.

The sample average is the MLE of the population mean!

1. Estimating Variance σ^2:

Let v = σ^2.

∂l/∂v = - n / (2 v) + (1 / (2 v^2)) · sum(i=1 to n, (x_i - μ)^2) = 0.

Multiply by 2 v^2:

- n · v + sum(i=1 to n, (x_i - μ)^2) = 0

σ^2_hat_MLE = (1 / n) · sum(i=1 to n, (x_i - X_bar)^2).

The Inherent Bias of the Variance MLE & Bessel's Correction#

Evaluate the expected value of the variance MLE:

E[σ^2_hat_MLE] = E[ (1 / n) sum(i=1 to n, (X_i - X_bar)^2) ].

Recall identity: sum(X_i - X_bar)^2 = sum(X_i - μ)^2 - n(X_bar - μ)^2.

Take expectations:

E[ sum(X_i - μ)^2 ] = n · Var(X_i) = n · σ^2.

E[ n(X_bar - μ)^2 ] = n · Var(X_bar) = n · (σ^2 / n) = σ^2.

Therefore:

E[ sum(X_i - X_bar)^2 ] = n · σ^2 - σ^2 = (n - 1) · σ^2.

Dividing by n:

E[σ^2_hat_MLE] = ((n - 1) / n) · σ^2.

The Maximum Likelihood Estimator for variance is strictly biased! It underestimates population variance by a factor of (n - 1) / n because the sample mean X_bar is closer to the sample data points than the true population mean μ.

Bessel's Correction:

Multiplying by n / (n - 1) eliminates the bias, producing the familiar sample variance s^2:

s^2 = (1 / (n - 1)) · sum(i=1 to n, (x_i - X_bar)^2).

E[s^2] = σ^2 (Strictly Unbiased).

3. The Score Function & The Fisher Information Matrix#

To understand the mathematical mechanics of likelihood surfaces and quantify parameter uncertainty, we examine derivatives of the log-likelihood.

The Score Function#

The Score Function S(θ) is the gradient (first derivative) of the log-likelihood function with respect to parameter θ:

S(θ) = d/dθ [ l(θ; X) ] = sum(i=1 to n, d/dθ [ ln f(X_i; θ) ]).

The score measures the sensitivity of the log-likelihood curve to perturbations in θ.

At the MLE, the score vanishes: S(θ_hat_MLE) = 0.

The First Moment of the Score#

Theorem: Under standard regularity conditions (allowing the interchange of differentiation and integration), the expected value of the score evaluated at the true parameter is always identically zero:

E[S(θ)] = 0.

Rigorous Analytical Proof:

Consider a single observation X with density f(x; θ):

E[ d/dθ ln f(X; θ) ] = integral_{-∞}^∞ [ d/dθ ln f(x; θ) ] · f(x; θ) dx

Using d/dθ [ ln f ] = (1 / f) · (∂f / ∂θ):

= integral_{-∞}^∞ [ (1 / f(x; θ)) · (∂f(x; θ) / ∂θ) ] · f(x; θ) dx

= integral_{-∞}^∞ [ ∂f(x; θ) / ∂θ ] dx.

Interchange differentiation and integration:

= d/dθ [ integral_{-∞}^∞ f(x; θ) dx ].

By the total probability normalization axiom, integral f(x; θ) dx = 1 for all θ:

= d/dθ [ 1 ] = 0.

Summing across n independent observations: E[S(θ)] = sum 0 = 0.

The Fisher Information Matrix (FIM)#

The Fisher Information I(θ) measures the expected curvature of the log-likelihood surface around the true parameter, quantifying the amount of information that observable random variables carry about θ.

Definition: Fisher Information is defined as the variance of the score function:

I(θ) = Var(S(θ)) = E[(S(θ) - E[S(θ)])^2] = E[(S(θ))^2].

Theorem (Second Derivative Form of Fisher Information):

Under regularity conditions:

I(θ) = - E[ d^2 l(θ; X) / dθ^2 ].

Fisher Information equals the negative expected value of the Hessian (second derivative) of the log-likelihood!

Geometric Meaning of Curvature:

• High Curvature (-d^2 l / dθ^2 >> 0): The log-likelihood forms a steep, sharp peak around θ_hat. Parameter θ is tightly constrained by the data. Fisher Information is high, and estimation error is small.
• Low Curvature (-d^2 l / dθ^2 ≈ 0): The log-likelihood is flat and broad. Many different parameter values explain the data nearly equally well. Fisher Information is low, and estimation uncertainty is large.

Additivity of Fisher Information:

For n independent observations, information accumulates linearly:

I_n(θ) = n · I_1(θ).

4. The Cramér-Rao Lower Bound (CRLB) & Estimator Efficiency#

How accurate can an unbiased statistical estimator possibly be? Is there a theoretical floor on variance?

The Cramér-Rao Inequality proves that no unbiased estimator can have a variance lower than the reciprocal of Fisher Information!

The Cramér-Rao Inequality#

Theorem (Harald Cramér & C. R. Rao):

Let X1, ..., Xn be an i.i.d. sample, and let θ_hat be ANY unbiased estimator of parameter θ (meaning E[θ_hat] = θ).

Under standard regularity conditions:

Var(θ_hat) >= 1 / I_n(θ) = 1 / [ n · I_1(θ) ].

Complete Analytical Proof via the Cauchy-Schwarz Inequality#

1. Consider the covariance between unbiased estimator θ_hat and the score function S(θ):

Cov(θ_hat, S(θ)) = E[θ_hat · S(θ)] - E[θ_hat] · E[S(θ)].

Since E[S(θ)] = 0:

Cov(θ_hat, S(θ)) = E[θ_hat · S(θ)].

1. Evaluate E[θ_hat · S(θ)]:

E[θ_hat · S(θ)] = integral ... integral θ_hat(x) · [ sum(i=1 to n, (1 / f(x_i; θ)) · (∂f(x_i; θ) / ∂θ) ) ] · [ prod f(x_j; θ) ] dx1 ... dxn

Using identity d/dθ [ prod f(x_i; θ) ] = [ sum (1/f)(∂f/∂θ) ] · prod f(x_i; θ):

= integral ... integral θ_hat(x) · [ d/dθ ( prod_{i=1}^n f(x_i; θ) ) ] dx1 ... dxn

Interchange derivative and integral:

= d/dθ [ integral ... integral θ_hat(x) · prod_{i=1}^n f(x_i; θ) dx1 ... dxn ]

= d/dθ [ E[θ_hat] ].

Because θ_hat is an unbiased estimator, E[θ_hat] = θ:

= d/dθ [ θ ] = 1.

Therefore: Cov(θ_hat, S(θ)) = 1.

1. Apply Cauchy-Schwarz Inequality to Random Variables:

For any random variables Y and Z: [ Cov(Y, Z) ]^2 <= Var(Y) · Var(Z).

Substitute Y = θ_hat and Z = S(θ):

[ Cov(θ_hat, S(θ)) ]^2 <= Var(θ_hat) · Var(S(θ)).

1^2 <= Var(θ_hat) · I_n(θ)

Var(θ_hat) >= 1 / I_n(θ).

This completes the proof.

Efficient Estimators & UMVUE#

Definition: An unbiased estimator θ_hat that attains the Cramér-Rao Lower Bound with exact equality:

Var(θ_hat) = 1 / I_n(θ)

is termed Efficient, or a Uniformly Minimum-Variance Unbiased Estimator (UMVUE).

An efficient estimator achieves the absolute mathematical limit of precision attainable for a given sample size.

5. Asymptotic Properties of Maximum Likelihood Estimators#

Maximum Likelihood Estimators possess four extraordinary asymptotic guarantees that make them the gold standard in statistical computation.

The Four Asymptotic Guarantees of MLE#

As sample size n approaches infinity (n → ∞), under mild regularity conditions:

1. Asymptotic Consistency:

The MLE converges in probability to the true population parameter θ_0:

plim_{n → ∞} θ_hat_MLE = θ_0.

1. Asymptotic Normality:

The sampling distribution of the MLE converges to a Gaussian bell curve centered at the true parameter:

sqrt(n) · (θ_hat_MLE - θ_0) →^d N(0, 1 / I_1(θ_0)).

In practice, for large n:

θ_hat_MLE ~ N(θ_0, 1 / I_n(θ_0)).

1. Asymptotic Efficiency:

The variance of the MLE asymptotically reaches the Cramér-Rao Lower Bound!

lim_{n → ∞} [ n · Var(θ_hat_MLE) ] = 1 / I_1(θ_0).

No competing estimator can exhibit lower asymptotic variance.

1. Functional Invariance:

If θ_hat is the MLE of parameter θ, and g(θ) is any continuous transformation, then the MLE of g(θ) is simply:

g_hat_MLE = g(θ_hat_MLE).

(For example, if p_hat is the MLE of Bernoulli success probability, then the odds ratio p_hat / (1 - p_hat) is the exact MLE of the population odds ratio!).

Constructing Asymptotic Wald Confidence Intervals#

Leveraging asymptotic normality, we construct (1 - α) confidence intervals for unknown parameter θ using Fisher Information:

Standard Error: SE(θ_hat) = 1 / sqrt(I_n(θ_hat_MLE)).

Wald Confidence Interval:

[ θ_hat_MLE - z_{α/2} · SE(θ_hat), θ_hat_MLE + z_{α/2} · SE(θ_hat) ].

For a 95% confidence interval, z_{α/2} = 1.96.

6. Machine Learning, Deep Learning & Telemetry Applications#

Maximum Likelihood is the generative foundation of modern Artificial Intelligence.

1. Classification & Neural Network Cross-Entropy Loss#

In machine learning, why do neural networks minimize Cross-Entropy rather than Mean Squared Error for classification?

Consider binary classification with labels y in {0, 1} and model probability p(x; W) = σ(W · x):

Likelihood of observing sample (x_i, y_i):

L(W) = prod_{i=1}^n [ p(x_i; W) ]^(y_i) · [ 1 - p(x_i; W) ]^(1 - y_i).

Taking the negative log-likelihood:

Negative Log-Likelihood (NLL) = - sum(i=1 to n, [ y_i ln p(x_i; W) + (1 - y_i) ln(1 - p(x_i; W)) ]).

This is the EXACT formula for Binary Cross-Entropy Loss!

Minimizing cross-entropy loss in deep neural networks is mathematically identical to finding the Maximum Likelihood parameter weights W.

2. Ordinary Least Squares (OLS) as Gaussian MLE#

Consider linear regression y = X · β + ε, where noise is normally distributed ε ~ N(0, σ^2 · I).

The log-likelihood of observations y given features X is:

l(β, σ^2) = - (n / 2) ln(2π σ^2) - (1 / (2 σ^2)) · ||y - X · β||_2^2.

Maximizing log-likelihood with respect to β is equivalent to:

argmin_β ||y - X · β||_2^2.

Ordinary Least Squares regression is the Maximum Likelihood Estimator under Gaussian noise.

3. Distributed Cloud Telemetry: Failure Rate Estimation#

In distributed database clusters (e.g., CockroachDB, TiDB), server node lifetimes follow an Exponential distribution Exp(λ).

Given n observed crash recovery intervals t1, t2, ..., tn:

l(λ) = n ln(λ) - λ · sum_{i=1}^n t_i.

Score equation: S(λ) = n / λ - sum t_i = 0 => λ_hat_MLE = n / sum(t_i).

Fisher Information: I_n(λ) = n / λ^2.

Asymptotic Variance: Var(λ_hat) ≈ λ^2 / n.

Cloud orchestrators deploy this MLE to dynamically estimate Mean Time to Failure (MTTF = 1 / λ_hat) and trigger replica provisioning before quorum degradation occurs.

7. Common Architectural Pitfalls & Misconceptions#

• Pitfall 1: Assuming All Maximum Likelihood Estimators Are Unbiased.

While MLEs are asymptotically unbiased (lim Bias = 0 as n → ∞), they can be significantly biased for finite samples. The sample variance MLE divides by n instead of n - 1. For a Uniform distribution U(0, θ), the MLE is max(x_i), which has expectation [n / (n + 1)] θ, strictly underestimating the true upper bound.

• Pitfall 2: Confusing the Score with the Likelihood.

The likelihood is a scalar measuring probability density; the score is the gradient vector of the log-likelihood. A score of zero indicates that the optimizer has reached a stationary point (local minimum, maximum, or saddle point).

• Pitfall 3: Applying CRLB to Biased Estimators.

The standard Cramér-Rao bound Var >= 1 / I_n applies strictly to unbiased estimators. For an estimator with bias b(θ), the generalized bound is:

Var(θ_hat) >= [ 1 + b'(θ) ]^2 / I_n(θ).

• Pitfall 4: Misinterpreting Asymptotic Normality for Small Sample Sizes.

Constructing symmetric Wald intervals [ θ_hat ± 1.96 · SE ] requires large n. For bounded parameters (like probability p in [0, 1] near 0 or 1), Wald intervals can produce absurd bounds outside [0, 1] (e.g., [-0.02, 0.08]). For small samples, use Wilson Score intervals or profile likelihood intervals instead.

8. Summary & Checkpoint Exercises#

Solidify your mastery of Maximum Likelihood Estimation, score functions, Fisher Information, and the Cramér-Rao Lower Bound with these four checkpoint problems. Full step-by-step mathematical solutions are provided.

Problem 1 (Deriving MLE, Score & Fisher Information for a Poisson Distribution):

Let X1, X2, ..., Xn be an i.i.d. sample from a Poisson distribution with parameter λ > 0:

P(X = x) = (e^(-λ) · λ^x) / x! for x in {0, 1, 2, ...}.

1. Formulate the log-likelihood function l(λ; x).
2. Derive the Score function S(λ) and find the Maximum Likelihood Estimator λ_hat_MLE.
3. Compute the Fisher Information I_n(λ).
4. Prove that λ_hat_MLE achieves the Cramér-Rao Lower Bound, and is therefore an efficient estimator (UMVUE).

Solution:

Part 1: Log-Likelihood Function.

Likelihood:

L(λ; x) = prod_{i=1}^n [ (e^(-λ) · λ^(x_i)) / x_i! ] = e^(-n λ) · λ^(sum x_i) / prod_{i=1}^n x_i!.

Take natural logarithm:

l(λ; x) = -n · λ + (sum_{i=1}^n x_i) · ln(λ) - sum_{i=1}^n ln(x_i!).

Part 2: Score Function and MLE.

Differentiate l(λ; x) with respect to λ:

S(λ) = d/dλ [ l(λ; x) ] = -n + (1 / λ) · sum_{i=1}^n x_i.

Set score to zero:

-n + (1 / λ) · sum_{i=1}^n x_i = 0

n = (1 / λ) · sum_{i=1}^n x_i

λ_hat_MLE = (1 / n) · sum_{i=1}^n x_i = X_bar.

The sample mean is the MLE of the Poisson rate parameter λ.

Part 3: Compute Fisher Information.

Differentiate the score function a second time:

d^2 l / dλ^2 = d/dλ [ -n + (1 / λ) sum x_i ] = - (1 / λ^2) · sum_{i=1}^n x_i.

Take the negative expectation:

I_n(λ) = - E[ d^2 l / dλ^2 ] = E[ (1 / λ^2) · sum_{i=1}^n X_i ]

= (1 / λ^2) · sum_{i=1}^n E[X_i].

Since X_i ~ Poisson(λ), E[X_i] = λ:

I_n(λ) = (1 / λ^2) · (n · λ) = n / λ.

Fisher Information for n observations: I_n(λ) = n / λ.

Part 4: Cramér-Rao Lower Bound & Efficiency Verification.

The CRLB for any unbiased estimator of λ is:

CRLB = 1 / I_n(λ) = 1 / (n / λ) = λ / n.

Now evaluate the actual variance of λ_hat_MLE = X_bar:

Var(X_bar) = Var((1/n) sum X_i) = (1 / n^2) · sum Var(X_i).

Since X_i ~ Poisson(λ), Var(X_i) = λ:

Var(X_bar) = (1 / n^2) · (n · λ) = λ / n.

Comparison:

Var(λ_hat_MLE) = λ / n = CRLB.

Because the variance of λ_hat_MLE equals the Cramér-Rao Lower Bound with exact equality:

λ_hat_MLE is strictly efficient and is the Uniformly Minimum-Variance Unbiased Estimator (UMVUE) for the Poisson distribution.

Problem 2 (Proof of Bias in Normal Variance MLE & Bessel's Correction):

Suppose X1, ..., Xn are i.i.d. draws from N(μ, σ^2) with known mean μ = 0, but unknown variance σ^2.

1. Derive the MLE for σ^2 when μ = 0 is known: v_hat = σ^2_hat.
2. Determine whether this estimator is biased or unbiased.
3. Contrast this with the scenario where mean μ is unknown and estimated by X_bar.

Solution:

Part 1: Derive MLE when μ = 0 is Known.

Let v = σ^2. The likelihood function is:

L(v) = prod_{i=1}^n [ (1 / sqrt(2π v)) · e^(-x_i^2 / (2 v)) ] = (2π v)^(-n/2) · e^(-(1 / (2v)) sum x_i^2).

Log-likelihood:

l(v) = - (n/2) ln(2π) - (n/2) ln(v) - (1 / (2v)) · sum_{i=1}^n x_i^2.

Differentiate with respect to v and set to zero:

dl/dv = - n / (2 v) + (1 / (2 v^2)) · sum_{i=1}^n x_i^2 = 0

Multiply by 2 v^2:

• n · v + sum_{i=1}^n x_i^2 = 0

v_hat_known = (1 / n) · sum_{i=1}^n x_i^2.

Part 2: Bias Evaluation.

E[v_hat_known] = E[ (1 / n) sum_{i=1}^n X_i^2 ] = (1 / n) sum_{i=1}^n E[X_i^2].

Since μ = E[X_i] = 0, E[X_i^2] = Var(X_i) + (E[X_i])^2 = σ^2 + 0 = σ^2.

E[v_hat_known] = (1 / n) · (n · σ^2) = σ^2.

Bias(v_hat_known) = E[v_hat_known] - σ^2 = σ^2 - σ^2 = 0.

Conclusion: When the true mean μ is known, the variance MLE is strictly UNBIASED!

Part 3: Contrast with Unknown Mean.

When μ is unknown, we must replace μ with X_bar = (1/n) sum X_i.

The sample deviations (X_i - X_bar) satisfy sum(X_i - X_bar) = 0, which imposes a linear constraint, consuming 1 degree of freedom.

As proved in Section 2: E[ sum(X_i - X_bar)^2 ] = (n - 1) σ^2.

Thus, when μ is unknown, dividing by n introduces a downward bias of -σ^2 / n. Bessel's correction divides by (n - 1) to account for the single degree of freedom lost to estimating the mean.

Problem 3 (Calculating the Cramér-Rao Lower Bound for an Exponential Parameter):

Let X1, ..., Xn ~ Exp(λ) with PDF f(x; λ) = λ · e^(-λ x) for x >= 0.

1. Compute the Fisher Information for a single observation I_1(λ).
2. State the Cramér-Rao Lower Bound for any unbiased estimator of λ.
3. Given that the MLE is λ_hat = n / sum(X_i): Using the fact that sum(X_i) follows a Gamma(n, λ) distribution with E[1 / sum(X_i)] = λ / (n - 1), compute the exact expected value and bias of λ_hat.
4. Construct an adjusted estimator λ_tilde that is strictly unbiased, and compare its variance to the CRLB.

Solution:

Part 1: Compute Fisher Information I_1(λ).

For a single observation X:

ln f(X; λ) = ln(λ) - λ · X.

First derivative (score):

d/dλ [ ln f(X; λ) ] = 1 / λ - X.

Second derivative:

d^2/dλ^2 [ ln f(X; λ) ] = - 1 / λ^2.

Fisher Information:

I_1(λ) = - E[ - 1 / λ^2 ] = 1 / λ^2.

For n observations: I_n(λ) = n / λ^2.

Part 2: Cramér-Rao Lower Bound.

CRLB = 1 / I_n(λ) = 1 / (n / λ^2) = λ^2 / n.

Part 3: Expectation and Bias of MLE λ_hat = n / sum(X_i).

Let S = sum_{i=1}^n X_i.

E[λ_hat] = E[ n / S ] = n · E[1 / S].

Given that E[1 / S] = λ / (n - 1):

E[λ_hat] = n · [ λ / (n - 1) ] = [ n / (n - 1) ] · λ.

Bias:

Bias(λ_hat) = E[λ_hat] - λ = [ n / (n - 1) - 1 ] · λ = [ 1 / (n - 1) ] · λ > 0.

The MLE is positively biased for finite n, though asymptotically unbiased as n → ∞.

Part 4: Construct Unbiased Estimator λ_tilde.

Multiply λ_hat by (n - 1) / n:

λ_tilde = [ (n - 1) / n ] · λ_hat = (n - 1) / sum_{i=1}^n X_i.

Check expectation:

E[λ_tilde] = [ (n - 1) / n ] · E[λ_hat] = [ (n - 1) / n ] · [ (n / (n - 1)) · λ ] = λ. (Unbiased!).

Compute Variance of λ_tilde:

For Gamma(n, λ), E[1 / S^2] = λ^2 / [ (n - 1)(n - 2) ] for n > 2.

E[λ_tilde^2] = (n - 1)^2 · E[1 / S^2] = (n - 1)^2 · [ λ^2 / ((n - 1)(n - 2)) ] = [ (n - 1) / (n - 2) ] · λ^2.

Var(λ_tilde) = E[λ_tilde^2] - (E[λ_tilde])^2 = [ (n - 1) / (n - 2) ] · λ^2 - λ^2

= [ (n - 1) / (n - 2) - 1 ] · λ^2 = λ^2 / (n - 2).

Compare with CRLB = λ^2 / n:

Notice that Var(λ_tilde) = λ^2 / (n - 2) > λ^2 / n = CRLB.

The variance is slightly strictly above the CRLB for finite n, but approaches the CRLB as n → ∞ (asymptotically efficient).

Problem 4 (Constructing Asymptotic Wald Confidence Intervals from Fisher Information):

A network reliability engineer observes n = 200 server crash recovery events. The observed sample mean recovery time is X_bar = 4.2 hours. Assuming recovery times follow an Exponential distribution Exp(λ):

1. Compute the point estimate λ_hat_MLE.
2. Compute the estimated Fisher Information I_n(λ_hat) and standard error SE(λ_hat).
3. Construct a 95% asymptotic Wald confidence interval for λ.
4. Using the invariance property of MLE, construct a 95% confidence interval for the Mean Time to Recover (MTTR = 1 / λ).

Solution:

Part 1: Point Estimate λ_hat_MLE.

From Problem 3, the MLE for an Exponential rate parameter is:

λ_hat = 1 / X_bar = 1 / 4.2 ≈ 0.2381 recoveries per hour.

Part 2: Fisher Information and Standard Error.

Fisher Information for n = 200:

I_n(λ) = n / λ^2.

Evaluate at λ_hat:

I_n(λ_hat) = 200 / (0.2381)^2 = 200 / 0.05669 ≈ 3527.87.

Standard Error:

SE(λ_hat) = 1 / sqrt(I_n(λ_hat)) = 1 / sqrt(3527.87) ≈ 1 / 59.396 ≈ 0.01684.

(Direct formula: SE = λ_hat / sqrt(n) = 0.2381 / sqrt(200) = 0.2381 / 14.142 ≈ 0.01684).

Part 3: 95% Asymptotic Wald Confidence Interval for λ.

For 95% confidence, z_{0.025} = 1.96.

Margin of Error:

ME = 1.96 · SE = 1.96 · 0.01684 ≈ 0.03301.

Confidence Interval:

[ λ_hat - ME, λ_hat + ME ] = [ 0.2381 - 0.0330, 0.2381 + 0.0330 ]

= [ 0.2051, 0.2711 ].

We are 95% confident that the true population failure rate λ lies between 0.2051 and 0.2711 recoveries per hour.

Part 4: 95% Confidence Interval for MTTR = 1 / λ.

By the functional invariance of continuous monotonic transformations:

Invert the interval boundaries:

• Lower Bound for MTTR: 1 / λ_upper = 1 / 0.2711 ≈ 3.689 hours.
• Upper Bound for MTTR: 1 / λ_lower = 1 / 0.2051 ≈ 4.876 hours.

95% Confidence Interval for MTTR:

[ 3.69 hours, 4.88 hours ].

Point estimate MTTR = 4.20 hours.