Kolmogorov Axioms, Conditional Probability & Bayes' Theorem Posterior Updating

In computing, scientific exploration, machine learning, and systems engineering, determinism is frequently an idealized abstraction. Network packet arrivals, hardware component failures, sensor noise, user search intents, and distributed consensus latencies all exhibit inherent stochasticity. Probability theory provides the rigorous mathematical calculus for quantifying, reasoning about, and acting under conditions of uncertainty.

1. Foundational Architecture of Probability: Sample Spaces & Kolmogorov Axioms#

Historically, probability was conceptualized through informal heuristics such as physical symmetry (e.g., rolling fair dice) or long-run frequencies. However, these formulations collapsed when applied to continuous variables or infinite sets. In 1933, Soviet mathematician Andrey Kolmogorov established the modern measure-theoretic foundation of probability.

The Probability Space Triple (Ω, F, P)#

A formal probabilistic model is defined by a mathematical triple (Ω, F, P):

1. The Sample Space Ω:

The set of all conceivable mutually exclusive outcomes of a random phenomenon.

- Discrete Sample Space: Finite or countably infinite outcomes (e.g., bit flips Ω = {0, 1}, number of HTTP requests n in {0, 1, 2, ...}).

- Continuous Sample Space: Uncountably infinite outcomes (e.g., execution latency t in [0, ∞), voltage levels V in R).

1. The Event Space (Sigma-Algebra) F:

A collection of subsets of Ω (called events) satisfying three fundamental closure properties:

- Ω belongs to F (Ω in F).

- Closure under Complementation: If event E in F, then its complement E^c in F.

- Closure under Countable Unions: If E1, E2, E3, ... belong to F, then their union (bigcup_{i=1}^∞ E_i) belongs to F.

Any subset E in F is an event to which a probability can be rigorously assigned.

1. The Probability Measure P:

A real-valued function P: F → [0, 1] that maps every event E in F to a real number, satisfying Kolmogorov's Three Axioms.

Andrey Kolmogorov's Three Axioms of Probability#

For any events E in F:

1. Axiom 1: Non-Negativity:

P(E) >= 0 for all E in F.

Probabilities can never be negative.

1. Axiom 2: Normalization (Unitarity):

P(Ω) = 1.

The probability that some outcome in the sample space occurs is absolute certainty.

1. Axiom 3: Countable Additivity (Disjoint Events):

If E1, E2, E3, ... is a sequence of mutually exclusive (pairwise disjoint) events (meaning E_i ∩ E_j = ∅ for all i ≠ j), then:

P(bigcup_{i=1}^∞ E_i) = sum_{i=1}^∞ P(E_i).

+-----------------------------------------------------------------------------+
|                 KOLMOGOROV PROBABILITY SPACE ARCHITECTURE                   |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Universal Sample Space Omega (Total Measure P(Omega) = 1.0)               |
|  +-----------------------------------------------------------------------+  |
|  |                                                                       |  |
|  |     Event A                Disjoint Event B                           |  |
|  |   .---------.            .-----------------.                          |  |
|  |  (  P(A)>=0  )          (    P(B) >= 0      )                         |  |
|  |   `---------'            `-----------------'                          |  |
|  |                                                                       |  |
|  |    Countable Additivity: P(A union B) = P(A) + P(B) when A cap B = {} |  |
|  |    Empty Event: P({}) = 0.0                                           |  |
|  |    Complement: P(A^c) = 1.0 - P(A)                                    |  |
|  +-----------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------+

Fundamental Derivations from Kolmogorov's Axioms#

From these three minimal axioms, the entire algebra of probability is derived:

• The Probability of the Impossible Event:

Ω and ∅ are disjoint, and Ω ∪ ∅ = Ω. By Axiom 3: P(Ω ∪ ∅) = P(Ω) + P(∅).

By Axiom 2: 1 = 1 + P(∅) => P(∅) = 0.

• The Complement Rule:

For any event E, E and E^c are disjoint, and E ∪ E^c = Ω.

P(E ∪ E^c) = P(E) + P(E^c) = P(Ω) = 1 => P(E^c) = 1 - P(E).

• Monotonicity:

If A ⊆ B, then B can be decomposed into disjoint sets B = A ∪ (B \ A).

P(B) = P(A) + P(B \ A). By Axiom 1, P(B \ A) >= 0 => P(A) <= P(B).

Since E ⊆ Ω, P(E) <= P(Ω) = 1. Thus, 0 <= P(E) <= 1 for all events.

• The Principle of Inclusion-Exclusion (General Addition Rule):

For any two arbitrary events A and B (not necessarily disjoint):

P(A ∪ B) = P(A) + P(B) - P(A ∩ B).

The overlapping intersection P(A ∩ B) is counted twice in P(A) + P(B) and must be subtracted once.

• Boole's Inequality (Union Bound):

For any sequence of events E1, E2, ..., En:

P(bigcup_{i=1}^n E_i) <= sum_{i=1}^n P(E_i).

In distributed systems and algorithm design, Boole's inequality allows upper-bounding the failure probability of a complex system without requiring independence.

2. Conditional Probability & The Algebra of Evidence#

In real-world decision-making, we constantly acquire new information. Conditioning formalizes how probabilities update when we learn that a specific event B has occurred.

Definition of Conditional Probability#

Let A and B be events in F with P(B) > 0. The conditional probability of A given that B has occurred, denoted P(A | B), is defined as:

P(A | B) = P(A ∩ B) / P(B).

Geometric Intuition:

When event B occurs, B becomes the new effective sample space. Outcomes outside B are eliminated. The probability of A is rescaled to measure what fraction of event B overlaps with A.

The Multiplication Rule & The Probability Chain Rule#

Rearranging the conditional probability definition yields the general Multiplication Rule:

P(A ∩ B) = P(B) · P(A | B) = P(A) · P(B | A).

Generalizing by induction to n arbitrary events yields the Probability Chain Rule:

P(A1 ∩ A2 ∩ ... ∩ An) = P(A1) · P(A2 | A1) · P(A3 | A1 ∩ A2) · ... · P(An | A1 ∩ ... ∩ A_{n-1}).

In natural language processing (autoregressive Large Language Models like GPT-4), the joint probability of a sequence of tokens w1, w2, ..., wT is factorized exactly using this chain rule:

P(w1, w2, ..., wT) = prod_{t=1}^T P(w_t | w1, w2, ..., w_{t-1}).

Statistical Independence vs. Mutual Exclusivity#

Definition: Two events A and B are statistically independent if and only if:

P(A ∩ B) = P(A) · P(B).

Equivalently, if P(B) > 0:

P(A | B) = P(A).

Learning that B occurred provides zero information regarding whether A occurs.

Crucial Pitfall: Independence vs. Mutual Exclusivity:

• Mutual Exclusivity means A ∩ B = ∅ (the events cannot happen together).
• Independence means P(A ∩ B) = P(A) · P(B).

If two non-empty events are mutually exclusive, then P(A ∩ B) = 0. If P(A) > 0 and P(B) > 0, then P(A) · P(B) > 0 ≠ 0.

Therefore: Two non-empty mutually exclusive events are NEVER independent! Knowing that A occurred proves that B CANNOT occur.

Conditional Independence#

Events A and B are defined to be conditionally independent given C (denoted A ⊥ B | C) if:

P(A ∩ B | C) = P(A | C) · P(B | C).

Equivalently: P(A | B ∩ C) = P(A | C).

Once condition C is known, learning B provides no additional information about A.

Conditional independence is the architectural core of Naive Bayes Classifiers and Markov Random Fields.

3. The Law of Total Probability & Bayes' Theorem Architecture#

When an event of interest cannot be observed directly, we evaluate it across a comprehensive set of alternative scenarios.

Partitions of a Sample Space#

A collection of events { B1, B2, ..., Bk } forms a partition of the sample space Ω if:

1. They are mutually exclusive: B_i ∩ B_j = ∅ for all i ≠ j.
2. They are collectively exhaustive: B1 ∪ B2 ∪ ... ∪ Bk = Ω.
3. Each scenario has non-zero probability: P(B_i) > 0 for all i.

The Law of Total Probability#

Theorem: For any event A and any partition { B1, B2, ..., Bk } of Ω:

P(A) = sum_{i=1}^k P(A ∩ Bi) = sum_{i=1}^k P(A | Bi) · P(Bi).

The total probability of event A is the weighted average of its conditional probabilities across all possible background states of nature Bi, weighted by the probability of each state.

+-----------------------------------------------------------------------------+
|                      THE LAW OF TOTAL PROBABILITY                           |
+-----------------------------------------------------------------------------+
|  Partition B1          Partition B2          Partition B3                   |
|  +-------------------+---------------------+---------------------+          |
|  |                   |                     |                     |          |
|  |       .-----------+---------------------+-----------.         |          |
|  |      / A cap B1   |       A cap B2      |  A cap B3  \        |          |
|  |     (             |                     |             )       |          |
|  |      \            |                     |            /        |          |
|  |       `-----------+---------------------+-----------'         |          |
|  |                   |                     |   Event A           |          |
|  +-------------------+---------------------+---------------------+          |
|                                                                             |
|  Formula: P(A) = P(A|B1)*P(B1) + P(A|B2)*P(B2) + P(A|B3)*P(B3)              |
+-----------------------------------------------------------------------------+

Bayes' Theorem Architecture#

Formulated by Reverend Thomas Bayes and published posthumously in 1763, Bayes' Theorem provides the mathematical framework for updating beliefs in light of observed empirical evidence.

Theorem (Bayes' Theorem):

Let { B1, B2, ..., Bk } be a partition of Ω, and let A be an observed event with P(A) > 0.

For any specific hypothesis B_j:

P(B_j | A) = [ P(A | B_j) · P(B_j) ] / P(A)

= [ P(A | B_j) · P(B_j) ] / [ sum_{i=1}^k P(A | B_i) · P(B_i) ].

Deconstructing the Four Bayesian Components#

1. Prior Probability P(B_j):

The degree of belief or historical baseline frequency of hypothesis B_j before observing the current evidence A.

1. Likelihood P(A | B_j):

The probability of observing evidence A assuming hypothesis B_j is true.

1. Marginal Evidence P(A):

The total probability of observing evidence A across all possible hypotheses, computed via the Law of Total Probability. It acts as a normalizing constant.

1. Posterior Probability P(B_j | A):

The updated, rational degree of belief that hypothesis B_j is true after observing evidence A.

In proportional form:

Posterior ∝ Likelihood × Prior.

4. Discrete Random Variables, PMFs & Expected Values#

While events are sets of abstract outcomes, computational algorithms process numerical data. A Random Variable maps abstract events into concrete real numbers.

Definition of a Random Variable#

A random variable X is a measurable function X: Ω → R that assigns a real number X(ω) to each outcome ω in sample space Ω.

Probability Mass Function (PMF)#

For a discrete random variable X (taking values in a countable set { x1, x2, ... }), the Probability Mass Function p_X(x) is defined as:

p_X(x) = P(X = x) = P({ ω in Ω : X(ω) = x }).

A valid PMF must satisfy two criteria:

1. p_X(x) >= 0 for all x.
2. sum_{all x} p_X(x) = 1.

Mathematical Expectation (Expected Value)#

The expected value E[X] (also denoted μ) is the probability-weighted average of all possible values of X:

E[X] = sum_{all x} x · p_X(x).

The Law of the Unconscious Statistician (LOTUS):

For any real-valued function g(X):

E[g(X)] = sum_{all x} g(x) · p_X(x).

Linearity of Expectation: The Master Invariant#

Theorem: For any random variables X and Y (whether independent or highly dependent!) and constants a, b in R:

E[a · X + b · Y] = a · E[X] + b · E[Y].

Linearity of expectation holds unconditionally. It is one of the most powerful tools in probabilistic analysis, permitting the calculation of complex system expectations by decomposing them into indicator random variables.

Variance and Standard Deviation#

The variance Var(X) (denoted σ^2) measures the expected squared deviation of X from its mean:

Var(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2.

Properties of Variance:

• Var(X) >= 0, with Var(X) = 0 if and only if X is a deterministic constant.
• Scaling: Var(a · X + b) = a^2 · Var(X). Adding a constant shift b does not alter dispersion; scaling by a scales variance by a^2.
• The standard deviation is σ = sqrt(Var(X)), returning dispersion to the original measurement units.

Covariance and Correlation of Joint Random Variables#

When tracking two random variables X and Y:

• Covariance: Cov(X, Y) = E[(X - E[X])(Y - E[Y])] = E[X · Y] - E[X] · E[Y].

- If X and Y are independent, then E[X · Y] = E[X] · E[Y], which guarantees Cov(X, Y) = 0.

- Caution: The converse is not true; Cov(X, Y) = 0 does NOT guarantee independence (non-linear relationships can have zero covariance).

• Variance of a Sum:

Var(X + Y) = Var(X) + Var(Y) + 2 · Cov(X, Y).

If X and Y are independent (or uncorrelated): Var(X + Y) = Var(X) + Var(Y).

• Pearson Correlation Coefficient:

ρ(X, Y) = Cov(X, Y) / [ sqrt(Var(X)) · sqrt(Var(Y)) ] in [-1, +1].

5. Systems, Cybersecurity & Machine Learning Applications#

Probabilistic reasoning is the fundamental framework underlying modern algorithms and system architectures.

1. Cybersecurity: Intrusion Detection & The Base Rate Fallacy#

In Security Information and Event Management (SIEM) systems, machine learning detectors monitor server network traffic for cyber attacks.

Suppose:

• An intrusion detection system has 99% Sensitivity: P(Alarm | Attack) = 0.99.
• It has a 1% False Positive Rate: P(Alarm | Clean) = 0.01.
• In a production cloud network, cyber attacks are rare: Prior P(Attack) = 0.001 (1 in 1000 requests).

When the SIEM system raises an alarm, what is the probability that the network is actually under attack?

Using Bayes' Theorem:

P(Alarm) = P(Alarm | Attack) · P(Attack) + P(Alarm | Clean) · P(Clean)

= (0.99)(0.001) + (0.01)(0.999) = 0.00099 + 0.00999 = 0.01098.

P(Attack | Alarm) = [ P(Alarm | Attack) · P(Attack) ] / P(Alarm)

= 0.00099 / 0.01098 ≈ 0.09016 (9.02%).

Result: Over 90.9% of all alarms raised are false alarms!

This phenomenon is the Base Rate Fallacy. Even an extremely accurate detector is overwhelmed by false alarms when the prior base rate of the target event is minuscule. Understanding Bayes' Theorem prevents security teams from experiencing alarm fatigue.

2. Machine Learning: Naive Bayes Text Classification#

In spam filtering, sentiment analysis, and document categorization, text is processed as a bag-of-words vector x = [ x1, x2, ..., xd ].

We seek the class c in {Spam, Ham} that maximizes posterior probability P(c | x):

P(c | x) ∝ P(c) · P(x | c).

Under the "naive" conditional independence assumption that words occur independently given the document class:

P(x | c) = prod_{j=1}^d P(x_j | c).

Log-Likelihood Classification Decision:

c_predicted = argmax_c [ ln P(c) + sum_{j=1}^d ln P(x_j | c) ].

Laplace Smoothing (Additive Smoothing):

If a novel word w appears in a test email that was never observed in training spam, raw frequency yields P(w | Spam) = 0. Because multiplying by 0 destroys all other evidence, Laplace smoothing adds pseudo-counts:

P(x_j | c) = [ count(x_j, c) + 1 ] / [ total_words(c) + |Vocabulary| ].

3. Distributed Systems: Reliability Engineering & MTTF#

In cloud data centers (AWS, GCP, Azure), high availability is achieved via redundancy.

Consider a storage cluster with n independent drives, each having failure probability p during a 1-year period:

• Series System (All must function for system to work):

P(System Works) = (1 - p)^n.

For n = 100 and p = 0.01: (0.99)^100 ≈ 0.366 (63.4% failure probability!).

• Parallel Redundant System (At least one must function):

P(System Works) = 1 - P(All Fail) = 1 - p^n.

For n = 3 and p = 0.01: 1 - (0.01)^3 = 1 - 10^(-6) = 0.999999 (Six Nines availability!).

Probability models guide replication strategies across distributed storage clusters.

6. Common Architectural Pitfalls & Misconceptions#

• Pitfall 1: Confusing Independence with Mutual Exclusivity.

Novice engineers frequently write P(A ∩ B) = 0 for independent events. In reality, independent events satisfy P(A ∩ B) = P(A) · P(B) > 0. Mutually exclusive events satisfy P(A ∩ B) = 0 and are strongly dependent.

• Pitfall 2: The Prosecutor's Fallacy.

Equating P(Evidence | Innocent) with P(Innocent | Evidence). A forensic DNA match might have a tiny false match probability of P(Match | Innocent) = 1 in 100,000. But in a city of 10,000,000 people, there are 100 innocent individuals who match the profile. Without a high prior suspect probability, P(Innocent | Match) remains very high.

• Pitfall 3: Assuming Variance Distributes Linearly Over Dependent Sums.

Writing Var(X + Y) = Var(X) + Var(Y) without accounting for the covariance term 2 · Cov(X, Y). If X and Y are negatively correlated, the variance of their sum is significantly smaller than the sum of variances (the foundational principle of financial portfolio risk diversification).

• Pitfall 4: Neglecting Zero-Frequency Probabilities in Multiplicative Models.

Multiplying raw un-smoothed empirical conditional probabilities can cause a single missing feature to zero out the entire product. Always apply Laplace or Lidstone smoothing in probabilistic classifiers.

7. Summary & Checkpoint Exercises#

Solidify your mastery of Kolmogorov axioms, conditional probability, Bayes' theorem, and discrete expectation with these four checkpoint problems. Full step-by-step mathematical solutions are provided.

Problem 1 (Kolmogorov Axiom Proofs & Probability Algebra):

Let A and B be two events in a probability space (Ω, F, P).

1. Prove rigorously using Kolmogorov's axioms that:

P(A ∪ B) = P(A) + P(B) - P(A ∩ B).

1. Given that P(A) = 0.6, P(B) = 0.5, and P(A ∩ B) = 0.3, calculate P(A ∪ B), P(A | B), P(B | A), and determine whether events A and B are statistically independent.

Solution:

Part 1: Rigorous Axiomatic Proof.

Decompose event A ∪ B into three pairwise mutually disjoint sets:

1. A \ B = A ∩ B^c (Outcomes in A but not B)
2. B \ A = B ∩ A^c (Outcomes in B but not A)
3. A ∩ B (Outcomes in both A and B)

By inspection:

• A = (A \ B) ∪ (A ∩ B), and the two sets are disjoint.

By Kolmogorov Axiom 3 (finite additivity):

P(A) = P(A \ B) + P(A ∩ B) => P(A \ B) = P(A) - P(A ∩ B).

• B = (B \ A) ∪ (A ∩ B), and the two sets are disjoint.

By Axiom 3:

P(B) = P(B \ A) + P(A ∩ B) => P(B \ A) = P(B) - P(A ∩ B).

• The union A ∪ B is the union of all three disjoint components:

A ∪ B = (A \ B) ∪ (B \ A) ∪ (A ∩ B).

By Axiom 3:

P(A ∪ B) = P(A \ B) + P(B \ A) + P(A ∩ B).

Substitute the expressions for P(A \ B) and P(B \ A):

P(A ∪ B) = [ P(A) - P(A ∩ B) ] + [ P(B) - P(A ∩ B) ] + P(A ∩ B)

= P(A) + P(B) - P(A ∩ B).

This completes the proof.

Part 2: Numerical Computations & Independence Check.

Given P(A) = 0.6, P(B) = 0.5, P(A ∩ B) = 0.3:

• Union Probability:

P(A ∪ B) = P(A) + P(B) - P(A ∩ B) = 0.6 + 0.5 - 0.3 = 0.8.

• Conditional Probability P(A | B):

P(A | B) = P(A ∩ B) / P(B) = 0.3 / 0.5 = 0.6.

• Conditional Probability P(B | A):

P(B | A) = P(A ∩ B) / P(A) = 0.3 / 0.6 = 0.5.

• Statistical Independence Check:

Evaluate the product P(A) · P(B):

P(A) · P(B) = (0.6) · (0.5) = 0.3.

Since P(A ∩ B) = 0.3 = P(A) · P(B):

Events A and B are strictly statistically independent!

Notice also that P(A | B) = 0.6 = P(A) and P(B | A) = 0.5 = P(B), confirming independence.

Problem 2 (Bayesian Posterior Updating & The Base Rate Fallacy):

A rare genetic condition affects 0.2% of the population (Prior P(Condition) = 0.002).

A diagnostic laboratory test has:

• True Positive Rate (Sensitivity): P(Positive | Condition) = 0.98.
• False Positive Rate: P(Positive | No Condition) = 0.03.
• A randomly selected person tests positive. Compute the posterior probability P(Condition | Positive).
• What must the test's false positive rate be reduced to in order for a positive result to yield a posterior probability of at least 50%?

Solution:

Part 1: Posterior Probability Calculation.

Define Events:

• C: Person has the condition; P(C) = 0.002.
• C^c: Person does not have condition; P(C^c) = 1 - 0.002 = 0.998.
• T+: Test result is positive.
• P(T+ | C) = 0.98.
• P(T+ | C^c) = 0.03.

Step 1: Compute Total Marginal Probability P(T+) via Law of Total Probability:

P(T+) = P(T+ | C) · P(C) + P(T+ | C^c) · P(C^c)

= (0.98)(0.002) + (0.03)(0.998)

= 0.00196 + 0.02994

= 0.03190.

Step 2: Apply Bayes' Theorem:

P(C | T+) = [ P(T+ | C) · P(C) ] / P(T+)

= 0.00196 / 0.03190 ≈ 0.06144 (6.14%).

Result: Despite a 98% test sensitivity, a person testing positive has only a 6.14% chance of actually having the condition! Nearly 94% of positive tests are false positives due to the low base rate.

Part 2: Required False Positive Rate for P(C | T+) >= 0.50.

Let FPR = P(T+ | C^c). We require:

P(C | T+) = [ (0.98)(0.002) ] / [ (0.98)(0.002) + FPR · (0.998) ] >= 0.50.

0.00196 / [ 0.00196 + 0.998 · FPR ] >= 0.50.

Multiply through:

0.00196 >= 0.50 · (0.00196 + 0.998 · FPR)

0.00196 >= 0.00098 + 0.499 · FPR

0.00098 >= 0.499 · FPR

FPR <= 0.00098 / 0.499 ≈ 0.001964.

Conclusion: The false positive rate must be reduced below 0.196% (approx. 1 in 510) for a positive test to yield a >50% posterior probability.

Problem 3 (Law of Total Probability & Conditional Expectation):

A software microservice routes incoming requests to three server pools:

• Pool A handles 50% of traffic (P(A) = 0.50) with average latency 80 ms and standard deviation 10 ms.
• Pool B handles 30% of traffic (P(B) = 0.30) with average latency 120 ms and standard deviation 20 ms.
• Pool C handles 20% of traffic (P(C) = 0.20) with average latency 200 ms and standard deviation 40 ms.
• Compute the overall expected latency E[L] of the service.
• Using the Law of Total Variance Var(L) = E[Var(L | Pool)] + Var(E[L | Pool]), compute the overall variance and standard deviation of latency.

Solution:

Part 1: Overall Expected Latency.

By the Law of Total Expectation:

E[L] = E[L | A] · P(A) + E[L | B] · P(B) + E[L | C] · P(C)

= (80)(0.50) + (120)(0.30) + (200)(0.20)

= 40 + 36 + 40

= 116 ms.

Part 2: Overall Variance via Law of Total Variance.

The total variance decomposes into two components:

1. Expected Within-Pool Variance: E[Var(L | Pool)].

Var(L | A) = 10^2 = 100.

Var(L | B) = 20^2 = 400.

Var(L | C) = 40^2 = 1600.

E[Var(L | Pool)] = (100)(0.50) + (400)(0.30) + (1600)(0.20)

= 50 + 120 + 320 = 490.

1. Between-Pool Variance of Expected Values: Var(E[L | Pool]).

Recall: Var(Y) = E[Y^2] - (E[Y])^2.

Here Y = E[L | Pool] takes values {80, 120, 200} with probabilities {0.50, 0.30, 0.20}.

E[Y] = 116.

E[Y^2] = (80^2)(0.50) + (120^2)(0.30) + (200^2)(0.20)

= (6400)(0.50) + (14400)(0.30) + (40000)(0.20)

= 3200 + 4320 + 8000 = 15520.

Var(E[L | Pool]) = E[Y^2] - (E[Y])^2 = 15520 - 116^2 = 15520 - 13456 = 2064.

Combine Components:

Var(L) = E[Var(L | Pool)] + Var(E[L | Pool])

= 490 + 2064 = 2554 ms^2.

Overall Standard Deviation:

σ_L = sqrt(2554) ≈ 50.54 ms.

Problem 4 (Joint Distribution, Covariance & Linear Combinations):

Two discrete random variables X and Y have the following joint probability mass function:

• P(X = 1, Y = 1) = 0.2
• P(X = 1, Y = 2) = 0.1
• P(X = 2, Y = 1) = 0.1
• P(X = 2, Y = 2) = 0.6
• Find the marginal distributions of X and Y, and determine their means E[X] and E[Y].
• Compute E[X · Y] and the covariance Cov(X, Y). Are X and Y independent?
• Calculate the variance of the linear combination W = 3 X - 2 Y.

Solution:

Part 1: Marginal Distributions and Expectations.

• Marginal of X:

P(X = 1) = P(1, 1) + P(1, 2) = 0.2 + 0.1 = 0.3.

P(X = 2) = P(2, 1) + P(2, 2) = 0.1 + 0.6 = 0.7.

E[X] = (1)(0.3) + (2)(0.7) = 0.3 + 1.4 = 1.7.

E[X^2] = (1^2)(0.3) + (2^2)(0.7) = 0.3 + 2.8 = 3.1.

Var(X) = E[X^2] - (E[X])^2 = 3.1 - (1.7)^2 = 3.1 - 2.89 = 0.21.

• Marginal of Y:

P(Y = 1) = P(1, 1) + P(2, 1) = 0.2 + 0.1 = 0.3.

P(Y = 2) = P(1, 2) + P(2, 2) = 0.1 + 0.6 = 0.7.

E[Y] = (1)(0.3) + (2)(0.7) = 1.7.

E[Y^2] = 3.1.

Var(Y) = 3.1 - 2.89 = 0.21.

Part 2: Compute E[X · Y] and Covariance.

E[X · Y] = sum x · y · P(X = x, Y = y):

= (1)(1)(0.2) + (1)(2)(0.1) + (2)(1)(0.1) + (2)(2)(0.6)

= 0.2 + 0.2 + 0.2 + 2.4 = 3.0.

Covariance:

Cov(X, Y) = E[X · Y] - E[X] · E[Y]

= 3.0 - (1.7)(1.7) = 3.0 - 2.89 = +0.11.

Independence Check:

Because Cov(X, Y) = 0.11 ≠ 0, X and Y are positively correlated.

Also: P(X = 1, Y = 1) = 0.2, but P(X = 1) · P(Y = 1) = (0.3)(0.3) = 0.09 ≠ 0.2.

X and Y are strictly dependent.

Part 3: Compute Variance of W = 3 X - 2 Y.

Recall the general formula for variance of a linear combination:

Var(a X + b Y) = a^2 · Var(X) + b^2 · Var(Y) + 2 · a · b · Cov(X, Y).

Here a = 3, b = -2:

Var(W) = 3^2 · Var(X) + (-2)^2 · Var(Y) + 2 · (3) · (-2) · Cov(X, Y)

= 9 · Var(X) + 4 · Var(Y) - 12 · Cov(X, Y)

= 9(0.21) + 4(0.21) - 12(0.11)

= 1.89 + 0.84 - 1.32

= 2.73 - 1.32

= 1.41.

The variance of W = 3 X - 2 Y is 1.41.