Probability and Statistics

Why Probability and Statistics?

ML models output predictions under uncertainty. Understanding distributions, expectations, and inference lets you pick the right loss function, interpret confidence, and design experiments correctly. Nearly every ML concept has a probabilistic interpretation.

Probability Fundamentals

Sample Space and Events

• Sample space Ω: the set of all possible outcomes
• Event A ⊆ Ω: a subset of outcomes
• Probability P(A): a number in [0, 1] with P(Ω) = 1

Axioms (Kolmogorov)

1. P(A) ≥ 0
2. P(Ω) = 1
3. If A and B are mutually exclusive: P(A ∪ B) = P(A) + P(B)

Conditional Probability

P(A | B) = P(A ∩ B) / P(B) for P(B) > 0

The probability of A given that B has occurred.

Independence

A and B are independent if: P(A ∩ B) = P(A) · P(B)

Equivalently, P(A | B) = P(A) — knowing B gives no information about A.

Bayes' Theorem

P(A | B) = P(B | A) · P(A) / P(B)

In ML context (posterior = likelihood × prior / evidence):

P(θ | data) = P(data | θ) · P(θ) / P(data)

• P(θ): prior — what you believe about parameters before seeing data
• P(data | θ): likelihood — probability of data given parameters
• P(θ | data): posterior — updated belief after seeing data

Random Variables and Distributions

Discrete Distributions

Continuous Distributions

The Normal Distribution

The bell curve N(μ, σ²) is central to ML because: - The sum of many independent random variables converges to Normal (CLT) - It is the maximum-entropy distribution given fixed mean and variance - Assuming Normal errors in regression gives the ordinary least-squares loss

68–95–99.7 rule: - P(μ − σ < X < μ + σ) ≈ 68% - P(μ − 2σ < X < μ + 2σ) ≈ 95% - P(μ − 3σ < X < μ + 3σ) ≈ 99.7%

import numpy as np
from scipy import stats

# Generate normal samples
samples = np.random.randn(1000) * 2 + 5  # N(5, 4)

# PDF, CDF
x = np.linspace(0, 10, 200)
pdf = stats.norm.pdf(x, loc=5, scale=2)
cdf = stats.norm.cdf(x, loc=5, scale=2)

# P(X < 7) for N(5,2)
p = stats.norm.cdf(7, loc=5, scale=2)  # ≈ 0.841

Expectation, Variance, Covariance

Expected Value

E[X] = Σₓ x · P(X=x) (discrete) E[X] = ∫ x · f(x) dx (continuous)

Linearity: E[aX + bY] = a E[X] + b E[Y] — always, even if X and Y are dependent.

Variance

Var(X) = E[(X − E[X])²] = E[X²] − (E[X])²

Standard deviation: σ = √Var(X) — same units as X.

Covariance and Correlation

Cov(X,Y) = E[(X − μₓ)(Y − μᵧ)]

Correlation: ρ = Cov(X,Y) / (σₓ σᵧ) ∈ [−1, 1]

• ρ = 1: perfectly positively correlated
• ρ = 0: uncorrelated (not necessarily independent)
• ρ = −1: perfectly negatively correlated

Covariance matrix Σ for a vector x of d features: Σᵢⱼ = Cov(xᵢ, xⱼ)

Σ is always symmetric and positive semi-definite. PCA operates on this matrix.

# Covariance matrix
X = np.random.randn(100, 4)
cov = np.cov(X.T)   # shape (4, 4)
corr = np.corrcoef(X.T)

Maximum Likelihood Estimation (MLE)

Given data D = {x₁, …, xₙ} assumed i.i.d. from a distribution parameterized by θ, MLE finds:

θ_MLE = argmax_θ Π ᵢ p(xᵢ; θ) = argmax_θ Σᵢ log p(xᵢ; θ)

Taking logs converts the product to a sum (easier to optimize, avoids underflow).

Example — MLE for a Gaussian: Given samples, MLE gives μ̂ = (1/n) Σxᵢ and σ̂² = (1/n) Σ(xᵢ − μ̂)² — the sample mean and (biased) sample variance.

Cross-entropy loss is MLE for categorical distributions. Minimizing cross-entropy on a classifier IS maximizing the log-likelihood of the correct class.

Information Theory

Entropy

For a discrete distribution p, Shannon entropy measures uncertainty:

H(p) = −Σₖ p(k) log₂ p(k) (bits)

Maximum when all outcomes are equally likely; zero for a deterministic outcome.

Cross-Entropy

H(p, q) = −Σₖ p(k) log q(k)

Used as loss for classification: p = true label, q = model predictions (softmax output).

KL Divergence

KL(p ‖ q) = Σₖ p(k) log(p(k)/q(k))

Measures "extra bits" needed to encode samples from p using a code optimized for q. Always ≥ 0 (0 iff p = q). Not symmetric: KL(p‖q) ≠ KL(q‖p).

Relationship: H(p, q) = H(p) + KL(p ‖ q)

Variational autoencoders explicitly minimize KL divergence.

Mutual Information

I(X; Y) = KL(p(x,y) ‖ p(x)p(y)) = H(X) − H(X|Y)

How much knowing Y reduces uncertainty about X. Used in feature selection and information-bottleneck methods.

Hypothesis Testing

Used to evaluate if a model improvement is real or due to chance.

Common mistake: p < 0.05 does NOT mean the effect is large or important, only unlikely under H₀.

For comparing two ML models, the McNemar test is appropriate (paired binary predictions).

Statistical Estimators

Bias-variance tradeoff for estimators: sample variance divides by (n−1) not n to get an unbiased estimate of population variance.

Central Limit Theorem (CLT)

Let X₁, …, Xₙ be i.i.d. with mean μ and variance σ². Then:

(X̄ − μ) / (σ/√n) → N(0, 1) as n → ∞

This is why means and sums converge to Normal distributions, justifying many statistical procedures regardless of the original distribution's shape.

Distributions Arising in ML

Softmax (Categorical)

Converts logits z into probabilities:

pₖ = exp(zₖ) / Σⱼ exp(zⱼ)

Outputs are positive and sum to 1. Temperature T scales sharpness: softmax(z/T) — low T → one-hot, high T → uniform.

Sigmoid (Bernoulli)

σ(z) = 1 / (1 + e⁻ᶻ) ∈ (0, 1)

Squashes a real number to a probability. Used for binary classification output and gates in LSTMs.

Dirichlet Distribution

Generalization of Beta distribution over probability vectors. Used as a prior over categorical distributions in topic models (LDA).

Sampling Methods

# Monte Carlo estimate of π
n = 1_000_000
x, y = np.random.uniform(-1, 1, (2, n))
inside = (x**2 + y**2) <= 1
pi_estimate = 4 * inside.mean()  # ≈ 3.1415...