Coin flips, sample means, and the gambler's ruin
The Law of Large Numbers (LLN) is one of the most fundamental theorems in probability. It says that as you take more and more independent samples from a distribution, the sample average converges to the true expected value. This is why casinos always win in the long run, why polls become more accurate with larger samples, and why insurance companies can predict costs.
The weak LLN says convergence happens in probability; the strong LLN says it happens almost surely (with probability 1). Both versions require finite mean, but say nothing about the rate of convergence -- that's where the Central Limit Theorem comes in.
The simplest illustration: flip a fair coin and track the running proportion of heads. After 10 flips, you might see 70% heads or 30% heads -- that's normal randomness. But after 10,000 flips, the proportion is almost certainly within 1% of 0.5. Hit "Run" and watch it happen in real time.
Watch the proportion of heads converge to 0.5 as the number of flips grows. The shaded band shows the expected 1/√n convergence rate.
Key insight: The proportion converges, but the absolute deviation |heads - n/2| actually grows like √n. The LLN says the ratio converges, not that fluctuations disappear -- they just become negligible relative to the total.
The LLN works for any distribution with finite mean, not just coin flips. Each colored path represents an independent sequence of running sample means from the same distribution. Watch all paths converge to the true mean (dashed line), no matter how they start.
Each colored line is an independent sequence of running sample means. All paths start volatile but converge to the true mean (dashed line) -- this is the Law of Large Numbers.
Key insight: Try the exponential distribution -- early sample means are volatile because the distribution is skewed and has high variance. But convergence is guaranteed nonetheless. The speed of convergence depends on the variance: lower variance means faster convergence.
A gambler starts with $N and bets $1 each round, winning with probability p. They stop when they reach a target or go broke. When p = 0.5 (fair game), the gambler is eventually ruined with probability (target - start)/target. Even a tiny edge (p = 0.49) dramatically increases the ruin probability -- the law of large numbers ensures the house edge accumulates relentlessly.
10 gamblers each start with $50 and bet $1 per round. They win each bet with probability p and stop when they hit $100 (win) or $0 (ruin). At p=0.5, ruin probability = 1 - start/target.
Key insight: Try p = 0.49 vs p = 0.51. With p = 0.49, almost every gambler goes broke. With p = 0.51, most win! This tiny 2% difference becomes decisive over thousands of bets -- the very essence of the LLN applied to gambling.