Probability & Statistics

From the axioms to inference, regression and Bayes.

About this course

Fourteen lectures from the axioms of probability to practical statistical inference. It establishes sample spaces, conditional probability and Bayes' theorem, then random variables, the key distributions, joint distributions and moment generating functions. The law of large numbers and the central limit theorem explain why randomness becomes predictable at scale, which sets up sampling distributions, maximum-likelihood estimation, confidence intervals and hypothesis testing. It closes with linear regression, Bayesian inference, and a capstone on resampling and ANOVA.

What you will be able to do

  • Apply the probability axioms, conditional probability and Bayes' theorem to non-trivial problems
  • Work with random variables, expectation, variance and the standard distributions
  • State and use the law of large numbers and the central limit theorem
  • Construct point estimates, confidence intervals and hypothesis tests, and interpret them honestly
  • Fit and interpret linear regressions, and carry out basic Bayesian inference

What this course covers

  • Bayesian inference: the philosophical shift
  • Bayes' theorem applied to continuous parameters
  • Conjugate priors and the beta-binomial update
  • Credible intervals and the honest probability statement
  • Data swamping the prior
  • Point estimates from the posterior: MAP, mean, and median
  • Bayesian regularisation and the bridge to machine learning
  • Sample Space and Events as Sets
  • Kolmogorov's Three Axioms of Probability
  • Frequentist vs. Bayesian Interpretations
  • Equally Likely Outcomes and Counting Tools
  • The Birthday Problem
  • Solving the Chevalier de Méré's Bets
  • Course Spine and Road Ahead
  • What makes an estimator good: bias, variance, and MSE
  • Method of moments estimation
  • Maximum likelihood estimation
  • Asymptotic properties of MLE
  • The bias-variance tradeoff
  • MLE for the normal distribution and its bias
  • Clause nine: to infer is to invert
  • Joint distribution and its marginals
  • Conditional distributions and independence
  • Covariance: measuring direction of co-movement
  • Correlation coefficient and its limits
  • Variance of a sum and the covariance cross-term
  • Correlation is not causation
  • End-to-end worked example and course spine
  • Conditional Probability and Its Rules
  • Independence vs. Mutual Exclusivity
  • Law of Total Probability
  • Bayes' Theorem
  • The Base-Rate Fallacy
  • Monty Hall Problem as Conditioning
  • End-to-End Bayes Calculation
  • Change of Variable Formula for Densities
  • Non-Monotone Transformations and Branch Summation
  • Convolution of Independent Random Variables
  • The Moment-Generating Function (MGF)
  • MGF of a Sum: Multiplication Rule
  • Uniqueness Theorem and Distribution Identification
  • End-to-End MGF Worked Case: Exponential Distribution
  • The Bootstrap: Resampling for Any Statistic
  • Why the Bootstrap Is Valid, and Its Limits
  • Analysis of Variance (ANOVA)
  • The Course Arc: Randomness → Learning → Prediction
  • Statistics as the Foundation of Machine Learning
  • Course Field Notes: Fourteen Clauses on One Card
  • Five Takeaways and the Road Ahead
  • What a confidence interval is and what confidence means
  • Building the Z-interval for a mean with known spread
  • The T-interval for unknown spread and the proportion interval
  • Three trade-offs controlling interval width
  • Confidence intervals as standing hypothesis tests
  • End-to-end worked example and capstone summary
  • The 'Flaw of Averages' and the Two-Number Summary
  • Random Variables and Probability Distributions
  • Expectation as Center of Mass
  • Linearity of Expectation
  • Variance, Standard Deviation, and LOTUS
  • Law of Large Numbers (Preview)
  • Worked Examples: Die and Uniform Distribution
  • Logic of hypothesis testing and the null hypothesis
  • The P-value: definition and common misconception
  • Type I and Type II errors and the alpha trade-off
  • Power, its levers, and one-sided vs two-sided tests
  • Duality between hypothesis tests and confidence intervals
  • Multiple comparisons and the replication crisis
  • Additional pitfalls: P-hacking, base rate fallacy, and significance vs importance
  • Sample mean as a random variable
  • Law of large numbers
  • Central limit theorem
  • Two laws at two zoom levels
  • When is n large enough? Berry-Esseen and the role of skewness
  • Chebyshev vs CLT: a worked comparison
  • Spine clause seven and preview of sampling distributions
  • The discrete trial family: Bernoulli, binomial, geometric
  • The Poisson distribution: the law of rare events
  • The continuous uniform distribution
  • The exponential distribution and memorylessness
  • The normal distribution: the 68-95-99.7 rule and standardisation
  • Central limit theorem previewed: sums produce the normal
  • Matching the story to the distribution: the modeling principle
  • Sample mean and variance as random statistics
  • Chi-squared distribution
  • Student's t-distribution
  • F-distribution and the decision table
  • Cost of not knowing σ: a worked comparison
  • Spine clause eight: to estimate is to distribute
  • Regression to the Mean and Its Misinterpretation
  • The Linear Model and the Principle of Least Squares
  • Closed-Form Solution: Slope as Covariance over Variance
  • Goodness of Fit: R-Squared and the LINE Assumptions
  • Degrees of Freedom in Regression
  • Inference on the Slope: T-Test and Confidence Interval
  • Two Traps and the Bridge to Machine Learning

Syllabus

  1. Probability Foundations

    1. What Is Probability? — Sample Spaces, Events & the Axioms
    2. Conditional Probability, Independence & Bayes' Theorem
    3. Random Variables — Expectation & Variance
    4. Key Distributions

  2. Randomness at Scale

    1. Joint Distributions
    2. Transformations & MGFs
    3. LLN & the Central Limit Theorem
    4. Sampling Distributions

  3. Statistical Inference

    1. Point Estimation & MLE
    2. Confidence Intervals
    3. Hypothesis Testing
    4. Linear Regression & Least Squares

  4. Models & Bayes

    1. Bayesian Inference
    2. Capstone — Resampling, ANOVA & the Road to ML