Probability

(Adapted from Bertsekas and Tsitsiklis - Introduction to Probability)

• Probability Axioms (Kolmogorov Axioms)
• Random Variable
  • Discrete Random Variable
    • Probability mass function
    • Expectation and Variance
      • Law of The Unconscious Statistician (LOTUS)
      • Moments of a rv
      • Expectations of linear functions of rv
      • Variance of bounded rv
      • Expectations of general functions of rv
        • Jensen’s inequality
    • Multiple discrete random variables
    • Conditioning
      • Bayes theorem
      • Total law of probability
      • Conditional expectation
      • Law of iterated expectation
    • Notion of Independence
      • Expectation and variance for independent rvs
        • Mean and variance of sample mean
    • Some discrete random variables
      • Bernoulli
      • Multinoulli
      • Uniform
      • Binomial
        • The Birthday Problem
        • The Hat Problem
      • Geometric
      • Multinomial
      • Poisson
  • Continuous Random Variable
    • Probability density function
      • Probabilistic interpretation
    • Expectation and Variance
      • Centerisation, standardisation, skewness and kurtosis
      • Cauchy-Schwarz inequality
    • Cumulative distribution function
      • Properties of CDF
    • Multiple continuous random variables
      • Probabilistic interpretation
    • Conditioning
      • Conditioning on an event
        • Probabilistic interpretation
        • Total probability theorem
      • Conditioning on a rv
        • Conditioning on a continuous rv
          • Probabilistic interpretation
          • Definition of probability conditioned on an event with 0 probability
        • Conditioning on a discrete rv
          • Probabilistic interpretation
        • Total probability theorem
      • Bayes theorem
      • Conditional expectation
    • Notion of Independence
    • Some continuous random variables
      • Uniform
      • Exponential
      • Laplace
      • Gaussian
      • Multivariate Gaussian
        • Important Properties of Gaussian Densities
• Functions of Random Variable
  • Density of a function of a rv
    • Special cases
      • Linear functions
      • Monotonic functions
  • Density of a function of multiple jointly distributed rvs
    • Special cases
      • Sum of independent rvs: Convolution
        • Discrete case
        • Continuous case
  • Covariance and correlation
    • Scalar valued rvs
    • Vector valued rvs
  • Fundamentals of Point Estimation
    • Bayesian point estimation using conditional expectation
    • Conditional variance
    • Law of iterated variance
  • Transforms of rv
    • Moment Generating Function
    • Integral Transforms
      • KL Divergence
      • Integral Probability Metric: Wasserstein Distance
      • Integral Probability Metric: Maximum Mean Discrepancy (MMD)
• Tail Bounds, Convergence and Limit Theorems
  • Tail Bounds : Inequalities
    • Mean known: Markov
    • Mean and variance known: Chebyshev
    • MGF known: Chernoff’s Bound
    • Distribution known (Gaussian): Mill
  • Convergence of sequences involving rvs
    • Convergence in distribution
    • Convergence in probability
    • Convergence in \(L_1\)
    • Convergence in quadratic mean
    • Almost surely convergence
  • Convergence of sequences involving parametric models
    • Convergence of functions
      • Point-wise convergence
      • Uniform convergence
    • Convergence of statistical functionals
      • Point-wise convergence in probability
      • Uniform convergence in probability
  • Limit Theorems
    • Weak Law of Large Number
      • Special case: bounded rvs
        • Tail bounds from Chebyshev’s inequality
        • Tigher bounds from Hoeffding’s inequality
    • Strong Law of Large Number
    • Central Limit Theorem
    • The Delta Method
• Stochastic Processes
  • Bernoulli Process
  • Poisson Process
  • Markov Process
  • Gaussian Process
• Information Theory
  • Entropy and Mutual Information
• Graphical Models