Hypothesis Testing

Note

• [wiki] A/B testing

• [paper] Top Challenges from the first Practical Online Controlled Experiments Summit

  • Section 5: Heterogeniety in Treatmnt Effect

  • Section 3: Long-term effects

  • Section 4: Overall evaluation criterion metric

• The Example of Computational Advertising Bing: Counterfactual Reasoning and Learning Systems

• Estimation and Inference of Heterogeneous Treatment Effects using Random Forests

• [chrisstucchio.com] Bayesian Bandits - optimizing click throughs with statistics

• Amazon papers

See also

• [nathanieldake.com] Bayesian A/B Testing

• Bandit Algorithms

• Misunderstandings between experimentalists and observationalists about causal inference

• useless [interviewquery.com] Statistics & A/B Testing Interview Questions

Classical Inference

Generic Setup

Likelihood Ratio Test (LRT)

Specific Setup

Mean Difference : Wald Test

Multinomial Parameter Value : Chi-Squared Test

Two Sample Test : Permutation Test

Bayesian Inference