The book on A/B testing statistics

Book: Statistical Methods in Online A/B Testing
Welcome to the most in-depth book on statistical methods in online controlled experiments, a.k.a. A/B testing! It is the culmination of many years of work by the author Georgi Georgiev and aims to bring much needed clarity and context to the application of statistical tools for business decision-making and risk management through experimentation.

"This is the book I wish I had when I started learning A/B testing stats!"

- Georgi Georgiev, the Author

What readers are saying

Some of the reviews given by readers of "Statistical Methods in Online A/B Testing"

Peep Laja
Founder and Principal, CXL Institute

"If you're running online experiments, you absolutely need this book. It will help you avoid costly testing mistakes stemming from misuse and misunderstanding of statistics."

* you absolutely need this book *
Alex Birkett
Senior Growth Marketing Manager, HubSpot

"This is simultaneously the most comprehensive and the most accessible book on A/B testing statistics I've encountered. If you run tests, then, this book should be required reading. [...] Read it, and then gift it to the rest of your team to read, and watch your program's maturity rise in front of your eyes."

* comprehensive and accessible *
Barry Garner
Barry Garner
Sales Development Manager, SiteGainer

"This book transforms the often-complex area of statistical modeling into an understandable and useful framework for those will little or no math background. Starting from the ground up, it introduces the reader to important statistical concepts with the goal of making the science behind A/B testing interpretable and useful."

* understandable & useful *
Evan Miller
Evan Miller
Statistics Engineer @ Eppo

"If I could only recommend one book on A/B testing, this is it. "Statistical Methods in Online A/B Testing" is comprehensive, innovative, and practical – and dives into the underlying equations without getting caught up in formalism. Highly recommended."

* highly recommended *
Philip Cross
CEO & Conversion Rate Optimizer, Premeni Services Company

"Finally, a comprehensive reference on statistics for my own industry! No longer do I need 8 different books on statistics that were each bought for only one or two chapters each, because Georgi has delivered a text whereby each and every chapter is applicable to success in my daily work. Thank you for making such a complex subject both interesting and easier to understand."

* a comprehensive reference *
Hristian Kambourov
Hristian Kambourov
Chief Conversion Craftsman, Revise

"One of the few books that would actually make you a better marketer and bring you more sales. Period. No fluff, straight to the point, practical manual you need on your desk. Pick up a copy of this book and finally understand how to properly evaluate, analyze and use your data when running A/B tests. All masterfully delivered by a true practitioner and an excellent teacher of the somewhat complicated world of statistics."

* no fluff, practical manual *

Topics covered in the book

Below is a short list of the main topics covered in this A/B testing book. Click here to see the full Table of Contents (PDF), as well as a preview of the introductory pages of the book.
Risk management & Decision-making

The role of statistics in business decision-making. The art and science of balancing risk & reward through optimal test design.

Types of Statistical Hypotheses

Testing for superiority, non-inferiority, and strong superiority. One-sided versus two-sided hypotheses.

Statistical Models in A/B Testing

Statistical assumptions, assessing model adequacy, nominal vs. actual error rates. External validity of experiment outcomes.

Multivariate tests a.k.a. A/B/N tests

Understanding MVTs. Statistical adjustments for p-values and confidence intervals. Complications in power and sample size calculation.

P-values & Confidence Intervals

Calculation and interpretation of p-values and confidence intervals. What is and what isn't warranted by a statistically-significant outcome.

Sequential Testing

Statistical tests and decision-making in the presence of continuous monitoring. Group sequential tests with alpha and beta spending.

Statistical Power & Sample Size Calculations

Calculation and understanding of the concept of statistical power and the false negative error rate. Sample size formulas.

& Much more

Handling segmentation, multiple KPIs, percentage change as KPI, running multiple tests at the same time, communicating statistical results...

Learn how to gain business insights and control risk through statistical methods

This book will help you navigate the statistical jungle of jargon & formulas. You will understand the advantages and limitations of statistical approaches for assessing the uncertainty of data from online controlled experiments. You will learn how to design efficient statistical tests and to adapt statistical models in a number of practical scenarios, primarily from e-commerce A/B testing.

Who is this book for?

This A/B testing book would be most useful to:

  • Conversion rate optimizers
  • Landing page optimizers
  • Data analysts & web analytics specialists
  • Product managers & growth experts
  • UX & user testing specialists

and anyone else interested in a deep understanding of A/B testing statistics.

About the author

Georgi Georgiev

Georgi Z. Georgiev

Founder of Analytics-Toolkit.com

Georgi is a veteran digital marketer, data analyst, and statistician specializing in statistical methods for online controlled experiments. He is the author of the popular blog at Analytics-Toolkit.com, several technical whitepapers on statistical methods in A/B testing, a comprehensive glossary of A/B testing statistical terms, and primary contributor to the OneSided project. He has developed a number of statistical tools used by the A/B testing industry.

Georgi is a lecturer at industry events and the author of the CXL Institute course "Statistics for A/B Testing". He's on the judge panel of Experimentation Elite Awards since their founding in 2020. Georgi was distinguished as the winner of the 2024 Experimentation Thought Leadership Awards in the Data & Analytics category. Follow Georgi on LinkedIN.