Beyond basic A/B testing

A/B testing became the standard tool of online businesses to aid almost every decision. A lot of software offer easy solutions without the requirement of understanding the statistics behind. However, the success of such tests hinges upon the details of the implementation and interpretation that do require the basic understanding of the underlying theory.

My presentation gives a detailed yet intuitive introduction to the statistics of A/B testing, with many practical tips and tricks that follow. I offer some guidance about what measure to use regarding their variance (e.g. conversion rate or amount of purchase), and how to construct groups to achieve higher precision (randomized block design). I draw the attention to the common mistake of multiple comparison, and talk about a new line of research that handles this problem (casual trees). I use a real-world example implemented in R for illustration.

Divényi János
Data scientist, Emarsys

I am a PhD Candidate in Economics at Central European University, currently working as a data scientist for Emarsys. We strive for valuable insights in the field of online marketing to help our customers to target their audience in a one-one, personalised way. My primary task is to give established answers backed up by visualisations and dashboards to broadly defined research questions regarding our product. I want to excel in how to communicate complicated methods and sophisticated concepts in an intuitive manner that I practice also as a teaching fellow at various institutions (Central European University, Budapest University of Technology and Economics, Mathias Corvinus Collegium, and Rajk László College).