A Closer Look at A/B testing




How do you verify your hypothesis?






Is there another way?




4 thoughts on “A Closer Look at A/B testing

  1. How do you account for door A and B not being shown the same number of times? e.g. if you look at the number of conversions and you see that A delivered more conversions than B the result is clearly different if B was shown 10% of the time or 50% of the time.


  2. Hi Ryan,
    I think you misunderstood the post. users don’t have choice, they just been sent through door A or door B


  3. I’m not sure that the door A and door B example you use accurately describe the A/B testing that is normally done in internet marketing. Users are not presented with door A and door B, then choose one. They are presented with door A or door B, then a behavior either happens or it does not. I would argue that this results in two different populations and that “conversion rate” is an attribute of the population. Additionally, I think marketers get far too concerned with statistical significance in A/B tests and fail to understand the reasons for the behaviors that are taking place.


Comments are closed.