6

I have set-up and simple A/B test with 100% of my traffic, everything seems to be running fine except the original page and the page variants aren't getting an equal split of visits.

It seems that the highest performing page is getting the most visits but my sample size is still relatively small.

2

Tricky question, the default settings for Google's content experiements are a 2 week duration with a confidence level of 95%. If you got low traffic volume, these confidence levels may not be reached, because of differences in user behaviour on different days of the week (weekdays vs. weekends).

Even if you implement a full 100% traffic split, the testing suite may shift traffic volumes to different variants trying to find confidence levels and thus be able to declare a winner.

TL;DR
A/B testing does not route the first visitor to variant A and the second to variant B. Instead it's trying to segment types of traffic to be able to compare them. I have never seen a A/B test where traffic was split exactly 50% (between two variants), no matter how long the experiement was running.

| improve this answer | |
  • Can you provide a source for your answer please? – Heraldmonkey Apr 29 '14 at 7:16
5

Google content experiments uses a technique called "multi-armed bandit" methodology.

https://support.google.com/analytics/answer/2844870?hl=en-GB&ref_topic=2844866

Basically, it seeks to speed up the test and increase conversions throughout the test by sending more traffic to the 'winning' pages throughout the experiment.

If version 2 performs better on day one then it will receive more traffic the following day and the cumulatively afterwards. The original page still gets traffic so if that then performs well one day it will get a larger share of traffic the following day.

There are lots of thoughts about multi-armed bandit testing vs 50:50 testing but basically, don't worry as with google content experiments there isn't anything you can do about it.

| improve this answer | |
  • I believe that this should be the accepted answer to the original question – Matt Wilko Jan 5 '16 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.