Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've been running a number of A/B tests using Google Content Experiments, and have had the same problem across all the tests I've run in the last couple weeks. One variant (the original page) receives about 2.5 times as many visitors as the other.

The pages I'm testing receive about 40,000 visits a week, and I've run single tests for over a week, so it's astronomically unlikely that this imbalance is just a statistical anomaly.

Any ideas on why this imbalance might be occurring?

EDIT: Turns out this is a "feature" (per obscure Google documentation). My question, then, is how can I turn this feature off?

share|improve this question
    
I don't understand the question. Maybe some information is left out? The Google tool you apperently use ("Multi-armed bandit experiments") is designed shift traffic towards the most succesful page. Since traffic imbalance is the whole point, you can't "turn it off". –  Free Radical Jan 7 '13 at 15:01
    
I'm using Google Content Experiments (Google's platform for A/B testing). I do not want the tool to automatically shift traffic to the pages it thinks is successful, though. (Because of the site's low conversion rate, I'd like to get more data before trusting one page over another---at present, the tool is making decisions based on fewer than 10 conversions.) Instead, I want to run the experiment until it reaches my own pre-determined confidence levels. –  s3cur3 Jan 7 '13 at 15:27
2  
On the page Overview of Content Experiments, it says: "Content Experiments is a somewhat different approach from either standard A/B or multivariate testing." I.e. according to Google, it is not a platform for standard A/B testing. –  Free Radical Jan 7 '13 at 15:33

1 Answer 1

up vote 1 down vote accepted

the imbalance is probably well explained by the "multi-armed bandit" approach that Google's statistical engines take in the back-end. I have noticed this with other users too. However, have you tried setting the option "Distribute traffic evenly across all variations" to ON? Google does the auto splitting and auto allocation when you don't explicitly say to distribute the traffic.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.