Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I set up a test in Google Website Optimizer that has a 3 variations - original (A), B, and C. In order to further validate the results of the test, I added a variation C that is exactly the same as the original. And thats where the results get weird.

6 days into the test, the best performing variation is C. It outperforms the original by 18.4%! How is that possible?

Do I now discount the results of this test entirely?

share|improve this question
1  
How many times has the test been run? Then we can work out the expected variation between A and C. –  paulmorriss Mar 17 '11 at 15:05
    
Each variation has had about 480 visits and between 74-91 conversions –  Yisroel Mar 17 '11 at 15:09

2 Answers 2

Not enough experiments or divergence in the results to be statistically significant yet. Try doing a Google search on AB Testing confidence and work out what sort of confidence level you're at with your experiment. This will give you a much better idea of whether your results are significant or whether you should just let the experiment run longer.

Speaking from personal experience I cannot tell you how many times I've tested an altered landing page layout that instantly showed a huge improvement in an AB test in the first day or two. But then as the amount of data increases my "improvement" turned out to be a pipe dream/

share|improve this answer

If C was displayed to visitors even a small percentage over A it would have a higher conversion rate. With now 3 variations you cannot distribute your traffic evenly, so there's probably always one variation with slightly more views than another.

I often use the multivariate tool with a single variation for A/B tests on dynamic pages.

6 days is also a short amount of time even with a few hundred visits. But that also depends on what your conversion is considered. Leaving the tests to run a month or more is ideal to start to see a steady pattern. In the beginning of tests it's common to see variations bouncing up and down with conversions. After a while the lines steady out and you'll see a clear winner.

I suggest not adding a variation that is the exact same as your test / original page

share|improve this answer
    
"I suggest not adding a variation that is the exact same as your test / original page" - why? the entire purpose was to see if there is a difference in conversion rate between the A and C, even though they are exactly the same –  Yisroel Mar 13 '12 at 22:15
    
"If C was displayed to visitors even a small percentage over A it would have a higher conversion rate." - I don't understand this either. Why would C have a higher conversion rate just because its displayed a bit more? –  Yisroel Mar 13 '12 at 22:16
1  
Would you post your website optimizer report from beginning to end so we can see the stats? –  Anagio Mar 13 '12 at 22:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.