Should I disallow robots on A/B test pages?

Question

I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php.

Should I be putting in rules to my robots.txt to disallow search engines from spidering these pages? I know there is an SEO punishment for duplicate content--which is why I ask--but I'm not sure if Google exempts pages that are actively using WebsiteOptimizer.

Thoughts?

Answer 1

Instead of doing something with your robots.txt which doesn't always do what you want, the better option is to just include a <link rel='canonical' href='XYZ' /> tag in all version of the page you are A/B testing and just make XYZ equal to the original/master version of the page. That way you know for sure that Google is crawling at least one page. That way Google will find the 1 version of the page you want them to even if they get sent to one of the other versions of the page.

Another alternative is to use <meta name="robots" content="noindex"> in each pages header except for the original/master page. Doing it this will way will Google to pass through each of the test pages and see the rest of your pages in the site but it won't give rankings to any specific page. I think this is a less ideal solution but if for some reason your other page versions have links that aren't in your master version it will at least still give you some value.

Answer 2

WebsiteOptimiser redirects to your a/b test cases, so Google itself should be ok, but...

I add exclusion rules to robots.txt, as it is possible (albeit not that likely) that someone goes to your site whilst you're testing, and reposts the link ./index_b.php to their blog etc, from which point your split test URL is likely to be crawled.

So much depends on the volume of traffic you're likely to encounter during your testing, and how long you're running test cases for.

Once you're done, 301 redirect to the normal index.php page from the old test cases.