What are some good resources for doing A/B testing of a website?

Question

I know of Google Website Optimizer and have used it some, and it's pretty good. Are there better options? Any good blogs about A/B testing?

Answer 1

Read a great introduction to A/B testing on the smashingmagazine.com. The piece includes links to multiple A/B testing tools.

Answer 2

One of the key things to understand when doing A/B testing is the statistical significance of your results.

Some good posts on this can be found here:

• Easy statistics for AdWords A/B testing, and hamsters
• Statistical significance & other A/B test pitfalls
• Significance of Statistically Significant Results in A/B Testing

Answer 3

Dustin Curtis wrote an interesting and informative article about his experience a/b testing a specific phrase/link on his site.

Answer 4

Here's a great slidedeck: A/B Testing Framework Design by Patrick McKenzie.

Wikipedia has a good definitional article on A/B Testing:

A/B testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

Answer 5

If you're looking for specific A/B testing tools, A/Bingo is a great plugin for Ruby on Rails sites looking to do A/B testing. And since its built into the framework, you don't have to physically build two pages to test against each other. Instead, you can easily just create two buttons or images and it will handle the rest.