Hot answers tagged

10

Read a great introduction to A/B testing on the smashingmagazine.com. The piece includes links to multiple A/B testing tools.


8

The Visual Website Optimizer blog has an A/B test duration calculator designed to answer your question. It's published as an Excel spreadsheet, but don't let that put you off. Enter your original conversion rate, the difference you're hoping to detect, the number of variations you're testing, and your average daily traffic, and it spits out the number of ...


7

One of the key things to understand when doing A/B testing is the statistical significance of your results. Some good posts on this can be found here: Easy statistics for AdWords A/B testing, and hamsters Statistical significance & other A/B test pitfalls Significance of Statistically Significant Results in A/B Testing


5

I hope I'm understanding your situation correctly. If I'm not, leave a comment and I'll clean it up a bit. In your situation, I'm not sure how well A/B testing will work - I'm concerned with being able to get usable results. There are two problems to overcome - getting statistically valid results and then understanding those results. The first problem that ...


3

Someone in the client's office read an article about how all the large agencies do A/B testing of concepts prior to going live....so it must be important for them to do too! All well and good if you're being paid large agency money for large agency results on a large agency site... unfortunately, as Kris and Thomas Owens have noted, statistical ...


3

One option would be to set up multiple channels for each site, and then use those channels for your split testing. Earnings on a particular page in analytics could be somehow done if you add a parameter to one version of the page depending on your frameowrk and how you are modifying the pages themselves for the test. I am not quite sure how the different ...


3

This mighty list has a selection of A/B & MVT tools: http://www.whichmvt.com/ No relationship with either the list compilers or any of the listed vendors (we have out own in-house MVT tool).


3

A/B testing has its place but it requires some things to hold. Most importantly you need a clear success metric. I.e. it must be straightforward to judge which performed better A or B. Primarily this holds if your site is oriented at a specific commercial goal (e.g. get people to place purchase orders, get people to click on ad links etc.). You also need ...


3

Google website optimizer is free and used by a lot of people I've not tested it with Drupal but here's a link with step by step instructions so I would think you would have no problems http://www.drupalseobook.com/news/google-website-optimizer-with-drupal. (since it's free it's easy enough to test it first). A paid option which I use and highly recommend is ...


3

I would be reluctant to do it. I assume you're trying to optimize your conversions. Here's why: You're breaking a well known UI pattern. It will cause confusion and unease to see something "new" like this. If the field is a password field (i.e. masked characters) there will be a decent number of people that will type their password wrong, or will freak ...


3

You are looking to do "Multivariate Testing" where you test more than just two variables. Here is a guide to walk you through setting up Google Adsense tests using Google Website Optimizer and Google Analytics. Here is a summary of the steps involved from the linked article: Create Google Analytics and Website Optimizer accounts if you don't already ...


3

You could do it on the same page by passing a parameter to the page, so http://www.yourwebsite.com/page.html?version=2 and http://www.yourwebsite.com/page.html Set the colour of the button based on the parameter sent to the page. Google Experiments work by redirecting anyone who lands on http://www.yourwebsite.com/page.html to any URL of your choice ( ...


3

MixPanel is reasonably priced, has an excellent feature set and has emailed reports.


2

Not enough experiments or divergence in the results to be statistically significant yet. Try doing a Google search on AB Testing confidence and work out what sort of confidence level you're at with your experiment. This will give you a much better idea of whether your results are significant or whether you should just let the experiment run longer. ...


2

Instead of doing something with your robots.txt which doesn't always do what you want, the better option is to just include a <link rel='canonical' href='XYZ' /> tag in all version of the page you are A/B testing and just make XYZ equal to the original/master version of the page. That way you know for sure that Google is crawling at least one page. ...


2

Look for the _utmx cookie for the site in question. From Google You can use Google Analytics with Google Website Optimizer (GWO), which is a tool that helps determine the most effective design for your site. When a website optimizer script executes on your page, a _utmx cookie is written to the browser and its value is sent to Google Analytics. See the ...


2

Confidence threshold (or level) is a statistical term which, in a nut shell, denotes the confidence with which you can say that experiment A is better than experiment B. It denotes the statistical significance with which you can declare the winner. You can read more about it here - http://www.stat.yale.edu/Courses/1997-98/101/meancomp.htm To reach higher ...


2

In the past, I've done limited A/B testing and just used a modulus operator on the page to split on a random number. Then I showed one or the other items, and tracked in the log file... Not ideal I know. I did see this though that may give you more info down the path you're looking for... ...


1

Google Website Optimizer sets the ___utmx cookie - your site can respond to this cookie (either via server-side script or via GWO's default Javascript) Using Javascript to display text content is suboptimal given that search engine spiders will miss the content: I would recommend serving your default "A" content for users with ___utmx variant A and for ...


1

If you're looking for specific A/B testing tools, A/Bingo is a great plugin for Ruby on Rails sites looking to do A/B testing. And since its built into the framework, you don't have to physically build two pages to test against each other. Instead, you can easily just create two buttons or images and it will handle the rest.


1

Dustin Curtis wrote an interesting and informative article about his experience a/b testing a specific phrase/link on his site.


1

Here's a great slidedeck: A/B Testing Framework Design by Patrick McKenzie. Wikipedia has a good definitional article on A/B Testing: A/B testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct ...


1

What you ask is a cost problem in your UX evaluation. As you know, cost = frequency (how many times a problem occurs) x gravity (the cost of the problem for you and the user). You can have a first evaluation of the cost for the repeated field : it always append you can measure the evident cost of this field, ie the cost you can mesure without a test ...


1

Why wouldn't you use Google Website Optimizer? On dynamic pages such as big ecommerce sites with template systems and SEF URL's I use the multivariate option with a single variation rather than the A/B test which forces a redirect to a static page. I've also ran multivariate tests with 8 variations of a single pages content. I would use GWO as Google will ...


1

We visual website optimizer http://visualwebsiteoptimizer.com and there is a drupal module for that too, it's made us convert a good deal better, can't recommend it enough.


1

This answer is Drupal specific, as I believe your question was project/module based. We've used Multivariate to do a range of multivariate or AB testing. I know there is a Webform AB test module as well, but the options are somewhat more limited. While this question was over a year ago and you've no doubt moved on - hopefully someone else can benefit from ...


1

Google's Content Experiments is meant for separate A/B URLs, but it can be hacked to use the same URL, as described in this article. The general idea is to: Use the usual Content Experiments wizard, Fill the 2 A/B URL fields with dummy URLs, Insert JavaScript code to modify page content on the client side. Thanks Eike for the link!


1

What you're looking for is Google's Content Experiments. You're right, this is an A/B test, most specifically a content test in order to take conclusions on what is the highest converting page. It's possible to send just a percentage of the traffic as you request and the results are all inside Google Analytics, so you can integrate with Goals and ...


1

A/B testing via PHP will not break your SEO as long as you aren't using any kind of SEO hacking "tricks". Webpages around the web have all kind of different PHP scripts and companies like Google are pretty good at figuring out what is going on. Webpage A: Has structured information and links. Webpage B: Has (generally) the same structured information and ...


1

If the content and internal link structure is different between both servers (I am not talking about the presentation), bots will notice the changes and will likely increase their visits to your site. They will be very confused if they are being served different content back and forth and this may trigger a manual review. Google wants to be sure about the ...



Only top voted, non community-wiki answers of a minimum length are eligible