Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We've setup an A/B in Optimizely to test a completely new design:

http://offerta.se

that we're testing against our legacy design:

http://offerta.se/?classic=1

Optimizely is setup to redirect 50% of our CPC traffic to the legacy design.

The old design is beating the new with about 10% better convertion rate and in trying to fiqure out why we started looking at if there was anything standing out in Google Analytics.

Optimizely creates the events with labels "New design" and "Legacy design" for us in GA.

And filtering on these and looking at "Screen Resolution=(not set)" we see some weird numbers:

enter image description here

If we remove all traffic with "Screen Resolution=(not set)" the new design instead performs 10% better than the old.

Anyone that can enlighten me?

share|improve this question

1 Answer 1

According to the following article "Screen Resolution=(not set)" means that there was no information regarding the visitor's screen resolution saved: https://support.google.com/analytics/answer/2820717?hl=en

Looking at your data for all aggregated visitors, or removing the visitors from which the screen resolution is unknown, is basically looking at different segments of your data.

It can be that visitors with different screen resolutions react differently to your designs. So including or excluding screen resolutions (or visitors from which no screen resolution is known) can have a different result.

If you would like optimize your website for overall traffic you should look at the aggregated result.

To further elaborate: (from Google Support)

It is quite common that the screen resolution that has the highest Ecommerce conversion rate is (not set). Additionally, a high incidence of (not set) Ecommerce rates can occur for these dimensions:

Flash Version (occurs if Flash is disabled in browser settings)    
Screen Colors    
Language (in demographics report)    
Generally, these traffic segments are small, and so are not statistically significant.

and

Browser and Screen Resolution

Use Browser or Screen Resolution as the dimension so you can see whether traffic is distributed among browsers or screen resolutions as you expect, and to see whether there are any pages that experience an unusual drop-off with a particular browser or screen resolution. If you notice a significant drop-off from a page, it might not be rendering properly in that browser or at that resolution. For example, your page may not render correctly on a mobile browser or at a smaller resolution, causing links or buttons to be unavailable or not easily seen."

Also, your results are correctly showing 50/50 allocated results on your two variants (and their target) offerta.se/google and the redirect: offerta.se/google?classic=1. The GA report through the screen resolution=(not set) string is showing you segmented data that Optimizely has no affect over.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.