What is a good solution for UA testing multiple projects simultaneously?

Question

My client often has several projects/tasks going at once that sometimes need to be tested simultaneously on one website. They are often separate applications on the website, but sometimes share UDFs, etc. We currently have 3 public-facing environment websites - i.e. dev.website.com, test.website.com, and www.website.com.

As the programmer, I'm trying to find a good solution to allow for UA testing of multiple projects/tasks at once. Currently, I'm finding myself switching between code branches (using SubVersion).

What some of my options?

Answer 1

My favorite is quite a mix of projects (I've been automating quite some tests in my career).

Skipping Unit- and Integration testing...

For the front-end UI testing I use Watin. To make the Watin tests a little more data-driven (chunk multiple actions together into a procedure for common tasks, that can be called with different data) there is a Controller Framework that can be wrapped around it.

Then for Acceptance Testing (UA as you mention) I very much like to wrap Fitnesse around the above mentioned. Fitnesse will allow you to eventually delegate some of the functional stuff (use cases) to non-developers. These tests are like tables with a column for all possible input into a test, and a final column with the expected result value. For example a column for each field in a form plus a validation value. You set it up so that a Fitnesse test calls one of the Watin-controller tests with parameters for each field, and return a checksum or value from the watin-framework which can be validated by Fitnesse (which may be a simple boolean for valid/invalid, or be a calculated value, depending on what you're testing).

The combination of these 3 projects works very well (but I'll advise you to familiarize yourself with Fitnesse before writing the other stuff, because the implementation is a little different that you'd expect: Fitnesse calls methods for each input parameter for example so yo'll have to keep that in mind when writing the Watin-framework-wrapper layer).

Add TeamCity to the collection and you'll have speedy test-results after check-ins, while the code is still fresh in your mind. (this is the only commercial package in this list, but it's free for up to 20 projects).

It's quite a mix of Java (Fitnesse & TeamCity) and .Net (Watin), and it takes some time to figure everything out, but it saves tons of development time once you get the hang. (I'm a .Net developer by the way, but it never stopped me from looking over the fence and putting anything else to work when it saves time).

Answer 2

Keeping separate branches is messy and creates a lot of overhead.

A good way to deal with this problem is using experiments (Optimizely is a great tool, but you can start by implementing a very simple mechanism by yourself). You deploy a version that contains all the features, but open only some of them to the regular users.