Complexity in Split Testing

What is a simple, advanced, or complex test?

What skills are required?

For the uninitiated, it is not always easy to tell the difference between a simple test and a complex test. When assessing the difficulty of carrying out a test, the operational and functional changes, extending as far as the back-end, must also be taken into consideration.

A simple test involves limited changes to the design of one or multiple pages. With an effective A/B testing tool, changes of this nature (such as changes to the text, color, product images, the size of call-to-action buttons, switching blocks, redirecting to a new page or creating banners) can be carried out by anyone with no particular technical skills.

An advanced test does not involve major functional changes, but more advanced design changes. Changes that require moderate skills in CSS and HTML include switching from a three-column layout to a two-column layout for product lists, reorganizing a navigation menu, removing fields from a form, URL rewriting, or adding a call-to-action.

Complex test often involves significant functional changes and require hands-on experience in web technologies: Changing form a page redirection to an overlay window with a pop-up, redesigning product datasheets, breaking a form down into multiple steps without a URL redirect, shortening the sales funnel, etc.

Fine-tune your user interface

Web testing is often used to confirm the hypothesis that relates to the user interface or user experience. 

In this area, there are universal rules that have been confirmed hundred of times on actual cases.

For example, the reading direction of a web page is such that navigation menus should be placed on the left of the screen, as placing them on the right would disorient your users.

A/B testing is not about challenging established web design best practices, but to refine them.

Web testing does not set out to contradict these established rules, and it would be a waste of time to test versions of pages that deliberately ran against them. 

Nonetheless, if you are looking to achieve additional gains in conversion rates of tenths of a percentage point, you need to fine-tune these best practices, because what is optimal for one website isn't necessarily so for another, even if they operate in the same market and attract similar demographics.

This uncertainty is what makes it so challenging to optimize conversion rates but it is also what makes it worthwhile. If all it took to achieve optimal conversion rates was to follow a series of established rules, there would be no more room for discussion and all websites would be identical!

Mirror the approach taken by UX experts, and drill down into the following areas that are likely to improve your user experience:

• the location, size, and design of CTAs

• the ideal number of steps and required information to fill your online forms

• the format of feedback provided to users (e.g. what should be displayed when an item is added to cart)

• General information and browsing assistance: What is the optimal length and should it be shown by default or dynamically?

• the format and size of messages designed to reassure users.

Measure everything

Before launching your A/B tests, it is important to set your goals in a specific manner that allows them to be monitored in your reporting tools, in the form of KPIs.

In e-commerce world, the traditional KPIs are the number of items added to cart, the number of purchases completed, and the mean shopping cart value. While these KPIs (which are known as primary KPIs, because of their fundamental importance) will generally be available via your reporting tool, this is not necessarily the case of KPIs that might have a less structural or strategic impact, which is referred to as "secondary KPIs."

You can't correct problems that you don't know about.

For example, do you have a precise vision of your sales conversion process, with a specific figure for the abandonment rates at each stage? Do you measure the number of support requests or the number of customer contacts with the after-sales team? Do you know the number of advertising revenues generated by each location, etc.?

Any data required to achieve an effective understanding of visitors' behavior must be measured correctly in order for you to take action on an informed basis, both from an operational perspective (ongoing detection of problems, bugs, or underperforming page elements) and from a strategic on (decisions that will have long-term effects).