The importance of A/B Testing in Design
A/B testing, also known as web testing, is a technique that consists of comparing a web page with one or more variations and retaining the most effective one in production.
A/B testing is now sufficiently well-established that it forms an essential part of the online marketer's toolkit but, as is often the case, it was born of necessity: As competitive pressure continues to increase, so do the costs of acquiring new customers, meaning that optimizing customer conversion rates became a must for all businesses, with the goal of acquiring visitors who will ultimately be converted into customers.
More than ever, the challenge is to "do more with less." In other words, to improve conversion rates at constant traffic levels, rather than increasing traffic indefinitely without addressing conversion rates.
In this article, I will use the terms A/B testing and web testing interchangeably to describe the practice of creating A/B tests, split tests, and multivariate tests. For a definition of these terms, please refer to this glossary.
Nonetheless, A/B testing is more than just a stopgap to deal with the effects of constantly increasing customer acquisition costs.
We note four major advantages for the use of A/B testing:
Promoting a data-driven culture
When choosing between different versions of a website, there is an abundant history of counter-intuitive examples, not to mention how many decisions have been taken based on criteria that were not relevant to the initial objectives.
A/B testing overcomes these challenges by providing objective results generated by the behavior of your visitors. By relying on statistically proven facts, A/B testing is a powerful decision-making tool, provided that the sample is large enough and the test continues for long enough.
The promise of A/B testing is sometimes reduced to the ability to increase conversion rates, but this is an oversimplification. In reality, web testing can also be used to avoid losing customers and to prevent counter-productive decisions.
Shortening your decision cycles
E-commerce has resulted in significant reductions in "time to market", or the time that is taken by organizations to adapt to changes in the markets in which they operate. More than in other sectors, purchasing behaviors develop quickly, while competition increases. Decision cycles need to be extremely short for organizations to remain viable.
Precisely because it is agile and data-driven, A/B testing allows the decision to be taken "on the ground" - or at least closer to it. Whether the goal is to eliminate sticking points that can damage customer engagement and conversion rates or to test structural changes (such as a new version of a website), A/B testing is a perfect fit with decision-making processes that suit the changing needs of today's organizations.
Learning more about your visitors
A/B testing is not the only way to get to know your visitors better. Other techniques, such as user panels, focus groups, and eye tracking, can also provide the kind of information. A/B testing is the only tool that allows you to engage with your visitors under real circumstances. You will collect objective data about what your visitors really want, both consciously and unconsciously, without your visitors even knowing that you've been analyzing them in the first place.
Testing, therefore, provides invaluable behavioral information about your visitors.
Adopting an intensely ROI-focused approach
The investment required to carry out web testing remains trivial compared to the observed gains: Compared with traffic generation (using SEO for example) that requires a monthly budget, the same budget invested in A/B testing will have a long-term impact and can be believed to have generated asset value (at least until the next major redesign of your website).
Even better, our experience shows that increased conversion rates from a winning variation generate both short and long-term effects. Online merchants who adopted A/B testing notice increased customer engagement and loyalty, leading to higher conversion rates over time.
It is worth mentioning that, as far as A/B testing is concerned, while winning variations can have positive effects, the impact of a losing variation is minimal. Indeed, it can be identified very quickly and prevented by reverting to the original. The impact is reduced even further if only a small proportion of your traffic was directed to the unsuccessful variation.
The limitation of A/B testing
Although A/B testing is an essential tool, it is not a complete, end-to-end solution in terms of developing and implementing a customer conversion policy, and it is certainly not a substitute for a marketing strategy.
A/B testing can confirm a hypothesis, but can't generate one
Web testing requires scenarios and test hypotheses to be developed in advance. Test hypotheses might be closely related to best practices for the web or a particular sector, but once the "quick wins" have been implemented, understanding what needs to be tested becomes less and less obvious.
A comprehensive conversion optimization policy often requires an analysis of the user experience and/or your data beforehand. This internal analysis (or external audits) will identify the under-optimized areas of your website and will, therefore, feed into a "testing backlog", consisting of a range of hypotheses to be tested for their ability to generate enhanced conversion rates.
A/B testing shows what's better, not what's good
Web testing allows you to carry out tests, sometimes very inexpensively, to generate improvements in conversion rates of a few percentage points (or fractions of a point).
Although we strongly recommend the practice of continuous improvement of conversion rates, it cannot be justified in isolation. Also, it requires a critical review of your website. Customers' needs and behaviors are in perpetual evolution, and improvements in conversion rates cannot compensate for a poorly designed website.
The situation is similar to that in a brick and mortar store, where daily cleaning and maintenance tasks don't offset the eventual need to undertake a major renovation to address changing customer expectations.
A/B testing gives results but doesn't tell you why
A/B testing gives extremely satisfying outcomes when a new hypothesis demonstrates enhanced conversion rates.
However, it naturally leaves marketers frustrated, because part of the marketer's role is to understand why. Therefore, while the involvement of UX experts or web analytics is useful upstream, their implication is just as important downstream. The results will feed into their own findings, creating a virtuous cycle that leads to the identification of an increasingly relevant hypothesis.
A/B testing maximizes conversion rates, not the brand image
Although web testing is a must among the panoply of management tools available to e-commerce websites, in practice it must also be integrated with respect to the firm's marketing strategy.
It is possible to generate percentage-point improvements in conversion rate in the short-term, which will harm the business' image over the long-term. The impact of this will vary depending on the brand, but we recommend that our clients always bear this risk in mind, because of the risk of harming the credibility of the practice as a whole.
In conclusion, AB testing is a tool that should be part of your software development cycle, to be deployed in harmony with your other marketing and development efforts.