Overview

A/B Testing

Experimenting with variants: collect reliable information on real users’ behaviour with relatively little resources.

Go to Guidelines

Image by Storyset from Freepik

Method Overview

Reading Links

Evaluative method

Suggested Time

One week per feature

Required Expertise

Medium
Statistical expertise

Materials

A launched product or a prototype

Participants

Developers, product manager, tester

Nice to have
These methods might be important for some innovations only.

Essential Reading

What

A/B tests consist of a randomized experiment that usually involves two variants (A and B), although the concept can be also extended to multiple variants of the same variable.

It is a relatively quick real-time test for product ideas. Via A/B testing you can settle a design team conflict, take informed, user-focused decisions, get mostly quantitative data about your designs, and improve user experience quickly.

When

You are in the middle of designing your product, and two rivaling ideas seem equally attractive – for example, whether to use text or an image to convey an information, or two different styles of text. You have done some prior research and observation, that led to a hypothesis about how to fix a problem or improve an aspect of your product that you would like to test.

Who

You will need the tech team on board to publish two versions of your prototype and collect the necessary information on how your users reacted. You’ll also need someone with the statistical knowledge to analyze any quantitative results and understand what they mean. Qualitative A/B tests should be conducted by experienced UX researchers in order to gather unbiased information and insights.

How much time

Depending on the question and A/B testing variant, the test might have to run for several weeks before you have sufficient data to analyze or it might be done in a few hours.