Deconstructing A/B Test

Posted on In Views:

Deconstructing A/B Test

A practical guide to building custom A/B test frameworks without overhead

When talking about A/B test, many organizations always think of the need for “certain” commercial tools, as if those tools are the only way to implement A/B test.

In fact, in the previous article, we talked about systems thinking, and the most important aspect of any system change is a shared vision, followed by a mental model, and the specific tools are at the surface level.

Therefore, the problem with the introduction of the A/B test is the mindset, not the tools.

This article aims to deconstruct the A/B test process and make the introduction of A/B test easier and more natural, without having to rely on a huge enterprise solution.

Breaking Down the A/B Test Workflow

An A/B test experiment consists of the following four core elements.

  1. Setting up experiment conditions: Someone needs to design an experiment situation to verify the hypotheses, e.g. we want to test whether the new version of checkout process will increase the conversion rate.
  2. Implementing experiment behaviors: The application implements different workflows based on group information.
  3. Event logging: Events need to be sent out in different workflows and these events need to contain group information.
  4. Post-analysis: Based on the event logs, the analysis is done to determine whether the initial hypothesis is correct or not, and whether the initial experiment conditions are appropriate or not.

Let’s describe an A/B test interaction with a complete time sequence.

  1. Front-end receives group information from somewhere.
  2. Front-end interacts with back-end with this group information.
  3. Either front-end or back-end needs to do something based on the group information.
  4. Throughout the process, the events with the group information are sent to the event collector based on the tracking points.
  5. The event collector will store all the events for subsequent analysis.

This completes an A/B test interaction, and there will be enough information to analyze it continuously.

Identifying and Addressing Gaps

Let’s abstract this whole process to see what components are essential.

I believe front-end and back-end should both exist, so there are three key components.

  1. Rule Manager: This is the role to manage the experiments and rules, which is what the first step of the time sequence above refers to somewhere.
  2. Event Collector: This is the event collection platform, i.e., the receiver of events sent from front-end or back-end.
  3. Event Store: This is where events are stored, either as part of the event collector or as a dataset pulled down through ETL.

For an application developed over a period of time, I feel that event collector and event store should already be in place, after all, even if no A/B test needs, there will be a need for various types of event management.

But for some projects that are just starting out, these two roles may be unfamiliar. However, I have to say that for small projects, the implementation of these two roles can be very simple.

Event collector is actually an api service, and event store is a database. Of course, as the project gets bigger and there are more events, the volume of the event store is likely to become huge, but BigQuery or Redshift can solve many of the difficulties.

So the only thing that really matters is the rule manager.

What is Rule Manager?

A rule is a relatively abstract concept that describes a sequence of logical branches from which we can derive the final result from the inputs.

A rule manager, as you can tell from its name, is a role that manages a rule set and processes the inputs and produces the outputs through a rule engine.

Martin Fowler also has an excellent article on rule engine.

Let’s use a concrete example to describe what a rule manager does.

Suppose we want to design an experiment to verify the checkout behavior of adult customers in all stores that have the “A” feature. In this case, we want to set 60% of users who meet the criteria to be A, 40% to be B, and any who do not meet the criteria to be recognized as B by default.

This is an experiment scenario, and we can create a configuration file based on this experiment scenario.

1
2
3
4
5
6
- experiment: test_checkout  
  A:  
    - "a_func = true"  
    - "age >= 18"  
    - "percentage: 60%"  
  default: B

This configuration file creates three rules and relates them to the experiment test_checkout.

Accordingly, we need to create a context on the client side that contains the necessary information for the rule engine to operate on the rules.

Here is a possible context.

1
2
3
4
5
{  
  "session_id": <str>,  
  "a_func": <boolean>,  
  "age": <int>,  
}

Because we have three rules, the context needs to contain three fields, of which session_id is a special one. The other two fields are easy to understand, but what is session_id?

This usually refers to the user’s GUID, which may be user_id or device_id or some other kind of GUID, in order to avoid the possibility that users may be assigned to different groups during a sequence of operations, which would not only pollute the experimentation data, but also make for an inconsistent experience for the users.

In an experiment where we need to assign a specific percentage of users to different groups, there are two ways to accomplish this.

  1. Having a record of all the users allows us to accurately assign the ratio.
  2. We can get the relative ratio by using a consistent hashing algorithm with a mod.

It is much more difficult to realize the precise ratio than the second approach, so most of the rule engine adopts the relative ratio.

The implementation of a whole rule engine is more difficult depending on the number of supported rules, but if we keep a good code structure, I’m sure it won’t complicate the rule manager.

The following is a simple example, since it uses Chain of Responsibility pattern, it is not difficult to add more rules in the future.

Why Avoid Commercial Tools?

Why do we need to deconstruct the A/B test process? Wouldn’t it be better to just use a ready-made tool?

By deconstruct the A/B test process, we should be able to understand that A/B test is not complicated, and that the function of each component is quite simple. If there are already existing components in the organization that can be reused, even better.

The biggest problem with introducing commercial tools is integration. Those enterprise solutions are end-to-end solutions and require some rewriting of the entire application.

At the initial stage of introducing A/B test, if we are not even familiar with the mentality and methodology of A/B test, we will make drastic changes, which will only result in a greater Resistance to Change.

Moreover, at the beginning, the experiment scenario we design must not be too large and complicated, and we don’t need a very complete solution to realize it.

Take the example in this article as an example, the rules of comparison, equality, and percentage, which are commonly used in A/B tests, can be implemented in just a few lines of code.

Conclusion

I understand that those commercial tools have their strengths and specialties, and I agree that in most cases it’s important not to reinvent the wheel.

However, when we learn from systems thinking that it’s more important to be at the bottom of the iceberg model than at the top, we should look back and see if our current needs are really a wheel.

Or can we actually build a pyramid with a sledge?

From my point of view, purchasing an enterprise product involves going through layers and layers of financial reviews, and convincing departments to change existing applications one by one, which involves a huge amount of risk when the results are impossible to estimate.

Therefore, I would choose a relatively simple approach to achieve almost the same result.

Originally published on Medium