Evaluating and Choosing A/B Testing Services

One of the first documented A/B tests was run by Google nearly 18 years ago as a spearpoint test into the world of web performance optimization. The test ultimately glitched due to slow load times. (Nice to know that even Google isn’t infallible!)

Yet now, many years later, A/B and multivariate testing are staples of the product development lifecycle and bedrock elements to creating an iterative practice around software--Google now routinely runs greater than 10,000 A/B tests per year. Whether your company is looking to improve sales conversions, keep rapt attention of your users, or simply direct more users to a specific page ,  A/B testing should be a frequently used tool in your repertoire. But, how do you get started and what services do you use?

In 2018, we’re well beyond the laborious and somewhat archaic practices of yesteryear in which a company would simply have to create a separate version of the site and then analyze traffic with little chance of getting tangible data from it.

From these dark ages optimization testing has spawned hundreds of companies and services that are vying for your attention, your patronage, and your dollar.

Here at Metal Toad we are continuously analyzing the best services for not only ourselves, but for our clients.

So when we started to evaluate A/B testing services and companies that would best serve our clients in an agency context we looked at several different factors and numerous companies. This was a challenge as not all companies are as transparent about their platform capabilities as we’d expect.

We primarily researched the following companies:

1. Adobe Target

2. AB Tasty

3. Google Optimize

4. Mixpanel

5. SiteGainer

6. SiteSpect

7. Kameleoon

8. Freshmarketer

9. Maxymiser

10. Dynamic Yield

11. Visual Web Optimizer (VWO)

In beginning this evaluation it was important for us to identify, by looking at the options, the features that are universally standard and offered--and features that make each product unique.

The Table Stakes

Every company that we chose to look at needed to provide a handful of table stake features.

We consider the basics and non-negotiables to be: A/B testing, multivariate testing, in browser editing and reporting. We also needed standard experiment admin features, like the ability to pause, stop, start, and cancel experiments. Provided that a service offered these key elements, they came under consideration.

In addition to these basics, we evaluated the service’s appearance of ease of use, how large the community is, whether or not it can support multi-platform services, the industries of the companies that utilize the service currently, and whether or not that it was apparent that the documentation was excellent.

Almost every company made the cut for the basics--including Mixpanel, which is a mobile only testing service and has limited use-case for our purposes here at Metal Toad. We had already removed Optimizely from consideration. We have loved using Optimizely, but we took it out of the running because of the recent price changes.

The Differentiators

The nice-to-haves we were most excited about are the following features: funnel testing, having the ability to edit using code (instead of a visual editor), being able to define the specific goals and track associated metrics, and--in an ideal world--integrated heat mapping to track user behavior.

Funnel testing would enable us to better identify the user’s path through a site, so we could identify areas where drop-offs were occurring and generate hypothesis off of those findings. Though most commonly applied to e-commerce purchasing paths, we feel that there is value in analyzing funnels to for other KPIs that our clients care about - such as subscription sign-ups.

We also want to be able to be a little more “under the hood” in most cases, so though visual editors are wonderful for ease of use - being able to edit at the code level for the precise changes we are looking to make was a must.

Having enough data to work with is also an incredible boon to work with when doing A/B, multivariate, or any form of testing that these services offer. For our needs, integrations with analytics will allow us to set proper goals - and having a system that would help us define those goals using real traffic data for our client sites was a top priority.

Lastly, and most importantly, heat-mapping was at the top of our list. Heat maps allow UX, UI, and marketers to see real-life interactions of user and website over time. The aggregate data shows the paths that the users take on the site and provide awesome visual data to help inform testing decisions. With heatmap technology baked in, we concluded we could shorten our cycle time: observing user behavior, hypothesizing, and implementing an experiment.

Agency Factors

As an agency we need to consider the varying need and budgets of our clients. So within this thinking we were seeking a  “one-stop shop”: a service providing not only the basics but some (if not all) of the higher level “nice to haves” that many other services rely on integrations for.

The driver for this, in our eyes, was to minimize the amount of partnerships that would be required to cover all our bases with hypothesis testing and experimentation for our clients. Which would, in turn, reduce billing complexities.

Ultimately for Metal Toad we found our answers in Freshmarketer, which was a decision that was further solidified in by the fact that they have an agency model, allowing us, as the trusted partner of our clients, to be the account executors and hold all of our client sites in one place.

Evaluating for Yourself

Our evaluation process was unique for our needs and the needs of our clients, but your criteria may be very different. Ultimately we recommend that you take stock of the most important elements that will be necessary to drive your site to success - and we believe that most of those testing features can be found in the majority of the services out there.

The outlier features that made the difference for us at Metal Toad may not be the above-and-beyond features that you find necessary for your site - but hopefully now you have a baseline understanding of the features that set some of these services apart.