A/B testing is challenging, even when companies have dedicated resources. But the issue becomes even more complicated when trying to A/B test on your website with a headless CMS.

If set up properly, headless CMS can be extremely powerful to help you conduct A/B tests on your website and allow you to make rapid updates to content without developing new pages or implementing additional software.

This guide will walk you through how Zesty has implemented A/B testing with our own headless CMS, and how you can implement this in your own website testing process.

Ways to A/B test on your website

When we approach A/B testing, we consider there to be three ways to create your test.

• Create full landing pages with unique URLs

• Make one smart landing page that adapts to query parameters

• Use a tool like Optimizely or Google 360 that changes the JS in real-time

Most companies opt for the first or third options, which can have drawbacks. Creating full landing pages with unique URLs to test can be difficult to scale and measure if you don’t get enough traffic to each page. Utilizing a tool outside of your CMS works similarly to creating a page that adapts to query parameters, but can increase cost and potentially slow down your page.

We recommend leveraging a “smart” landing page that adapts based on query parameters. What this means is that your URL parameters will match a unique identifier that you define in the CMS, and the page content will load based on that parameter. This allows you to reduce the number of unique landing pages, increase scalability, and rapidly test without developer involvement.

Content to A/B test

There are several types of content you can A/B test on your website. We recommend considering the following elements:

• Titles & descriptions

• Media (images & video)

• Layout changes & design elements

• Component swapping

• CTA buttons

Ways to determine how your A/B test is performing

Some quantitative determinations for performance include:

• Observe Google Analytics based upon page views, time on site, bounce rate, etc.

• Conversions based upon page URL (either unique URL or with query params)

• Event tracking with GTM data layer

• Observing CTR for PPC campaigns

Qualitative insights are also important when A/B testing. These can include:

• Internal team feedback

• User feedback

• Paying test subjects for qualitative feedback / survey

Important considerations for A/B testing

• Test only one element at a time: Make sure tests are of the same element all the way across

• Create a good mixture of timeframe and data samples to have a large enough sample to observe and make decisions

• Test to make sure there are no errors in your data, including events, tracking and KPIs

How to set up A/B testing with a headless CMS

Now we will walk you through the basic principles of how you can set up an A/B test within your headless CMS.

1. Set up a webpage you can control content in

Your webpage will need to dynamically render OR be able to randomly return a certain amount of static pages with unique elements (static).

Static: for mass scale, you can use load balancers or content delivery networks (CDNs) mixed with headers and cookies to choose to load a page at random (you would build 4 different URLs)

Ex: Unique URL with cookies or headers will serve a unique page, OR multiple unique static URLs will serve a unique page - this requires a controlled source to control URL such as PPC or email (will not work organically)

Dynamic: for mass scale, you can use a system like Zesty.io or another dynamically-rendered CMS platform that builds the page based on randomness or query parameters

Ex: if you go to the same URL, it will randomly pick from 3-4 pieces of content that you define OR the defined content will change based upon the query parameter - this requires a controlled source to control URL such as PPC or email (will not work organically)

We recommend leveraging dynamic rendering.

2. Data you need to implement A/B testing

Baseline presentation layer (webpage/code) that has set elements that can accept different content data (ex: titles, descriptions, media).

A data source that can contain one or multiple items and multiple fields. This data source needs to be query-able at the time of render to either fetch information randomly OR based upon the identifier

For example, a database that has columns, title, description, and identifier

• You would set up 3 different title and description concepts with 3 different identifiers that would be passed in a query parameter

• Identifier examples: UTM_campaign, a numeric ID, random unique text such as a #, or direct labeling such as “campaign 2” or “test 3”

Set up A/B testing with Zesty

1. Design your page and content

2. Access Schema and create a new single-page model

• Add fields to match the content you want to edit on your page (title, description, images, etc.)

3. Navigate to Content and enter your content for the single-page model

• This will be the baseline “control group” page

4. Code: Have your web developer implement the initial page design to your single-page model

5. Return to Schema and create a headless data model (example name: “Landing Page A/B test”)

• Add fields that you want to be dynamic for your A/B test (title, description)

• Make sure to include an identifier field

6. Return to Content and enter a couple of test concepts for your headless data model “Landing Page A/B test”

• Make sure that the title, description, and identifier are unique for each new content item

7. Have your developer write logic into the template to load dynamic content 

• This logic will detect the query parameter that includes your identifier

• That identifier should be used to look up the relative A/B content from “Landing page A/B test”

  • If there is no identifier in the query parameter, default back to the “control group” of content

  • Identifiers should be stored so that an event trigger (such as GTM data layer trigger or another marketing technology) will send data back to your tracking tool (such as sending to your CRM, analytics platform, ABM platform, etc.) - this is critical to measure and score your tests

Now that you have all the pieces set up for your A/B test, it's time to test and track your campaign's success.

Measuring the success of your A/B campaign

Vital to the success of your campaign is setting up proper instrumentation and tracking. Your developer will need to set this up in code. The following items should be considered:

• Your analytics - track pageviews & clicks through Google Analytics, or other

• Form submissions - how do you want to track these? Are there specific parameters you need to set up with event tracking?

  • For Zesty, this is tracked by the UTM parameters related to UTM_source & UTM_medium

• Run manual/automated tests to make sure data is being fed into your analytics software/CRM

• Check that pageviews & event tracking data are being captured

• Check that the query parameters with identifiers are loading the appropriate A/B content

Marketing: Configuring your campaigns

Create ads related to the A/B test content that use specific URL identifiers

Example: https://www.zesty.io/?UTM_source=advertisement&UTM_medium=linkedin&UTM_campaign=myABtest1 

myABtest1 is your unique identifier

Marketing: measuring your campaigns

Determine KPIs such as conversion ratio, CTR, bounce rate, time on page

• Most importantly: conversion ratio = % of pageviews that turn into conversions = conversions/pageviews

• If you’re not tracking conversions, you have to use time on site or bounce rate as KPI

And last but not least: continuously iterate :)

• Add new content examples

• We recommend testing 3-5 at a time

  • Set a boundary that is either based on time or data set (ex: 1,000 pageviews) that indicates when you need to analyze your data

• After you hit this target, you will drop underperforming pages and slate in new tests

• Measure against industry averages (ex: conversion ratio)

And that's it! A/B testing can be extremely powerful if set up correctly (plus you don't need a bunch of separate tools to get the job done). Want to try it out for yourself? Start a free Zesty instance today.