A/B testing is challenging, even when companies have dedicated resources. But the issue becomes even more complicated when trying to A/B test on your website with a headless CMS.
If set up properly, headless CMS can be extremely powerful to help you conduct A/B tests on your website and allow you to make rapid updates to content without developing new pages or implementing additional software.
This guide will walk you through how Zesty has implemented A/B testing with our own headless CMS, and how you can implement this in your own website testing process.
Ways to A/B test on your website
When we approach A/B testing, we consider there to be three ways to create your test.
Create full landing pages with unique URLs
Make one smart landing page that adapts to query parameters
Use a tool like Optimizely or Google 360 that changes the JS in real-time
Most companies opt for the first or third options, which can have drawbacks. Creating full landing pages with unique URLs to test can be difficult to scale and measure if you don’t get enough traffic to each page. Utilizing a tool outside of your CMS works similarly to creating a page that adapts to query parameters, but can increase cost and potentially slow down your page.
We recommend leveraging a “smart” landing page that adapts based on query parameters. What this means is that your URL parameters will match a unique identifier that you define in the CMS, and the page content will load based on that parameter. This allows you to reduce the number of unique landing pages, increase scalability, and rapidly test without developer involvement.
Content to A/B test
There are several types of content you can A/B test on your website. We recommend considering the following elements:
Titles & descriptions
Media (images & video)
Layout changes & design elements
Ways to determine how your A/B test is performing
Some quantitative determinations for performance include:
Observe Google Analytics based upon page views, time on site, bounce rate, etc.
Conversions based upon page URL (either unique URL or with query params)
Event tracking with GTM data layer
Observing CTR for PPC campaigns
Qualitative insights are also important when A/B testing. These can include:
Internal team feedback
Paying test subjects for qualitative feedback / survey
Important considerations for A/B testing
Test only one element at a time: Make sure tests are of the same element all the way across
Create a good mixture of timeframe and data samples to have a large enough sample to observe and make decisions
Test to make sure there are no errors in your data, including events, tracking and KPIs
How to set up A/B testing with a headless CMS
Now we will walk you through the basic principles of how you can set up an A/B test within your headless CMS.
- Set up a webpage you can control content in
Your webpage will need to dynamically render OR be able to randomly return a certain amount of static pages with unique elements (static).
Static: for mass scale, you can use load balancers or content delivery networks (CDNs) mixed with headers and cookies to choose to load a page at random (you would build 4 different URLs)
Ex: Unique URL with cookies or headers will serve a unique page, OR multiple unique static URLs will serve a unique page - this requires a controlled source to control URL such as PPC or email (will not work organically)
Dynamic: for mass scale, you can use a system like Zesty.io or another dynamically-rendered CMS platform that builds the page based on randomness or query parameters
Ex: if you go to the same URL, it will randomly pick from 3-4 pieces of content that you define OR the defined content will change based upon the query parameter - this requires a controlled source to control URL such as PPC or email (will not work organically)
We recommend leveraging dynamic rendering.
- Data you need to implement A/B testing
Baseline presentation layer (webpage/code) that has set elements that can accept different content data (ex: titles, descriptions, media).
A data source that can contain one or multiple items and multiple fields. This data source needs to be query-able at the time of render to either fetch information randomly OR based upon the identifier
For example, a database that has columns, title, description, and identifier
You would set up 3 different title and description concepts with 3 different identifiers that would be passed in a query parameter
Identifier examples: UTM_campaign, a numeric ID, random unique text such as a #, or direct labeling such as “campaign 2” or “test 3”
Set up A/B testing with Zesty
- Design your page and content
- Access Schema and create a new single-page model
Add fields to match the content you want to edit on your page (title, description, images, etc.)
- Navigate to Content and enter your content for the single-page model
This will be the baseline “control group” page
- Code: Have your web developer implement the initial page design to your single-page model
- Return to Schema and create a headless data model (example name: “Landing Page A/B test”)
Add fields that you want to be dynamic for your A/B test (title, description)
Make sure to include an identifier field
- Return to Content and enter a couple of test concepts for your headless data model “Landing Page A/B test”
Make sure that the title, description, and identifier are unique for each new content item
- Have your developer write logic into the template to load dynamic content
This logic will detect the query parameter that includes your identifier
That identifier should be used to look up the relative A/B content from “Landing page A/B test”
If there is no identifier in the query parameter, default back to the “control group” of content
Identifiers should be stored so that an event trigger (such as GTM data layer trigger or another marketing technology) will send data back to your tracking tool (such as sending to your CRM, analytics platform, ABM platform, etc.) - this is critical to measure and score your tests
Now that you have all the pieces set up for your A/B test, it's time to test and track your campaign's success.
Measuring the success of your A/B campaign
Vital to the success of your campaign is setting up proper instrumentation and tracking. Your developer will need to set this up in code. The following items should be considered:
Your analytics - track pageviews & clicks through Google Analytics, or other
Form submissions - how do you want to track these? Are there specific parameters you need to set up with event tracking?
For Zesty, this is tracked by the UTM parameters related to UTM_source & UTM_medium
Run manual/automated tests to make sure data is being fed into your analytics software/CRM
Check that pageviews & event tracking data are being captured
Check that the query parameters with identifiers are loading the appropriate A/B content
Marketing: Configuring your campaigns
Create ads related to the A/B test content that use specific URL identifiers
myABtest1 is your unique identifier
Marketing: measuring your campaigns
Determine KPIs such as conversion ratio, CTR, bounce rate, time on page
Most importantly: conversion ratio = % of pageviews that turn into conversions = conversions/pageviews
If you’re not tracking conversions, you have to use time on site or bounce rate as KPI
And last but not least: continuously iterate :)
Add new content examples
We recommend testing 3-5 at a time
Set a boundary that is either based on time or data set (ex: 1,000 pageviews) that indicates when you need to analyze your data
After you hit this target, you will drop underperforming pages and slate in new tests
Measure against industry averages (ex: conversion ratio)
And that's it! A/B testing can be extremely powerful if set up correctly (plus you don't need a bunch of separate tools to get the job done). Want to try it out for yourself? Start a free Zesty instance today.
By Katie Moser
Katie is the Senior Marketing Manager for Zesty.io. Her expertise lies in crafting marketing strategy, with an emphasis on content and digital marketing. Having spent too much time in clunky CMS's, Katie understands the need for a CMS that works with you - not against you. In her free time, you can find Katie in San Diego enjoying the sunshine at the beach or on hikes.