The Best Ways to A/B Test Copy

Running A/B tests is something we’ve touched on quite a bit in our past articles - from Reengagement Content to Making Push Notifications Standout, it’s an important part of a game CRM’s day to day and a crucial step to crafting strategy and content that works for your mobile game. 

On Teak, you can test different imagery, different headlines, different amounts of rewards, and even different rich media in your Push Notifications and Email. Today, we’re looking at A/B testing copy, specifically. 

Understanding A/B Testing & Copy

A/B Testing, as you probably already know, is a feature that allows you to quickly create multiple versions of the same message with different variables (such as image or copy) and then evenly split those variations across an audience. This allows you to see which notification performs better with that audience, and thus, which variation. 

“Copy,” of course, means anything from the headline to body text. Anytime you’re writing something for content that a player will see, it’s testable copy. Additionally, you can A/B test on any platform that Teak supports - that means you can A/B test on all push notifications, but it also means you can A/B test on email.

Top Tips for A/B Testing Copy

Now, time to get into some of our recommendations when it comes to the ins and outs of A/B testing copy.

Here at Teak, these are some of the things we see really make an effective test:

• Limit Your Copy Variants
  • When you create Notification A and then drop it down to B, it will copy everything over and it’s up to you to determine what variable to change for testing. While you can technically add as many variants as you like, in practice, sticking to only one variant for each test gives you clearer insights.
  • So that means when it comes to testing copy, focus on changing one word in the header, or the phrasing of a sentence in the body. 
  • For example, you could test how you call out the reward attached to the push - are you explicit about the number of coins or keep it vague?
  • The less things you are varying between tests, the easier it will be to see what performs well and what resonates with your audience. Adding too much variance muddles the test and makes it difficult to tell what’s affecting the increase or decrease in click throughs.
• Different Copy Tests for Different Player Lifecycles
  • Each phase of the Player Lifecycle is going to resonate with very different copy - your high value players might want to see more substance in your pushes, such as updates to the game or storylines. However risk or lapsed players may be more enticed by copy directly calling out free coins, rewards, etc. 
• Try Advertising Rewards vs Not
  • Depending on your game, it could be interesting to see if including the reward amount in your copy for your push vs not makes a difference in click through rate. There are times when burying the lead a bit and creating some mystery around what the player will get can actually work to your benefit, especially when you’re running special events or holiday initiatives. 
• In General, Don’t Get Too Wordy
  • It can be tempting when you’re a CRM to want to get involved with the language in copy and take your time crafting every little part of a push notification. However, in general, we see that simple quick to the point wording always works best. Players are met with so many notifications in a day; keeping things simple and easy for them to digest will almost always work to your advantage.

iOS vs Android in A/B Test Results 🍎🤖

Push notifications on iOS and Android will have very different results, and it’s important to not compare them to each other, but instead compare them by their respective baselines. At Teak we see that Android has a 70% higher overall click through rate than iOS.

This has less to do with the types of players who might use iOS vs. Android and more to do with the differences in how notifications work and display. Android allows for a bit more freedom when it comes to rich media pushes, for example (more on that in our Rich Media Pushes article here). And iOS notifications are easier to swipe away and then not get back once they’re gone. 

This means, it’s important to ensure you are looking at the results for iOS and Android separately. If you’re using Teak, this platform split is done for you automatically, because A/B tests are set up on a per platform basis. If you’re not using Teak, you’ll want to make sure the platform you’re using splits iOS and Android results. Or, ensure that you can create separate audiences to differentiate the two.

Teak Tip: To find the results of your tests, go to the Notifications page and click on the tab for a given platform. Use the filter menu and select “Running A/B Tests,” to quickly find your active tests.

Should we do an A/B testing series? Any other types of A/B testing you are curious about? We’d love to hear from you! We’re here to help. 

Cheers!
Erica + Teak team