New Message Notification

How to Run A/B Tests That Matter

Profile photo of Sophie Walsh Submitted by Sophie Walsh April 12, 2016


You can A/B test anything if you really want to. However, the truth is that an awful lot of tests deliver inconclusive results, or worse still don’t make much of a difference to what actually matters – the bottom line.

To illustrate that, let’s take one common example. By testing the subject line of an email, we don’t necessarily learn much. Of course we might improve the performance of that specific campaign by sending the winning variant to the remainder of our audience (although in most cases the numbers used to make these decisions are nowhere close to large enough), but beyond that we aren’t any the wiser when it comes to the next time around.

The reason clever businesses (and I am going to include Swrve customers in that group!) A/B test isn’t just to improve a single campaign. It is to enable those organizations to implement informed changes to the mobile app experience on an ongoing basis. Changes that will engage users, retain users, and ultimately – drive revenues. And do so permanently.

So, what elements of your mobile app should you test in order to deliver maximum ROI to your business? Our customer success team has worked with many of the world’s leading app businesses, so we know what works.

  • Your onboarding process, or what a user sees the first time they open the app
    • How many initial screens are most effective?
    • Within these screens – what content are users most responsive to?


  • Getting ‘free trial’ users to subscribe
    • Which user groups should be targeted?
    • What is the optimal time to deliver campaigns?
    • Which campaign creative is most effective?


  • Push notification opt-in
    • Should you request this access during onboarding, or wait until the user is familiar with the app?
    • Should you provide the user with details of what kind of push they might receive? In the example below, a customized request for push notification opt-in on the left is compared to the defauly iOS experience on the right.


Measurement is the be all and end all when it comes to giving your customer a top notch mobile experience. And only they can tell you what they want – so listen, learn and keep improving. That’s what A/B testing is all about.

And the good news is we’re here to help! To learn how to run A/B tests that give you real insight, and how to avoid common pitfalls when testing, why not tune into Alex Gray, our Director of Sales Engineering on “How to Run A/B Tests That Matter”. April 14th 3pm GMT- register here!

This post was originally published in by Sophie Walsh

Facebook Share and earn points.
Share on Google+

The following two tabs change content below.
Profile photo of Sophie Walsh

Latest posts by Sophie Walsh (see all)

You Might Also Like

Most Valuable and Active Kinggers

Profile photo of Amit SharanProfile photo of James LingProfile photo of Winford MooreProfile photo of Janice WaldProfile photo of PaulProfile photo of DaveProfile photo of Judith ChangProfile photo of Gavin McDonaldProfile photo of David KilburnProfile photo of John A KarnishProfile photo of Tom KimpProfile photo of LukeProfile photo of MarkProfile photo of MaxProfile photo of VijayProfile photo of Black


5 Responses to “How to Run A/B Tests That Matter” Leave a reply ›

Leave a Comment

You must be logged in to post a comment

CommentLuv badge

Want MASSIVE Traffic? Then check out > KINGGED.COM'S SUPER PROMOTION To Reach MILLIONS OF PEOPLE On 100+ Blogs and 1,000+ Sites Click To See.