New Message Notification

How to Run A/B Tests That Matter

Profile photo of Sophie Walsh Submitted by Sophie Walsh April 12, 2016


You can A/B test anything if you really want to. However, the truth is that an awful lot of tests deliver inconclusive results, or worse still don’t make much of a difference to what actually matters – the bottom line.

To illustrate that, let’s take one common example. By testing the subject line of an email, we don’t necessarily learn much. Of course we might improve the performance of that specific campaign by sending the winning variant to the remainder of our audience (although in most cases the numbers used to make these decisions are nowhere close to large enough), but beyond that we aren’t any the wiser when it comes to the next time around.

The reason clever businesses (and I am going to include Swrve customers in that group!) A/B test isn’t just to improve a single campaign. It is to enable those organizations to implement informed changes to the mobile app experience on an ongoing basis. Changes that will engage users, retain users, and ultimately – drive revenues. And do so permanently.

So, what elements of your mobile app should you test in order to deliver maximum ROI to your business? Our customer success team has worked with many of the world’s leading app businesses, so we know what works.

  • Your onboarding process, or what a user sees the first time they open the app
    • How many initial screens are most effective?
    • Within these screens – what content are users most responsive to?


  • Getting ‘free trial’ users to subscribe
    • Which user groups should be targeted?
    • What is the optimal time to deliver campaigns?
    • Which campaign creative is most effective?


  • Push notification opt-in
    • Should you request this access during onboarding, or wait until the user is familiar with the app?
    • Should you provide the user with details of what kind of push they might receive? In the example below, a customized request for push notification opt-in on the left is compared to the defauly iOS experience on the right.


Measurement is the be all and end all when it comes to giving your customer a top notch mobile experience. And only they can tell you what they want – so listen, learn and keep improving. That’s what A/B testing is all about.

And the good news is we’re here to help! To learn how to run A/B tests that give you real insight, and how to avoid common pitfalls when testing, why not tune into Alex Gray, our Director of Sales Engineering on “How to Run A/B Tests That Matter”. April 14th 3pm GMT- register here!

This post was originally published in by Sophie Walsh

Facebook Share and earn points.
Share on Google+

The following two tabs change content below.
Profile photo of Sophie Walsh

Latest posts by Sophie Walsh (see all)

You Might Also Like

Most Valuable and Active Kinggers

Profile photo of Sherman SmithProfile photo of Amit SharanProfile photo of James LingProfile photo of Winford MooreProfile photo of Janice WaldProfile photo of CarlProfile photo of PaulProfile photo of DaveProfile photo of Judith ChangProfile photo of stevenProfile photo of Gavin McDonaldProfile photo of David KilburnProfile photo of Bruce P.Profile photo of John A KarnishProfile photo of Marsha Ingrao


5 Responses to “How to Run A/B Tests That Matter” Leave a reply ›

  • Profile photo of Jasper Oldersom Profile

    Hey Sophie,

    I agree, what’s the worth of an A/B test if it doesn’t give you any data that you can use (to make more money) in the future?

    You gave some interesting elements to test for mobile apps. I’ve never thought about it that way, because I haven’t really worked with app entrepreneurs and I don’t have one myself.

    There are some things I’d definitely test if I had an app, though:

    -The offer
    -The call to action
    -Price point
    -Premium versions
    -Pictures and design
    -free trials

    Thanks for sharing your insights with us, Sophie. I appreciate it.

    – Jasper
    Jasper Oldersom recently posted…Do You Leverage These Swiping Secrets To Make Your Web Copy More Effective?My Profile

  • Profile photo of Sunday William Profile

    Hey Sophie,
    A/B test run rightly is going to help the marketer make informed decision.

    However, where the test doesn’t yield the much needed result there would be no point using the result to make informed decision.

    A/B tests that matter are those targeting points that would bring about change in the campaign.

    Therefore, the skill of identifying which points of a product or process to be tested is more important that the test itself.

    If you don’t know where to test then testing just anywhere or place would be shear waste of time.

    I agree with the views of @opportunitybuilding(Jasper Oldersom) above!
    Sunday William recently posted…What Steps Do You Take To Keep Freelance Writers Honest?My Profile

  • Profile photo of Celine Profile

    Hi Sophie,
    Thanks for this post. Learning how to run A/B test is sure very vital for online success. Many successful companies have made their impacts based on successful testing.

    However, there are pitfalls associated with running A/B tests for apps and it would be wise if organizations ensure that the tests are run free of these pitfalls.

    It would be wise to tune in to Alex Gray’s “How to Run A/B Tests That Matter” for more insights!

  • Profile photo of Sarah John Profile

    Well, running an A/B Test doesn’t have to be complicated. Once the test objects are in place it becomes easy to implement.

    For the test that matters, the best processes should be applied to the right points so that best results are obtained.

    Thanks for sharing this awesome details

  • Profile photo of Jeff Miller Profile

    Hi, Sophie. Great post.

    A/B testing is absolutely crucial to success in online marketing.

    I don’t have much experience with mobile apps, but I do have some with CPA offers. The same principles apply.

    There’s actually a saying in the CPA world: The best campaigns are discovered, they’re made.

    Relentless testing and improvement is crucial for any successful endeavor, even outside of marketing.

    There’s a reason Thomas Edison tried 10,000 different designs before getting the light bulb correct.
    Jeff Miller recently posted…White Hat vs. Black Hat SEO – The Great DebateMy Profile

Leave a Comment

You must be logged in to post a comment

CommentLuv badge

FINALLY... after 3 years of starting this site, KINGGED.COM'S SUPER PROMOTION To Reach MILLIONS MORE PEOPLE Is now available. Click To See.