How to Run A/B Tests That Matter



You can A/B test anything if you really want to. However, the truth is that an awful lot of tests deliver inconclusive results, or worse still don’t make much of a difference to what actually matters – the bottom line.

To illustrate that, let’s take one common example. By testing the subject line of an email, we don’t necessarily learn much. Of course we might improve the performance of that specific campaign by sending the winning variant to the remainder of our audience (although in most cases the numbers used to make these decisions are nowhere close to large enough), but beyond that we aren’t any the wiser when it comes to the next time around.

The reason clever businesses (and I am going to include Swrve customers in that group!) A/B test isn’t just to improve a single campaign. It is to enable those organizations to implement informed changes to the mobile app experience on an ongoing basis. Changes that will engage users, retain users, and ultimately – drive revenues. And do so permanently.

So, what elements of your mobile app should you test in order to deliver maximum ROI to your business? Our customer success team has worked with many of the world’s leading app businesses, so we know what works.

  • Your onboarding process, or what a user sees the first time they open the app
    • How many initial screens are most effective?
    • Within these screens – what content are users most responsive to?


  • Getting ‘free trial’ users to subscribe
    • Which user groups should be targeted?
    • What is the optimal time to deliver campaigns?
    • Which campaign creative is most effective?


  • Push notification opt-in
    • Should you request this access during onboarding, or wait until the user is familiar with the app?
    • Should you provide the user with details of what kind of push they might receive? In the example below, a customized request for push notification opt-in on the left is compared to the defauly iOS experience on the right.


Measurement is the be all and end all when it comes to giving your customer a top notch mobile experience. And only they can tell you what they want – so listen, learn and keep improving. That’s what A/B testing is all about.

And the good news is we’re here to help! To learn how to run A/B tests that give you real insight, and how to avoid common pitfalls when testing, why not tune into Alex Gray, our Director of Sales Engineering on “How to Run A/B Tests That Matter”. April 14th 3pm GMT- register here!


  1. Hey Sophie,

    I agree, what’s the worth of an A/B test if it doesn’t give you any data that you can use (to make more money) in the future?

    You gave some interesting elements to test for mobile apps. I’ve never thought about it that way, because I haven’t really worked with app entrepreneurs and I don’t have one myself.

    There are some things I’d definitely test if I had an app, though:

    -The offer
    -The call to action
    -Price point
    -Premium versions
    -Pictures and design
    -free trials

    Thanks for sharing your insights with us, Sophie. I appreciate it.

    – Jasper
    Jasper Oldersom recently posted…Do You Leverage These Swiping Secrets To Make Your Web Copy More Effective?My Profile

  2. Hey Sophie,
    A/B test run rightly is going to help the marketer make informed decision.

    However, where the test doesn’t yield the much needed result there would be no point using the result to make informed decision.

    A/B tests that matter are those targeting points that would bring about change in the campaign.

    Therefore, the skill of identifying which points of a product or process to be tested is more important that the test itself.

    If you don’t know where to test then testing just anywhere or place would be shear waste of time.

    I agree with the views of @opportunitybuilding(Jasper Oldersom) above!
    Sunday William recently posted…What Steps Do You Take To Keep Freelance Writers Honest?My Profile

  3. Hi Sophie,
    Thanks for this post. Learning how to run A/B test is sure very vital for online success. Many successful companies have made their impacts based on successful testing.

    However, there are pitfalls associated with running A/B tests for apps and it would be wise if organizations ensure that the tests are run free of these pitfalls.

    It would be wise to tune in to Alex Gray’s “How to Run A/B Tests That Matter” for more insights!

  4. Hi, Sophie. Great post.

    A/B testing is absolutely crucial to success in online marketing.

    I don’t have much experience with mobile apps, but I do have some with CPA offers. The same principles apply.

    There’s actually a saying in the CPA world: The best campaigns are discovered, they’re made.

    Relentless testing and improvement is crucial for any successful endeavor, even outside of marketing.

    There’s a reason Thomas Edison tried 10,000 different designs before getting the light bulb correct.
    Jeff Miller recently posted…White Hat vs. Black Hat SEO – The Great DebateMy Profile