Dear MDT: I built a customer journey campaign. How the hell do I know if this thing is actually working?
There’s really only one way you can say with any certainty whether or not your automated customer
journey campaign is ‘working’, and it requires two steps:
- Setting up a randomized hold-out group, and
- Running statistical significance calculations
If you’re a marketer who is accountable for analyzing and communicating campaign performance, you need to either master these concepts, or get someone to help you. Because this is the only way you can say with any certainty (…or, confidence) whether or not your investment is yielding positive returns.
Setting up a randomized hold-out group
Hold-out groups are completely randomized, segmented data sets where some marketing contacts
receive your campaign and others don’t. We usually see a 50/50 split employed, meaning half the
records receive the live campaign and half receive nothing, but it really can be split by any ratio you
wish. All that matters is that the data gets segmented in a randomized way.
Running statistical significance calculations
Statistical significance calculations essentially tell you whether or not you’ve processed enough records, comparing your hold-out group to you live campaign, to say with statistical confidence that your campaign is actually working. The best part about this is, you don’t need to crunch the #s yourself. There are a lot of good statistical significance calculators online.
A simple one we use is: https://www.surveymonkey.com/mp/ab-testing-significance-calculator/
So you’ll end up plugging in 2 #s from each data set from your test:
- # of records processed
- # of conversions
From there, the calculator will tell you the percentage improvement achieved compared to your benchmark conversion rate, and its level of confidence in that improvement.
Confidence level is expressed as a percentage, and usually set at 90, 95, or 99 percent, with 99% representing the highest level of confidence.
So how the hell do you know if your campaign is actually working? Setup a hold-out group and crunch the results in a statistical significance calculator.
But if you feel like that’s a bit outside your wheelhouse, we’d be happy to help. We’ve been reporting ROI on customer journey campaigns for over 15 years, so…we’ve had a little practice. And we know how effective these campaigns should be. We’d love to share that knowledge with you, so…let’s chat. Schedule your discovery call here.
