One of the most mysterious areas in split testing seems to be – how to tell when a new ad or landing page has received enough traffic to know if it’s going to work or not. The quick answer is: A/B testing for statistical significance.
Today I’m going to do my best to explain that answer and show the results of a recent experiment I did that demonstrates what I’m talking about.
“He that can have Patience, can have what he will” – Benjamin Franklin
What is A/B Split Testing?
A/B Split testing is the practice of testing two or more versions of a creative element or elements against each other. By splitting the traffic into different groups and sending each group to different pages, you can see if new ideas will work better than old ones.
I previously wrote about A/A/B progressive split testing, a method of split testing that protects you from losing too much money.
Each split test should begin with a hypothesis.
For example: If you have a current landing page that is getting a 10% CTR and you need a 12% CTR to break even on the campaign you might try testing different Call to Action Button colors to see if more people click Red buttons than Blue ones.
In this case the hypothesis is: People will click Red more than Blue.
Using a split testing software like CPV Lab, you would add each of the landing pages to your campaign and give each one 50% of the traffic. That way half your traffic goes to the old page and half goes to the new page.
What is Statistical Significance?
From Wikipedia: Statistical significance is the probability that an effect is not due to just chance alone.[1][2] It is an integral part of statistical hypothesis testing where it is used as an important value judgment. In statistics, a result is considered significant not because it is important or meaningful, but because it has been predicted as unlikely to have occurred by chance alone.
For example: After setting up your split test you may quickly find that your new landing page is getting a 25% CTR versus the old page’s 10% CTR. Then, over time, as more visitor data is gathered those results may balance out or completely reverse.
In that case the early 25% CTR was due to chance alone and was not significant.
Many online marketers make the mistake of making decisions too soon. By doing this they actually pause the better pages and slowly decrease their campaign’s performance over time.
How Long Does it Take?
How long does it take to get statistical significance? Well, that is a hard question to answer because like many things in life it changes on a case by case basis.
*Remember this: The higher the payout the longer test.
Example: If you have a campaign spending $100/day and are selling a product with a $100 payout, breaking even would mean you make 1 sale a day. It could take weeks or months to get significant data.
If however, your offer has a $1 payout, breaking even would bring you 100 sales a day so you could get significant data much faster.
A/B Testing for Statistical Significance
CPV Lab Case Study
I recently read an article that talked about increasing conversion rate by adding Urgency words after the CTA button text. The words “Now” and “Today” were recommended as being the best.
I decided to give it a try.
Day 1
Using CPV Lab I setup a 5 landing page test. I used my control CTA and added their two recommendations (now and today) as well as two others that have worked well for me in the past “online” and “instantly”.
As you can see on the initial burst of traffic the landing page using the word “Now” did double the conversions of the control page. In the past, seeing a doubling of the conversion rate may have made me pause all other pages and call it a day… but these days I know better.
I ran it through the statistical significance calculator I use and got the following results.
According to the rules of significance a test is not over until one of the variations reaches a 95% confidence or higher. At this point I was just under 80% so I knew it was too early to decide.
Day 2
After another day of traffic you can see that the landing page with the word “Now” on it was still looking really hot. It had gotten 5 more conversions than the control and looked like it was going to be a runaway winner. Each page had over 2000 views and over 350 clicks. (“Today” and “Online” were eliminated early).
At this point I thought I had found something really special and was going to really improve results.
Again, I had to hold off making a decision because even with the stellar performance of the “Now” page, I was only at a 74% confidence.
Day 5
By day 5 of the test things had completely changed. My initial leader “Now” had fallen behind and was being destroyed by the control page (the original landing page).
After another few days of testing the control hit a 95% Confidence score, beating the living hell out of all the new versions.
I would have completely screwed my campaign if I had made a decision too soon.
What Happens When You Decide Too Soon
Has this ever happened to you?
You start a new campaign. It does well for the first few days and you start optimizing. Slowly, as time goes by performance dips and you just cannot seem to get it to make money anymore.
You know what may be happening?
Your rapid optimizations may actually be leaving you with sub-par landing pages that perform worse and worse with every change and every split test you do.
Why I Hate Most Split Testing Software
There are a lot of split testing softwares on the market. One of the most popular is called Visual Website Optimizer. The reason (I believe) it’s so popular is because they have done a very good job of designing it in a way that is simple, entertaining and addictive.
But I’m not sure it’s effective.
Looking at the screenshot you can see that the interface is slick. But when you see it in action it’s a thing of beauty. It’s more like a video game than a split-testing software.
Things light up, buttons change colors, levers and bars move around and when you find a 95% winner after just a few clicks – it makes you feel like a genius.
The problem is that entertainment aside, the results (for me) are rarely accurate. After several months of playing with VWO I decided to see what would happen if I let a test go past the point where VWO said I had a winner.
90% of the time the VWO declared winner did not actually perform best over time.
Does that mean VWO doesn’t work? No way, plenty of successful marketers use it and are making a killing. But for me – the entertainment value outweighs the results so it’s not a tool I use or recommend anymore.
Tools, Reference Links and Recommendations
I use three tools to split testing landing pages and calculate statistical significance. You’ve probably heard me talk about them before and that is because they work – and I will always use them.
- Crazy Egg. Crazy Egg is a heat mapping tool that sits on your landing page and records where your visitors click. It then shows you a “heat map” of hot zones on your page. On a new campaign I put Crazy Egg on the page, let some data pile up and see where people are clicking the most. Then, I optimize that exact area first. By doing this I maximize the chances that changes I make will have a massive impact on performance. Read my Crazy Egg review.
- abtester.com This is a free Statistical Significance calculator that I use to see if I have enough data to make decisions. I enter how many views and conversions the old page got versus the new pages and it tells me if I have a winner. In theory, if you reach a 95% significance, you have a 95% chance your new page is going to work better. It’s not fool proof but is very, very helpful.
- CPV Lab This is the industry standard for tracking campaigns and split testing landing pages. It’s a couple hundred bucks and is worth every dime. All of the screenshots of my live campaigns posted on this site, Facebook and Twitter are all from CPV Lab.
These three tools are more than enough to make sure you make smart decisions that are meaningful and help ensure you’re not driving your campaigns into the dirt.
CPV Lab Coupon Code
The guys at CPV Lab have given me a coupon code for my readers who want to try it. Click here to check it out and use CPV Lab coupon code “MALAN50” to save $50.
In Conclusion
Split testing landing pages is one of the most fun parts of the affiliate marketing game. And if you can get good at it you can make just about any campaign profitable.
The most important factor in split testing is Statistical Significance. If you make decisions before the data is meaningful you can end up hurting your campaign, keeping the worst landing pages and ultimately making less money with every test you do.
The tools that I use to test for Statistical Significance are:
– Crazy Egg
– www.abtester.com
– CPV Lab (use code MALAN50 for $50 off)
Go slow, be patient and rely on longer-term results. Doing this will give you a better shot of continually increasing your ROI over time.
And that my friend, is the name of the game.
For more, watch my YouTube video: A/B Testing for Statistical Significance
Hi Malan,
One more question. I know you have done some adwords before, when you started with affiliate marketing. How can you do LP split testing in adwords?
Thanks!
Hi Malan,
Nice post in here 🙂 and good videos too.
My main problem is that I never know when a campaign will die and if it will compensate for all the money I spent testing.
What are your thoughts on that?
there are some things you will never know. so there is not much sense worrying about it.
do everything you can to create a reliable campaign, including gathering information about it before you start. like this: https://www.malandarras.com/insider-information
Hi, Malan!
Thanks for yout blog, getting much from it. Can you make a post about tracking? Especially using subids? I have such a mess about subids in my head. Where should I put them: to my affiliate link from my cpa network? or I should put subids to my link, that is given for me from my tracker (after I put my cpa affiliate network link to tracker)? I use Bevo tracker now and guys helped me to setup my links. But I still did not understand how to use them properly myself. Read a ton info, but still confused.
Thanks
i have never used Bevo tracker and don’t know anyone that uses it. I also don’t understand your question because it’s all over the place. If you want an answer, ask again in a more simple, specific way.
Sorry, my English is still bad. Can you explain how to use subids, when using a tracker (let it be CPVlab, whatever)? I know how to use subid in Adwords, but when a tracker appears in the funnel, it becomes so complicated. I dont know how they work when using a tracker.
Thanks.
Hello Malan, would you use cpv lab to track mobile campaigns? I’ve heard that it does not give as many metrics as imobitrax, which is a much more expensive tracking software. Also, what does VWO stand for?
You mentioned it in the same sentence with Unbounce.
I can’t comment on CPV Lab and mobile, because I’m not a mobile guy. I have heard that it tracks mobile, but you’ll have to look into it yourself.
VWO = Visual Website Optimizer, a multi-variate split testing service.
Thanks for your reply.
you bet man, good luck with your mobile stuff
killer post!!
thanks vasco. glad you’re here.
Love this article Malan! I really enjoy reading your blog. I get a ton of value out of it! 🙂 Delighted to see the tool abtester as a way to test out landers. I’ll be using that in future. I have a couple of queries if you dont mind answering man.
1. What do you use for testing your ads?
2. Lets say I am split testing 2 ads, both text ads… one of them gets high ctr but no conversions, but the other gets low ctr and a few conversions. Where do you go from here? I am really confused on what step to take next.
Cheers Dude!
Ps. I notice that you use cpvlab. Me too.. but I am having bad trouble with bots screwing with my data when using the adwords platform. Have you experienced the same problems? What did you do so that your data was coming in accurately. Thanks 😉
Yes, all campaigns have stuff like that. They key is to learn to not worry about the little things and focus on making the campaign profitable.
I understand that, but with the bots screwing with the data, shouldnt I get rid of as much bots as possible, so that my data is more accurate and I’ll be able to make better decisions based on accurate data. I’ve been using cpvlab for a while now, but cant seem to get it working correctly for me as the bots keep messing with traffic data and landing page ctr. Thanks Malan!
An extraordinarily wealthy and successful product owner / advertiser once gave me some advice: “Keep your focus on making as much money as possible, and don’t worry about the small change.”
Hey Malan, Thanks again for getting in touch. I understand where you are coming from. What do you do with regards to viewing accurate data on cpvlab? Do you simply ignore the campaign stats and just let the campaign run until it picks up data and then upload your reports to read the hard data? From there you optimize your campaign from viewing the hard data to see whats going on?
Hey Malan, would you mind telling me how you approach cpvlab as far as getting accurate data to make informed decisions about how to optimize a campaign? Do u do this through the reports you create as apposed to looking at the data that can be riddled with bots? Thanks Malan.
Sean, I’ve already answered this twice.
Ok.. but those answers didnt answer my question mate. I still dont know how I can use cpvlab to make my campaign profitable. I want to make my campaigns profitable and I want to focus on that. But lost in terms of how to use the tool to make my campaigns profitable. Thats what I am looking to figure out.
Hey Sean, that’s a huge question (“How to use CPV Lab to make my campaigns profitable”). I can’t answer that here. It’d take an entire blog post and depend on the niche/product/campaign. You might look into the PDFs that come with CPV Lab or contact their support for help.
Also, one last thing about the bot traffic. If you really want them to stay out of your stats you can block them at an IP level. Looked under Settings > Blocked Traffic Rules.
Hey Malan, thanks for getting in touch. I am just trying to figure out how to use cpvlab properly with the data I collect whilst using it. Instead of looking at a campaign where bots mess with the collected data, should I be uploading hard data reports from my traffic source and my conversion data from my affiliate network? Is that how you use cpvlab to optmize your campaigns?
I am adding bots to the blocked traffic rules, it just seems that its a never ending process cleaning up unwanted bots!
Hey Sean,
1. Sometimes I use this to check stats for CTR on ads http://splittester.com/
2. I optimize to ROI, so if 1 ad has a high CTR and no sales and another has a lower CTR and several sales, I’ll pursue the ad with low ctr + several sales and work on improving it’s CTR. An ad with high CTR and no sales is only going to lose $$.
Thanks for the tip on splittester, I have used that in the past also. 😉
I understand that the ad with higher ctr and no conversions needs to be paused. But how do you improve the lower ctr ad with conversions? Do you continue running the lower ctr ad, pause the higher ctr ad and start split testing another ad against the lower ctr ad, but this time, use the same hook as the ad making the sales, just using different words that might engage a better ctr? I’d love to hear how you work this Malan
Hi Malan,
Great article – thanks for sharing your findings. Do you think this would apply well to contextual advertising instead of using LPs? By the way, having a CTR of more than 10% is amazing… of the LPs I’ve ever done, I’ve yet to get anything close to that yet. Not sure if you can throw a bone my way on that 🙂
Asher
Yes, statistical significance applies to all traffic/campaign types.
And Landing Page CTRs are higher/lower depending on traffic. On Adwords PPC you may have a 15% LP CTR, the same page on PPV will get .05% – because the traffic types are very different.
On adwords ppc people are actively searching for the product, so they click. On PPV, you are just popping up on their computer whether they like it or not, so they close the window without clicking.
Great intro and over view on A/B Testing and statistical significance. Thank you Malan.
Just this past week I signed up with Unbounce — a web tool that allows you to easily create and launch landing pages. You can also do A/B testing, and they make it really easy to duplicate your control page and then make changes to the Test page…
Anyways, so I’m beginning to follow a more methodological approach to my local affiliate campaigns, the only drawback is that since I’m driving low traffic, it will take me much longer to achieve statistical significance to least 95% or more…
But it’ll be a good learning experience.
So really, it doesn’t matter how many clicks you get… As long as you achieve enough statistical significance to determine a winner.. Am I correct?
Thanks again Malan!
Hey Edgar – well the “theory” is that it doesn’t matter how many clicks you get, as long as you hit a 95% statistical significance, you have a 95% chance that it’s a winner.
But – as I said in this post – that theory is not always accurate. Many times I have let traffic continue past the first “win” and the results will completely change.
Unbounce and VWO are really fun tools. To be safe – you can do your own testing and every once and a while let a test run past the point where Unbounce declares a “winner” and see what happens for yourself.
I see. Ok, will definitely keep this in mind Malan. Thanks!
Great Article! Thank you Malan for these detailed insights!
you bet Leon, thanks for commenting and letting me know you were here. – Malan