top of page

AB Testing: A Marketers Greatest Weapon?

Sept 6, 2023 | Article

By Andrew Penner




AB testing is amazing and all marketers should swear by it. This is AB testing: We need to run experiments to learn what marketing message customers will respond to best. Even if we have a message that is working, we should always be trying to improve it, tweak it, run it against other messages Often we think we're running good experiments, but they're set up in such a way so as to  . Let's not do that. Let's run AB tests like scientists do in air-locked laboratories.


B testing is a process of learning. When you create an ad for a customer,

ad for Colgate toothpaste you may choose to focus on how it will make a customer's breath smell amazing or how it will prevent them from getting painful cavities or gum disease. 


you are always, on some level, guessing that they will like it. If you make an

You may focus on how it will prevent that grody feeling on the surface of their teeth. You may focus on all three or just two. You may focus on how nice-smelling breath will help them start and maintain an intimate relationship or improve their chances of having a successful interview or help them make some cool new friends or just simply provide them with a nice tingly, icy feeling that accompanies the bacteria-killing ingredients of toothpaste.

All of those focuses could be effective. But at the end of the day you need to make a guess. Let’s suppose that you guess that the best idea is to focus on how using Colgate toothpaste will prevent cavities. You build a whole campaign around it. You run ads about how great it is to use Colgate to prevent cavities and the campaign is a smashing success. 


Let’s imagine that while the campaign is running, Colgate experiences more toothpaste sales than they’ve ever experienced. And let’s further suppose that once the campaign is over Colgate decides to focus on how their toothpaste prevents gum disease and sales during that time are not as good as they were during the time when the cavity campaign was running. 


Does this necessarily mean that focusing on cavities is better than focusing on gum disease. No. It’s doesn’t. But will people with this information conclude that cavity prevention is a better focus than gum disease? I don’t know, but there will be a strong psychological urge to believe that marketing cavity prevention is the way to go. This urge to believe that we can determine a cause and effect relationship regarding our marketing efforts is ever present and it is often ever damaging to our pursuit of marketing truth. 


People are obsessed with coming up with a cause, an explanation, a reason, an attempt to explain why things happen. When it seems so obvious that the cavity campaign out performed the gum disease campaign it may feel reasonable to conclude that focusing on cavities is better than focusing on gum disease. It can even seem irresponsible not to make such a conclusion. But making any conclusion in this scenario is a mistake. A terrible mistake plaguing marketers across the world. 


Here’s why: There are other factors that may have influenced the market’s reception to a focus on cavities and there are yet other factors that may have influenced the market’s reception to a focus on gum disease. The time of year, public dialogue, competitors focuses, the actors chosen for the ads, a study from oral researchers that had an interesting enough story to motivate morning shows to discuss it over and over again, a tag line that rubbed people the right way, a tag line that rubbed people the wrong way, a popular movie with a likable actor that looked like the lead actor in the cavity ad but not the gum disease ad, the music at the time and the music chosen in the ads, differences in the ease of purchase at local department stores, differences in the ease of purchase at online department stores, the colors used in the ads and the colors in season that year, and an infinite number of associations that people have with cavities and gum disease - always changing over time.


It’s fun to say that we know why something happened. But typically, we are dumb. And this is where A/B testing comes in. This is the golden arrow, the magical trick, the real thing, the way that works, the true path forward, the principle to rely on, the method to outsmart the madness, the system that scientists swear by. Have you ever heard of a double blind randomized controlled trial? Yeah, sure, of course. We all know that scientists take outrageously particular precautions in their experiments to avoid coming to false conclusions from their experiments. 


Why? It’s not just because the conclusions are so important. It’s also because they know from past experiments that one thing can seem to so obviously cause another thing but the two things are unrelated and the true cause was something else. Then later some other scientist makes a discovery, showing the original scientist to be wrong and making them look stupid. In marketing toothpaste, we should be at least as outrageous in the meticulousness of our experiments. 


Thus I introduce the AB test. A good AB. A great AB. A true AB. An AB test worth it’s weight in marketing dollars is one that desperately tries to account for every possible contributing factor. What does that look like in practice? It looks like this: Both campaigns must run at exactly the same time. You can’t always feasibly do this. I’m not suggesting that marketers do this for every campaign. Only when the marketer is trying to learn something with confidence.


Let’s imagine a head to head between cavities and gum disease. Which one will sell more toothpaste. The message can be communicated in all sorts of different ways. On a billboard, on a tv commercial, on a flyer, by an Tiktok influencer, by a dentist, and so forth and so on and so off.


We need to pick one delivery method. Let’s choose a TV ad. Two ads must be made. They must be identical in every possible way. Cast, location, length, music, props, everything. Everything! The tone of the script. Is it funny? Is it dramatic? Is it heartwarming? Is it like one of those ads for Febreeze where everyone is always smiling in a living room with almost-too-perfect lighting from every angle. Or is it like a Pantene Pro-V commercial where the camera swoops in so close that magical magnifying bubbles show the chemicals and hair follicles like cute tiny colorful objects dancing together in harmony. Or is it like a Gilette commercial where relatable people look in the mirror in a way that says “I’ve done this before and I’m pretty darn good at it”. Or is it a doctor in a white lab coat giving a smiling patient a reassuring nod while handing them a recommended product to cure some new ailment, then casually panning to that patient living their life at the park and the beach and some local community event where they non-begrudgingly get roped in to throwing a bean bag into a small wooden hole just far enough to make it challenging but not frustrating? 


Whatever it is, it needs to be the same for cavities and for gum disease. A likable, well-groomed actor says “This toothpaste has been designed from the ground up to be the greatest cavity preventer the world has ever known.” That’s A. Here’s B: The same likable, well-groomed actor says, in the same way, in the same setting “This toothpaste has been designed from the ground up to be the greatest gum disease preventer the world has ever known.” I'm not going to write the rest of the ad right now. You get the idea. 


Ok now we have two ads. They are as identical as we can make them. What next? We show them on the same platform at the same time to the same people. Perhaps the first ad slot during the first ad break on Hulu’s The Bear on Monday night between 8:00PM and 10PM EST. Or as a Youtube pre-roll on Dave Ramsey’s Fundamentals of Cooking on Saturday morning between 9:00AM and 12:00PM EST. It doesn’t have to be Hulu or Youtube or a cooking show (although a cooking show does seem like a nice place to run a toothpaste ad - AmIright?). It just needs to be the same. 


Now how do we track the performance? Just give people a coupon code for next 12 hours and put the word GUM in the gum disease ad and put the word CAVITY in the cavity ad. Then tell them to buy it online. Then track those coupons. If there is a difference, the reason for the difference is quite likely the thing we think it is. Could there be other contributing factors that we are not accounting for. Of course. We can never understand our customer's behavior with perfection. But this is the type of learning that we can take some confidence in. This is the type of learning that should inform future marketing campaigns. 


Are there additional ways in which we can make our test even more, so to speak, double blind controlled, and randomized? Yes and we should never stop being a little suspicious of factors lurching in the shadows.


Do you need to use a coupon code? No. In fact coupon codes are kind of dumb in my opinion, but they are always there if we need them for this type of test. If the ad is on an interactive web based device then we can simply include a different link in each ad which takes the customer to a landing page that looks identical but has a different url. A url that we can track. We can even do this on Hulu if the customer is watching on a phone or computer.


And there we have it. The basics of AB testing. This is where the learning happens. But only if we make as many things as identical as possible. Only if we show the ads at the same time. Literally the exact same time. Not a different time of the year or even a different time of day. Or even a different time of the evening. The exact same window of time. Why do I keep hammering the importance of showing both ads at the same time? Because running things at the same time is not convenient. Let this inconvenience slow down other marketers. Let it be your advantage. If we don’t commit to this outrageous level of identicalness, we’re just guessing with a fool's confidence. Let’s make our confidence justified. Let’s do more AB tests worthy of a peer reviewed journal. Let’s sell more toothpaste.

bottom of page