There is an old line in advertising that half the money is wasted, but nobody knows which half.
What I like about it is that it names the problem exactly. The trouble is not that advertising fails. It is that you cannot tell which part failed, so you cannot fix it. That was unsolvable for most of the time people have been repeating the line. It is solvable now.
I joined Jeremy Allen on the Big Branding Small Business podcast (opens in a new tab) to talk about how, which mostly means A/B testing and marketing technology. The whole conversation is worth a listen, but here is the argument in short, for anyone who would rather read it.
Most of what we discussed came out of enterprise work, on projects with budgets a small business will never have. That does not make it irrelevant. The tools scale down further than people expect, and the thinking behind them costs nothing at all.
Research is where you start, not where you stop
Good research tells you how a customer is likely to behave. That is genuinely valuable, and it is also the beginning of the job rather than the end of it.
The trouble is that the research was almost always done somewhere else. A different brand, a different segment, a different moment. Even when the situation looks close to identical, something about your audience or your offer or your design is not, and you cannot reliably predict which difference is the one that matters. The financial disclaimer applies here too: past results do not predict future performance.
So treat the research as a hypothesis. It says: I think my customer will behave this way. A/B testing is how you find out whether that is true.
Change one thing, or you will not learn anything
Here is the mistake I see most often. A company runs two campaigns against each other and asks which performed better. They get an answer. Then somebody asks why, and the room goes quiet.
You can read the tea leaves and construct a story, but you do not actually know, and because you do not know, you cannot apply it to the next campaign. The win does not compound. It was a coin flip you happened to call correctly. This is the old problem in new clothes: you have proof that half of it worked and no idea which half.
A valid test changes as little as possible, ideally exactly one thing. Same offer, same audience, same time of day, same everything, with one variable moved. Then the result means something, because there is only one thing it could be attributed to.
The other half of that discipline is measuring the right thing. Testing the right variable while measuring the wrong outcome gets you a number that looks like an answer and is not one.
Start from the business objective
Before any of that, decide what you are actually trying to move. Which number, and in which direction.
A common one is conversion rate on a multi-step checkout. Say your analytics and your user interviews both point at the same step, where people are dropping out. You form a hypothesis: the label on this field is unclear, so people either hesitate or enter something that gets rejected. So you write a second label and run the two against each other, splitting traffic evenly between the existing version and the new one, holding everything else constant, and letting it run until you have enough data to be confident the difference is real rather than noise.
The testing tools handle the mechanics of splitting traffic and reporting the result. What they cannot do is decide what is worth testing, which is the part that requires you to know your business.
The part people skip: being wrong is the point
This is the part I find most interesting, and the part that is hardest to sell inside an organization.
If you work in marketing or advertising, you are in the business of selling your ideas, and a great deal of your perceived value comes from having the right one. That creates a quiet incentive to be right, and an even quieter one to bury the results that say you were not.
Testing takes that away from you, which turns out to be freeing. You will be certain something is going to win, and it will lose, and you will learn more from that than from the tests that confirm what you already believed. It gets you past your own cognitive biases, which is not something you can do by thinking harder.
So the losing results belong in the front of the report, not the back. A line usually attributed to Edison has him saying he had not failed a thousand times at making a light bulb, he had successfully found a thousand ways not to make one. Every result moves you closer to the outcome you are actually after, whether or not it flattered your instincts.
Champion, challenger, and the champion you retired too early
The standard pattern is champion and challenger. Your current approach is the champion, the thing that has beaten everything put in front of it so far. A new idea comes in as the challenger. If it wins, it becomes the champion.
The refinement worth knowing is this: bring back an old champion occasionally. Something that won for a while and then lost its seat. Markets move, preferences move, technology moves, and an idea that failed three years ago may not fail today. The reverse is just as true, which is why what is working now deserves to be challenged rather than protected.
That is really the whole argument. Not being rigid about what you learned, not treating a past success as a permanent answer, and staying willing to test the thing you are most confident about. It matters more as the ground moves faster, and the ground is moving faster.
Listen to the episode
Jeremy runs No BS Branding (opens in a new tab), and the podcast is worth your time whether or not you care about testing specifically.
If any of this is something you want to talk through, I am happy to. That offer was genuine on the podcast and it is genuine here.