Running holdout tests on a mid-market media budget
Platform-reported conversions keep going up while revenue stays flat. A simple geographic holdout is the cheapest way we know to find out what your media is actually doing.
Every few months a CMO shows us the same chart. Meta and Google both report record attributed revenue. The ecommerce platform shows total revenue roughly where it was. Somebody is double counting, and it is usually everybody.
The enterprise answer is a media mix model or a platform lift study with a large minimum spend. Most of our clients spend between $80,000 and $600,000 a month on paid media. That is serious money, but it is not enough to make a full mix model reliable, and the platform lift studies tend to grade their own homework.
What has worked for us is a plain geographic holdout. We have now run 23 of them for 14 clients since early 2024. Here is how we do it and what we have learned.
The basic design
Pick a channel or campaign you want to test. Split US designated market areas into two groups that look alike on historical revenue, seasonality, and share of new customers. Turn the media off in one group for four to six weeks and leave it on in the other. Compare total revenue, from your own systems, between the groups against what the pre-test relationship predicted.
That is it. No pixels, no platform reporting, no attribution model. Just your own sales data and a calendar.
Choosing the markets
This is where most of the effort goes. We pull two years of weekly revenue by DMA and build matched groups so the test and control lines tracked closely before the test. For most clients we end up holding out between 15 and 25 percent of national revenue. Smaller than that and the noise swamps the signal. Larger and finance gets nervous.
We avoid the biggest markets in the holdout group. New York, Los Angeles, and Chicago behave differently enough that they make matching harder, and losing them for six weeks costs too much.
What we have found
The results vary by client, but some patterns have held across the 23 tests.
Branded search is usually overcredited. In 9 of the 11 branded search tests we ran, turning it off in the holdout markets reduced revenue by less than a third of what the platform claimed. Most of those customers came in through organic results instead. A few clients cut branded search spend by half afterward with no measurable revenue impact.
Prospecting social is often undercredited. In 5 of 7 tests on upper-funnel Meta campaigns, the measured lift was higher than last-click attribution suggested, sometimes by a wide margin. Those campaigns were creating demand that showed up later in search and direct traffic.
Retargeting is the most variable. We have seen everything from strong lift to effectively zero. It depends heavily on frequency caps and how broadly the audience is defined.
The mistakes we made early
Our first two tests ran for three weeks. That was too short for clients with a considered purchase cycle. We now run four weeks minimum and six for anything over $150 in average order value.
We also once ran a test across a holiday weekend without adjusting the baseline. The result was noise. Now we check the promotional calendar with the client before we lock dates, and we push tests away from major sales events.
And we learned to tell finance before we start, not after. A CFO who sees a revenue dip in eight markets without warning will end the test early.
What it costs
The lost revenue in holdout markets is the main cost, and it is usually smaller than people expect because the media was doing less than reported. Analyst time runs about 30 to 40 hours per test for design, monitoring, and readout. For our retainer clients that is part of the data practice scope.
The payoff is a number you can defend in a budget meeting. Last year one client reallocated about $90,000 a month from branded search into prospecting video after two holdouts. Revenue grew 11 percent year over year on a flat media budget.
Start with one test
If you have never run a holdout, start with branded search. It is easy to turn off, the results come quickly, and it is the channel most likely to be over-reported. If you want help designing one, email me or talk to Priya Raman on our data team.