Having the Data Isn't the Same as Testing It

The first three posts in this series were mostly about vocabulary, architecture, and the ambiguity buried inside a single headline metric. This post is about something that sounds like it should settle all of that: real outcome data.

Surely a platform sitting on actual purchase records - not licensed, not inferred, not appended from a survey - has solved the problem the rest of this series has been circling. It turns out that’s necessary, but not sufficient, and the gap between the two is worth understanding.

A small piece of industry structure matters here. Some retailers run their own data-science operations directly on their own transaction and loyalty data - the retailer sees the actual basket, tied to an actual loyalty ID, and can measure whether a specific household that saw a specific offer went on to buy the specific product. That’s a fundamentally different starting point than a vendor working from licensed, third-party, or modeled behavioral data, because the outcome being measured is a verified transaction rather than a proxy for one. Call this the “walled garden” model - it also describes how the major ad platforms operate, though for a different reason: they see the ad exposure and the resulting conversion event (a pixel firing, a tag reporting back) inside their own closed ecosystem, at a scale nothing else can match.

Here’s the distinction that matters, though: seeing both the exposure and the conversion event is not the same as testing whether the exposure caused the conversion.

Most reporting in this category - from ad platforms and from retail media alike - is attribution-based: a rule assigns credit for a conversion to whichever ads or clicks the model decides to credit, and those credits gets reported as the result. Attribution is a correlation exercise dressed as a causal one. It can’t distinguish “this ad caused a sale” from “this person was already going to buy, and the ad happened to touch them somewhere along the way” - the classic problem with a brand-search ad that shows spectacular ROAS for capturing people who already knew exactly what they wanted.

The only real test for that distinction is a holdout: deliberately withholding the ad, offer, or targeting from a randomized slice of the audience or geography, and comparing what actually happened against what would have happened anyway. That’s a genuinely different exercise than attribution - often built on top of open methodologies like Google's CausalImpact framework - and it’s still the exception rather than the default, though that’s shifting fast. Roughly half of US brand and agency marketers now report running some form of incrementality testing, and a majority now name incremental ROI as a top measurement priority. (I wrote more on why this distinction matters, and how causal validation is becoming accessible without individual-level tracking, in a recent piece.)

And this is the part that should recalibrate a bit of what a “walled garden” claim seems to promise: attribution may not just get the size of an effect wrong, it can be wrong about which channels deserve credit at all. One measurement vendor’s published dataset of 225 geo-based incrementality tests across direct-to-consumer brands found branded search - often the channel advertisers trust most, since someone typed the brand’s name and clicked before buying - carried the lowest median incremental return of any channel tested, sitting below breakeven, while upper-funnel channels that last-click attribution tends to shortchange came out well ahead. (Source: Stella’s 2025 DTC Digital Advertising Incrementality Benchmarks) Worth flagging plainly, in keeping with the standard this series keeps asking of everyone else’s numbers: that’s one vendor’s dataset, DTC-specific, and self-selected toward measurement-sophisticated advertisers is directional, not a universal benchmark.

But the shape of the finding is the useful part: attribution’s error isn’t random noise, it has a predictable direction, systematically over-crediting the last click and under-crediting the work that happened earlier, even in multi-touch models. That pattern shows up whether the platform in question is working from licensed behavioral signal, its own on-platform engagement data, or a retailer’s actual verified purchase records. Having real transaction data eliminates one problem, whether the sale happened, but doesn’t by itself eliminate the harder one: whether the marketing caused it, or just got credit for it.

This also reframes the “black box” concern from earlier in the series. Some ad platforms now treat an advertiser’s own carefully composed audience as an optional suggestion: the system can and will expand past it whenever its internal model predicts a better result, without much visibility into which traits actually drove that decision. Paired with a self-graded, attribution-based measure of success, that’s close to the least accountable version of “signal-based” targeting available today. Enormous scale, a genuinely closed loop, and a report card written entirely by the party being graded doesn't pass muster.

None of this means walled gardens or ad platforms are worse than open, signal-licensing vendors; if anything, the platforms with real transaction data are the ones capable of running a genuinely trustworthy test, because they have the infrastructure a holdout requires. The point is narrower: “we have real data” and “we tested this causally” are two different sentences, and the vocabulary in this category tends to let the first one imply the second.

That raises a question one level deeper than measurement methodology, and it’s the natural next one to ask even of a properly run holdout test: is the audience it validated actually new demand, or is it the same population that other channels, or the brand’s existing customer base, were already going to convert anyway? A well-run test tells you an audience is incremental relative to not running the campaign. It doesn’t automatically tell you whether that increment is durable, or whether the next campaign against the same signal will find anywhere near as much of it. That’s what we'll look at in the next installment.

Next: what happens when a signal-based audience actually gets scaled? Why the first activation of any newly discovered segment tends to be the best one it will ever have, and why “we found demand nobody else saw” deserves real skepticism as a category of claim.

What are your thoughts on this issue? Start a conversation in the comments below.