The Synthetic Data Delusion

Your AI personas don’t make impulse decisions.

Using AI agents as “synthetic” users to test marketing copy and predict campaign success is the latest attempt by marketers to achieve a long-standing goal: eliminating the massive cost and time penalties associated with launching a campaign that doesn't resonate.  The previous methods (in-person and online panels, focus groups, etc.) were about observing consumer attitudes; the new goal is to simulate consumer response. The industry is betting that a model trained on 100 years of consumer data can predict a flop faster and more accurately than a human who might be having a bad day or lying to stay polite in a focus group.

The Misfire: A brand uses AI personas to validate a new campaign.  The AI scores the campaign with 10/10 for resonance.  The campaign launches to real humans and flops.  There are lots of examples of this.

• Pepsi/Kendall Jenner: a global brand used AI personas to test a campaign focused on a sensitive social issue. The AI personas gave the campaign high marks, seeing the campaign as "inspiring and on-brand." Meanwhile, real humans found the campaign performative and tone-deaf, leading to a massive PR backlash.

• Quinoa in a QSR: a major fast-food chain used thousands of synthetic personas to test a new menu offering. The AI personas concluded that they would purchase this item because it would meet their caloric and nutritional requirements, and the data suggests a 20% lift in lunchtime traffic. Whoops. In reality, even health-conscious people going to a QSR are often driven by other things, like impulse or a craving for salt/sugar. The AI modeled who the consumers aspire to be, not as they actually are when they’re hungry.

• The Averaging Effect: a luxury streetwear brand used AI to predict which edgy designs would resonate with their primary audience. The AI personas validated a set of designs that are a perfect mathematical synthesis of what was popular on Instagram at the time. But... the core fans of the brand hate it. By validating designs against personas built on existing data, the brand created something that was statistically perfect but culturally missed the mark, because it only knew where the market had been, not where it was going.

The Reality: AI personas are logical, polite, and consistent, whereas real consumers are irrational, impulsive, and culturally fickle. Marketers are moving to AI agents because every previous method—even virtual panels—still requires human coordination. If a virtual panel is a motorcycle compared to the bicycle of focus groups, AI agents are seen as a space shuttle.

The Lesson: Simulated insights lead to simulated growth.  Using synthetic data may provide directional guidance, but needs to be combined with the years of insights and and intrinsic knowledge that already exists within your organization. When you use synthetic data, as with use of any predictive solutions, you have to temper your enthusiasm with realistic expectations of how to interpret the output. If your internal data shows that your customers hate BOGO offers, but the AI persona says it’s a top performer, the AI isn't smarter—it just doesn't know your customers as well as you do.

Have you had different results using synthetic data, or taken a different approach? Would love to hear about the experience others have had. Want to chat? Find a convenient time here.