Marketing evaluation experts warn of dangers of AI

Marketing evaluation experts have published an open letter warning of the potential dangers of using platform-based AI solutions in market mix modelling (MMM) and econometrics.

The letter is signed by 17 experts in the field, including adam&eveDDB head of effectiveness and co-author of The Long and the Short of It, Les Binet, and Marketing Week columnist and founder of consultancy Magic Numbers, Dr Grace Kite.

In recent times, new providers have emerged to offer automated platform-based MMM powered by AI, but experts in the field have identified serious issues after looking at the algorithms powering these automated models.

Artificial intelligence isn’t enough for this task, real people are needed.

Grace Kite, Magic Numbers

“We’ve looked at the algorithms and we have to tell you, they’re much simpler than they need to be,” the letter says.

“With MMM, every time you don’t include something that matters you get a wrong number for the effect of advertising, and there are models out there that don’t even include Covid-19 or price.”

The signatories of the letter are clear humans must be involved in market mix modelling for it to work.

“Artificial intelligence isn’t enough for this task, real people are needed,” says Grace Kite. “Machines can’t yet understand the nuances of the situation, identify things that are missing in data, or help people get comfortable that results are reliable for important decisions.”

While these platform-based solutions are fairly new, the evaluation experts who have signed the letter report that marketers have already fallen foul of unreliable systems, at great cost to their business.

“We’ve heard stories where 10 months of set up were followed by garbage results that were completely unusable,” says MeasureMonks co-founder Mike Cross. “Making the wrong decisions off a bad model costs up to 40% of your media driven revenue, versus MMM applied properly which can deliver +30%. That’s pretty costly to a CMO in austere times.”

Meanwhile, Nancy Smith, CEO and president of Analytic Partners, urges: “Don’t be a guinea pig to inexperienced vendors.”

The letter’s authors make an apology for not flagging enough issues with last-click attribution when this was new. “That’s why we wanted to write to you now. Because there’s danger on the horizon again,” it says.

The letter also offers advice to avoid being taken in by unreliable vendors. Marketers are urged to shop for MMM like “you’re buying a new kitchen” by asking for recommendations, getting at least three quotes and applying a healthy dose of cynicism.

The authors of the letter provide example questions for marketers to ask to ensure the MMM vendor is up to scratch. These include asking about the factors included in the model, how long it covers, whether it could be explained to finance people within the organisation and what would happen in the event results come back and don’t make sense to the team due to something else they had seen.

The overall message is for marketers to be cautious before they jump into signing up to automated MMM, which looks too good to be true.

“The push for faster and cheaper solutions is leading to approaches which are simply not able to embrace the complexities of brand response. These approaches risk misleading rather than enlightening,” says Holmes and Cook director Louise Cook.