Artificial intelligence can produce dozens of marketing messages in seconds, but deciding which one will actually connect with customers remains a much harder task. New research from Penn State suggests businesses can get more value from AI by combining its speed and creative capacity with lessons from past campaigns — and the judgment of experienced marketers.

Wreetabrata “Wreeto” Kar, assistant professor of marketing at Penn State’s Smeal College of Business, and his co-authors developed a framework that uses a company’s previous marketing results to evaluate newly generated AI content. Rather than relying solely on expensive and time-consuming testing, the approach helps marketers identify promising material before it reaches customers. The study was published in the Journal of Marketing Research.

AI is already becoming part of nearly every stage of marketing. Beyond drafting emails, advertisements, social media posts and product descriptions, the technology can generate images and videos, brainstorm campaign ideas, analyse markets, segment customers and personalise messages. Some businesses are also using AI to anticipate customer behaviour and automate parts of the customer journey.

That productivity, however, creates a new problem: too many choices. A marketer who once produced one email might now generate 50 versions in the same amount of time. AI can create those alternatives quickly, but it does not necessarily know which one will work for a particular business or audience. A polished, persuasive message may still fail to produce results.

Kar and his colleagues address that problem by teaching AI to learn from a company’s marketing history. The system converts the meaning of previous messages into numerical representations, effectively placing them on a map. Messages with similar meanings appear close together, even when they use different words, allowing the model to recognise relationships that simpler word-counting techniques might miss.

When a new message is created, the system compares it with previous campaigns. If it resembles messages that performed well, the model can estimate that it may also succeed. If the content is unlike anything the company has tried before, the system signals greater uncertainty and recommends traditional testing. This allows businesses to reserve A/B testing for genuinely new ideas rather than testing every variation.

The researchers also found an important limit to AI’s abilities. When an AI model was asked not only to generate marketing emails but also to select its own five best messages, its choices were predicted to perform poorly. The finding suggests that generating content and deciding what will work are distinct tasks — and that AI should not necessarily be trusted to judge its own output.

That leaves marketers firmly in the decision-making process. They determine the ingredients AI uses, decide how much experimentation and risk are acceptable, and make the final selection from the strongest candidates. In this model, AI handles much of the labour-intensive work of generating and analysing content, while people contribute business knowledge, context and judgment. As AI dramatically expands the number of possibilities available to marketers, those human skills may become more important, not less.

More information: Paul Ellickson et al, Evaluating Novel Unstructured Treatments with Generative AI: A Causal Prediction Framework, Journal of Marketing Research. DOI: 10.1177/00222437261476639

Journal information: Journal of Marketing Research Provided by Penn State