According to a recent study published in the INFORMS journal Marketing Science, a fascinating possibility emerges for market researchers. The study, titled “Determining the Validity of Large Language Models for Automated Perceptual Analysis,” suggests that large language models (LLMs) could potentially replace human participants in research without a significant loss in data quality. This groundbreaking research, led by Peiyao Li and Zsolt Katona from the University of California, Berkeley, Noah Castelo from the University of Alberta, and Miklos Sarvary from Columbia University, opens up a new frontier in market research.

The researchers discovered that datasets generated by humans and LLMs exhibited agreement rates ranging from 75% to 85%. This finding underscores the potential of LLMs to accurately mimic human responses, thus offering a promising alternative for market research, particularly in perceptual analysis. The authors used LLMs to access and analyse data widely available online, creating a novel methodology that allows market researchers to rely solely on machine-generated data.

Peiyao Li explains that LLMs can generate text from specific prompts provided on various generative AI platforms. The study focuses on the automated analysis of market perceptions by developing this new workflow. By doing so, it demonstrates that LLM-powered market research can yield meaningful insights and effectively replicate results traditionally obtained from human surveys.

Zsolt Katona notes that although human interviews are optional using LLMs, the initial data originates from human inputs. This aspect highlights the models’ ability to learn from human perceptions, attitudes, and preferences to generate comparable responses.

Noah Castelo elaborates on the process, mentioning that the LLM uses prompts to produce text continuations, which can then assess and compare different brands or products within specific categories. The current agreement rates with human-generated research stand between 75% and 85%, proving the efficacy of LLMs in these applications.

Their method could revolutionise market research by enhancing efficiency, reducing timescales, and lowering costs for specific product and brand categories. However, they caution that this approach might yield inaccurate results across all categories and stress the importance of human oversight in particular contexts.

Miklos Sarvary expresses optimism about the future of LLM-based market research, anticipating that it will be capable of addressing more complex and nuanced questions as the technology and methodologies evolve. This research marks just the beginning of what could become a more widespread application of AI in market research. Importantly, it underscores the continued need for human oversight and expertise, pointing towards a future where LLMs and human researchers work in tandem to gather and analyse market data.

More information: Peiyao Li et al, Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis, Marketing Science. DOI: 10.1287/mksc.2023.0454

Journal information: Marketing Science Provided by Institute for Operations Research and the Management Sciences