AI in Data Collection

Day 2 - Block 2

DGPs, instruments, and AI

Let’s think about a DGP that we measure with “instruments” (e.g. survey, experiment, administrative data, …)

DGP	Instrument	AI example
Natural	Same	Generating stimuli images
Natural	New	AI qualitative interviewer
AI	Both	Synthetic respondents

Natural DGP, same instruments

Adapted from Sarstedt et al. (2024)

Natural DGP, same instruments

Adapted from Blanchard et al. (2025)

Natural DGP, same instruments

“How complex do you think the design of the presented product is? (1=simple, 9=complex)” (Sarstedt et al. 2024)

Natural DGP, NEW instruments

Traditionally:

quantitative data is “long” (large \(N\), small \(d\))
qualitative data is “wide” (small \(N\), large \(d\))

This mustn’t be the case anymore!

AI DGP

Synthetic respondents
AI decision making (e.g. AI online shoppers)

Synthetic respondents

Step 1: Covariates

id	Age	Charity Frequency	Income Level
1	45	Occasionally	Middle

Step 2: Background Story Prompt (Moon et al. 2024)

Prompt:

BACKGROUND A 45-year-old donor who gives to charity occasionally, with a middle income level. QUESTION Would you donate to this new cause? ANSWER

Response:

Yes, I would consider donating, especially if I felt the cause was important.

Step 3: Record Answers

Why might this work?

Massive pre-training, emergent behavior (Argyle et al. 2023; Dillion et al. 2023)
Hybrid approach (Arora, Chakraborty, and Nishimura 2025)

Mixed Evidence

✅: Arora, Chakraborty, and Nishimura (2025), Moon et al. (2024), Park et al. (2024), Li et al. (2024)

0️⃣: Hermann (2025), Goli and Singh (2024), Gui and Toubia (2023)

❌: Lin (2025), Stromberg et al. (2025)

Twin-2k-500 dataset

Toubia et al.

References

Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis 31 (3): 337–51. https://doi.org/10.1017/pan.2023.2.

Arora, Neeraj, Ishita Chakraborty, and Yohei Nishimura. 2025. “AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators.” Journal of Marketing 89 (2): 43–70. https://doi.org/10.1177/00222429241276529.

Blanchard, Simon J, Nofar Duani, Aaron M Garvey, Oded Netzer, and Travis Tae Oh. 2025. “New Tools, New Rules: A Practical Guide to Effective and Responsible Generative AI Use for Surveys and Experiments in Research.” Journal of Marketing 89 (6): 119–39.

Dillion, Danica, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. “Can AI Language Models Replace Human Participants?” Trends in Cognitive Sciences 27 (7): 597–600. https://doi.org/10.1016/j.tics.2023.04.008.

Goli, Ali, and Amandeep Singh. 2024. “Frontiers: Can Large Language Models Capture Human Preferences?” Marketing Science 43 (4): 709–22. https://doi.org/10.1287/mksc.2023.0306.

Gui, George, and Olivier Toubia. 2023. “The Challenge of Using Llms to Simulate Human Behavior: A Causal Inference Perspective.” arXiv Preprint arXiv:2312.15524.

Hermann, Erik. 2025. “Self-Driving Labs: The New Frontier for GenAI-driven Marketing Research.” Business Horizons, June, S0007681325001028. https://doi.org/10.1016/j.bushor.2025.06.001.

Li, Peiyao, Noah Castelo, Zsolt Katona, and Miklos Sarvary. 2024. “Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis.” Marketing Science 43 (2): 254–66. https://doi.org/10.1287/mksc.2023.0454.

Lin, Zhicheng. 2025. “Six Fallacies in Substituting Large Language Models for Human Participants.” PsyArXiv. https://doi.org/10.31234/osf.io/uqxcb_v2.

Moon, Suhong, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Kohen Behar, and David M. Chan. 2024. “Virtual Personas for Language Models via an Anthology of Backstories.” arXiv. https://doi.org/10.48550/arXiv.2407.06576.

Park, Joon Sung, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. “Generative Agent Simulations of 1,000 People.” arXiv. https://doi.org/10.48550/arXiv.2411.10109.

Sarstedt, Marko, Susanne J. Adler, Lea Rau, and Bernd Schmitt. 2024. “Using Large Language Models to Generate Silicon Samples in Consumer and Marketing Research: Challenges, Opportunities, and Guidelines.” Psychology &Amp; Marketing 41 (6): 1254–70. https://doi.org/10.1002/mar.21982.

Stromberg, Malik, Wendy W. Moe, Thomas Reutterer, and David A. Schweidel. 2025. “Blind Spots in Broad Strokes: Caveats for the Use of LLMs in Marketing Research.” SSRN. https://doi.org/10.2139/ssrn.5145114.