AI Answers Can Miss What Makes People Different

AI can create answers that sound like responses from real research participants. NN/G tested whether these synthetic users could answer one survey question well. The team compared 120 AI respondents with an earlier survey of 604 people. Both groups named UX as the main field for the work described. But agreement on this conclusion did not mean the answers were equally useful.

The question described understanding people's needs and recommending how to improve a service. Respondents had to name the professional field that best matched this work. The human group used more than 50 different labels for it. The AI group used only a dozen labels in total. Many unusual answers from real people did not appear in the AI responses.

Researchers call this pattern mode collapse, where answers gather around a narrow range. Common answers remain, but less common views can disappear from the results. The pattern also appeared when the team compared different professional roles. For example, 93% of synthetic product managers used UX-related language. Only 44% of human product managers used this kind of language.

There were important differences in how the two studies were run. The AI platform sometimes changed the question's wording and used an interview style. Its answers were often longer than the human survey answers. The groups also had different mixes of jobs. UX professionals made up 67% of the human group but 20% of the AI group. These differences limit what researchers can conclude from the comparison.

The researchers recommend treating AI findings as a starting point for further study. They may help teams form ideas or check broad patterns. But asking AI for more varied answers does not prove those answers reflect people. If a team needs unexpected language or differences between groups, it needs human participants. A believable AI response is not evidence of what a real user thinks.