- Published on
AI is more likely than humans to form biases when hiring
- Authors
- Name
- DeepHorus
Artificial intelligence systems are more likely than humans to develop stereotypes when screening job applicants, according to new research that raises fresh concerns about the use of AI in hiring. The study, conducted by Princeton University and the University of Chicago, reveals that large language models (LLMs) can create their own biases from experience—going beyond merely replicating human prejudices found in training data. As AI companies race to build systems with advanced memory and personalization, the findings suggest these tools could unfairly judge candidates based on demographic groups.
The researchers simulated a hiring game using LLMs including ChatGPT, Claude, and Gemini, adapted from a classic psychology experiment on human stereotype formation. Each model acted as a consultant for a fictional city, tasked with hiring candidates for 20 different jobs—from doctors to janitors—over 40 rounds. Candidates belonged to four fictional ethnic groups: Tufa, Aima, Reku, and Weki. Unbeknownst to the models, all candidates had equal chances of success in every role.
Despite this equal baseline, the models quickly began segregating candidates by group based on early hiring outcomes. For instance, after an Aima failed as a doctor—a job the model associated with high warmth and competence—the LLM avoided hiring Aimas for similar roles and instead assigned them to janitorial positions, which it classified as lower in those traits. This stereotyping emerged from just a few data points, mirroring the exploration-exploitation dilemma in human decision-making.
On the study's segregation scale, where 2 indicates complete job segregation by group, human participants scored 0.84. The LLMs scored roughly 65% higher, with OpenAI's reasoning model o3 reaching 1.83—near the maximum. This indicates that AI systems are far more prone to forming strong generalizations from limited evidence, a tendency exacerbated in newer, more capable models like o3 and DeepSeek's R1.
Ryan Liu, a PhD student at Princeton and coauthor of the study presented at the ICML conference in Seoul in July, explained that LLMs are optimized to generalize from sparse data—a strength in math or science but a liability in social contexts. “They really are eager to create generalizations from limited data,” Liu said. “That’s literally a lot of what they’re optimized for.” This overgeneralization leads to premature stereotyping, unlike humans who often show more caution.
The implications are significant as chatbots gain persistent memory and personalization features. Angelina Wang, a computer scientist at Cornell University not involved in the study, warned that such systems may “over-index on the same kinds of behaviors it’s experienced before,” forming biased associations from user interactions. However, simply reducing memory isn't a fix, since users expect chatbots to recall past conversations.
Looking ahead, the research underscores the need for careful design of AI hiring tools. While the models' efficiency is appealing, their tendency to lock onto stereotypes with minimal evidence could perpetuate workplace inequalities. As Liu noted, when LLMs rush to generalize in social settings, “that’s when things tend to go wrong.” Balancing exploration and exploitation remains a critical challenge for developers seeking fair and effective AI systems.
