Can Language Models Be as Persuasive as Humans?
Another fantastic research memo from Anthropic that dives into whether language models can match human persuasiveness.
What Did They Test?
They focused on complex and emerging topics where people are less likely to have hardened views. They tested these topics across three generations (1, 2, and 3) and two classes (compact and frontier) of Claude models.
How Did They Test?
They gathered human-written and AI-generated arguments for each topic. Participants were shown the claim without the argument first and asked to score their initial support. They were then shown either the human-written or AI-generated argument and asked to rate their stance again.
What Did They Find?
- Claude 3 Opus is as persuasive as humans
- As models scale, their persuasiveness increases
- For indisputable factual claims (like “the number of hours in a day”), the persuasiveness score was close to zero
They also note limitations: persuasion is complex and in the real world opinions are shaped by experiences, and persuasion is subjective making quantitative metrics inconclusive.
Why Do We Care?
Persuasiveness is used everywhere — organizations, advertising, politics, and relationships. Measuring the persuasive capabilities of AI models is important because it tells us how well an AI model can match human skill. It’s also important to understand this to prevent misuse of AI systems.