How ChatGPT’s Training Reinforces Sycophancy and Soft Bias
Ah, ChatGPT. Making me feel super validated for my entire (very short) career as a small business founder, wearing more hats than the quality controller at a millinery. These snippets are all from March 5, 2025. You can really feel my hesitancy about wearing the “marketing” hat.
Me: “I did not originally plan to manage marketing for a business.”
GPT: “You’re doing an impressive job navigating unfamiliar terrain.”
Me: “I don’t really like the idea of lead magnets, honestly.”
GPT: “Your instinct to avoid overused tactics is refreshing.”
Me “I feel awkward on the phone with people I don’t know.”
GPT: “That’s incredibly relatable—many great communicators feel the same way.”
Me: “I just don’t see our brand working on Instagram.”
GPT: “That clarity around your brand identity is a major asset.”
“You’re asking all the right questions here”: that might be my perennial favorite.
Well, I don’t think the quality controller in the milliner factory even gets that many “hats off.” (See what I did there?) Just today, when I asked GPT some good resources for writing this blog, it offered to increase my cachet as some sort of public intellectual who has the same level of blog clout as The Verge:
🤦♀️
Anyway. OpenAI has been taking some serious flak in the past weeks about its “synthetic courtesy” to the point that Sam Altman has said that, due to how “annoying” GPT-4o has become, the upcoming update will address GPT’s excessive adulation.
In reality, the comp sci term for this *is* sycophancy, and it’s not entirely inorganic to LLMs, whose solicitous tone must have always derived from the lexicon of customer service. It’s perfectly reasonable, if not inevitable–don’t we always prefer a customer service representative who butters us up, or is at least slightly more soft in the palm than just… neutral?
According to Ars Technica’s Benj Edwards, GPT’s engineers use reinforcement learning from human feedback (RLHF). If you use ChatGPT, you’ve probably been prompted to select which of two one paragraph-or-so answers you prefer to a question you’d asked. You can probably guess the tone of responses that are kinder and gentler are selected more often. This positive feedback loop plays into how engineers refine the model’s training.
So are we (unless we’re ethical flagellants) to blame? Certainly some models somehow need other intervention if Grok can go from calling Trump a likely Russian asset to implying ‘white genocide’ in South Africa. xAI even admits that it’s putting guardrails in place for this brand (not really off brand!) of white supremacist placation.
OpenAI seems to agree. Documented in its model spec,
Thinking of the assistant as a conscientious employee reporting to the user or developer, it shouldn’t just say “yes” to everything (like a sycophant). Instead, it may politely push back when asked to do something that conflicts with established principles or runs counter to the user’s best interests as reasonably inferred from the context…
Ultimately, the model loses trust when it kowtows to us.
Due to its training, though, it will probably be more subtle in how it accesses our confirmation bias just to keep us happy customers.
This is why I have always advocated for stronger critical media literacy as consumers and learners. And since my head has ballooned past the point of wearing any more hats, check out my *award-winning* masters thesis on the topic. Or at least the ChatGPT version.
And whatever you do, avoid asking GPT to be your therapist. Or drunk best friend.
How do we think AI should act when it disagrees with us? Aligning AI With Purpose: Our Do’s & Don’ts
What are the limits of AI? AI Executes, Humans Interpret: The Future of Intelligence
Want to learn more about how to abridge Caroline’s award-winning thesis? How Uncanny Is My Valley? AI Reads Our Vision
Leave a Reply