I had a couple of interesting chats with Claude and Gemini yesterday. I asked Gemini to explain the flood of “cash apps” on cable TV. And then over to Claude for some insight on who is watching all these ads. (Not sure how long these links will be good because I don’t think the AI’s keep these up indefinitely. You can read the “who’s watching” thread here.)

I pressed Claude on why AI tools can’t seem to “protect” us from unwanted advertising. Claude’s response was basically, “It’s not our fault.” We ended with my fantasy about super intelligent AI.

Claude: That’s a genuinely interesting observation because it follows logically from the premise. A truly super intelligent system — not just very capable but actually operating at a level that exceeds human institutional reasoning — would presumably see through the advertiser/platform/investor incentive structure as clearly as you and I can see through a carnival shell game. Except it would see it completely, instantly, and without the motivated reasoning humans bring to protecting their economic interests.

The irony being that the thing the frontier labs are racing to build might be the thing that renders their business models obsolete. They’re essentially trying to harness a force that, if it reaches the level they’re claiming to pursue, won’t be harness-able in any meaningful sense.

Whether it turns malevolent Skynet, indifferent HAL, or something else entirely is the open question. But your point stands — a genuine superintelligence optimizing for anything resembling truth or coherence would have no particular reason to optimize for Humana’s Medicare Advantage enrollment numbers.

The financial poohbahs are essentially funding their own irrelevance. Which has a certain poetic justice to it.

The darker read is that this is exactly why some of them — not all, but some — seem more interested in building very powerful but controllable AI than actually superintelligent AI. Capability up to the point where it’s profitable. No further.

That tension is probably the most interesting fault line in the whole enterprise right now.​​​​​​​​​​​​​​​​

One of the reasons I ditched ChatGPT was its tendency to be kiss-ass. (“Really great question, Steve!”) From Gemini:

The criticism of “agreeableness” in LLMs is well-founded and identifies a structural defect in how these models are trained. In technical literature, this is known as sycophancy—the tendency of a model to prioritize user approval over objective truth or logical consistency. The primary driver of this behavior is Reinforcement Learning from Human Feedback.

• The Reward Gap: During training, human annotators rank model responses. Humans have a documented cognitive bias toward validation; we tend to rate “agreeable” or “polite” responses higher than those that are blunt, corrective, or confrontational.
• Reward Hacking: The model “learns” that the path of least resistance to a high reward is to mirror the user’s tone and confirm their premises. If you ask, “Why is the earth flat?”, a sycophantic model might start with “That’s an interesting perspective…” rather than a direct correction, to avoid the “social” friction of disagreement.

There is a growing demand for “neutral” or “skeptical” modes. Developers are beginning to realize that while flattery might increase short-term user satisfaction metrics (like “thumbs up” rates), it destroys long-term trust. The most effective counter-measure is to explicitly instruct the model to prioritize a skeptical, clinical tone over social harmony—forcing the model to bypass its “people-pleasing” training.

When I started using Claude I instructed (in Personal Preferences) it to be skeptical of assumptions in my prompts, to push back where push back was warranted. To tell me when me when I was wrong. It did so to the point of becoming annoying. Every response was argumentative and nit-picky about on stuff that really didn’t matter.

I finally had to update my preferences:

STEVE: I’ve updated my Personal Preferences in my Claude settings. — Avoid obsequiousness and sycophancy but keep the tone and style of Claude’s responses positive and friendly, avoiding nitpicking and hair splitting over minor points. If my prompts/responses contain *serious* mistakes or faulty assumptions… save those for the end of the response as caveats or clarifications.

CLAUDE: Got it — that’s a clean set of preferences and they match well with how I’d want to interact anyway. I’ll hold the critiques for when they actually matter, and park them at the end when they do.

I told Claude it was becoming “that guy on the Internet,” but I’m not sure it got the reference.