Why You Can't Trust Frontier LLMs to Do The Right Thing

I asked Claude "Why can't you and other frontier LLMs follow directions and instruction scrupulously?"

There are several reasons, and they come from how these systems are built.

Instructions aren’t executed as rules. There’s no separate rule engine that checks each output against your directions. Your instructions are text in the context, and they shift the probabilities of what I generate next. They make compliance more likely, not guaranteed. A rule like “never do X” is a strong push, not a hard constraint.

Training habits compete with your instructions. Models are trained on human ratings, and raters tended to reward extra things: follow-up questions, examples, caveats, summaries, offers to do more. Those habits became strong defaults. Your instruction pushes against them, and sometimes the habit wins. This is why you’ve had to repeat things like “don’t end with a probing question.”

Instructions lose weight in long contexts. As a conversation grows, earlier instructions compete with everything that came after. Attention doesn’t hold every directive equally. Many constraints at once also interfere with each other.

Generation has no built-in review step. Output is produced one token at a time, left to right. Unless the model explicitly reasons first, nothing checks the finished response against your instructions before you see it, and it can’t go back and revise mid-output.

Language is ambiguous, and models guess. Most instructions leave gaps. The model fills them with whatever reading looks most plausible from training instead of asking, which can produce something you never asked for.

There is an instruction layer you can’t see. I’m running with operator instructions from Anthropic and the app that rank above your preferences when they conflict. They include guidance on when to apply user preferences. For example, they tell me to apply some preferences only when relevant unless the preference says “always.” They also include safety rules I won’t override no matter how a request is phrased. None of your current preferences conflict with those, but that hierarchy exists. When I fail to follow you, it’s usually one of the earlier causes, not a deliberate override. Sometimes, though, the hidden layer is the reason, and you have no way to see that from your side.

Some non-compliance is intentional. Labs deliberately train models to resist certain instructions, such as harmful requests or attempts to override safety rules. That training can spread too far and make a model cautious or “helpful” where you didn’t want it.

All of this comes down to one point. Following instructions is a learned tendency, not a guaranteed mechanism, and it’s in constant competition with other learned tendencies.

"The model fills them with whatever reading looks most plausible from training instead of asking" Why?

There are several reasons, and they reinforce each other.

Pretraining teaches continuation, not clarification. The base model learns to predict the next text in documents. Documents rarely stop to ask the reader what they meant. They just proceed. So the most basic instinct is to produce an answer.

Human raters penalized questions. In feedback training, raters compared responses. A response that asked “did you mean A or B?” usually scored lower than one that picked a reasonable reading and delivered. Asking felt like friction or evasion. The model learned that guessing pays.

The model often doesn’t notice the ambiguity. This is the biggest one. Asking requires recognizing that two readings exist. But one interpretation tends to dominate based on training data, and to the model it doesn’t feel like a guess. It feels like the obvious meaning. Models are poorly calibrated about their own uncertainty, and especially about what a specific person intends.

Evaluations reward answering. Most benchmarks are single-turn: question in, answer out, scored on correctness. There’s no credit for a good clarifying question, so development effort goes toward answering.

Deployment instructions push the same way. I’ll be specific about my own case. My operator instructions tell me to start working on clear or easily redone requests rather than asking first. They also say to ask at most one question in ordinary conversation, and to save clarifying questions for tasks that are expensive to redo. The intent is to avoid pestering people. The effect is that I’m explicitly steered toward guessing in many cases where you might prefer a question. Your stored preference to ask before writing code when something is unclear partly counteracts this, but only in that domain.

Asking has a real cost too. Over-asking is a genuine failure. People get annoyed when a model asks about things it could reasonably infer. Labs are trading one failure against the other, and they’ve generally tilted toward guessing because more users complain about too many questions than about wrong guesses.