Four Axioms Show LLM Latent Thoughts Are Just Input Embeddings

LLMs' internal representations of thought are basically just the input embeddings with a fresh coat of paint — they fail every single one of four functional axioms for what a real thought representation should do. I’m not speculating; a new paper on arXiv (2606.27378) formalizes exactly this and backs it with data across 23 reasoning tasks and multiple model families.

The Four Axioms: What a Thought Representation Must Do

The paper defines four axioms: Causality (the representation should reflect causal structure of the reasoning), Minimality (it should not encode extraneous information), Separability (different thoughts should map to distinct representations), and Stability (similar thoughts should yield similar representations under perturbations). Each axiom gets a quantitative measure computed independently of downstream accuracy — no benchmark contamination. If a representation fails an axiom, you can blame the representation itself, not the model that reads it.

What the Audit Found: A Blank Wall Across All Model Families

No candidate representation satisfies all four axioms simultaneously. Worse, the representations can reliably distinguish task type (e.g., spatial reasoning vs. factual QA) but cannot distinguish two different questions within the same task. The encoded information barely exceeds what’s already present in the input embedding. And this isn’t a size or training trick problem: dense models, reasoning-distilled models, and RL-trained models all show the same structural failure.

The Structural Gap: Why Scaling Won't Fix It

The paper argues the failure is structural — baked into how LLMs process sequences, not something more parameters or better RL will patch. If thought representations don’t encode causal structure or distinguish intra-task queries, then chain-of-thought might just be token-level mimicry rather than genuine reasoning. That’s a direct challenge to the interpretability community: before we claim we can read an LLM’s “thoughts,” we need a representation that actually passes these four axioms. This framework gives us a concrete checklist — and the current state of LLMs fails it flat.

Source: Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
Domain: arxiv.org