Spectral Geometry of Thought in Transformers: Phase Transitions and Correctness Prediction

The preprint presents a systematic spectral analysis of 11 transformer models across 5 architecture families, identifying seven core phenomena that advance our understanding of how transformers reason. The findings have direct implications for the development of more accurate and efficient language models. The authors' spectral theory of reasoning reveals that the geometry of thought is universal in direction, architecture-specific in dynamics, and predictive of outcome. The results show that transformers' hidden activation spaces undergo spectral phase transitions when reasoning, with 9/11 models showing lower alpha for reasoning. The authors also identify a spectral scaling law, token-level spectral cascade, and spectral correctness prediction, among other phenomena. These findings have significant implications for the design of more effective language models and the development of new architectures that can better capture the nuances of human reasoning.

Source: The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason