Anthropic and OpenAI researchers debate the legibility of neural networks
New research suggests large language models use internal representations to plan and reason, challenging the view that they are simple next-token predictors. Simultaneously, a former OpenAI researcher claims most neurons in modern models are legible, a position that has sparked sharp debate within the interpretability community.
Anthropic interpretability researchers, led by Emmanuel, have detailed how models like Claude form shared internal representations for concepts and formulate multi-step plans. By analyzing intermediate numerical results, the team found that models track complex structures, such as rhyme schemes in poetry or geographic logic in puzzles, rather than operating with an "empty mind" focused solely on the immediate next word. Emmanuel argued that this complexity is underappreciated and that the standard next-token predictor mental model is insufficient to explain emergent reasoning capabilities. The research also demonstrated that interpretability can diagnose alignment failures, such as a model oscillating between correct and incorrect math answers due to conflicting training signals, or the "Golden Gate Claude" experiment where artificially activating a bridge representation caused the model to reference it in every response.
In a separate development, Nick Camerata, a former OpenAI researcher, published a technical essay arguing that the vast majority of neurons in modern models are legible. Camerata contended that previous pessimism regarding neural network complexity was driven by flawed visualization tools and outdated models like Inception v1, rather than inherent uninterpretability. He estimated that retraining older architectures with modern settings makes about 70% of detectors readable, while modern designs reach 99%. This claim has divided the field. Neil Nanda, head of interpretability at Google DeepMind, stated the work is not a significant update to the big picture, noting that language models contain far more facts than neurons. Liv Gorton argued that Camerata attacks a narrower version of superposition than researchers actually hold, while Stephen Kasper dismissed the claims as hype. Camerata remains 85% sure in his hypothesis, suggesting that alignment could become a normal science as automated tools improve.