LLM Limits: Correlation vs Causation for AGI
Columbia professor Deshal Mesra argues that current LLMs are sophisticated Bayesian engines but lack causal reasoning and plasticity. This analysis explores the mathematical proof of transformer behavior and the strategic implications for the path to AGI.
The Mathematical Reality of Large Language Models
Recent research by Columbia University professor Deshal Mesra provides a definitive mathematical framework for understanding how Large Language Models (LLMs) function. Contrary to popular misconceptions, LLMs are not merely pattern-matching engines; they are sophisticated Bayesian inference machines. Through the development of a "Bayesian wind tunnel," Mesra and his colleagues demonstrated that transformer architectures update their predictions in a precise, mathematically predictable way. In controlled experiments where memorization was impossible, transformers matched the theoretically correct Bayesian posterior with an accuracy of 10 to the power of minus three bits. This confirms that the core mechanism of LLMs is Bayesian updating, where the model adjusts its belief about the next token based on new evidence provided in the prompt.
The Critical Gap: Correlation vs. Causation
Despite this mathematical elegance, Mesra argues that current LLMs are fundamentally limited by their reliance on correlation rather than causation. Deep learning excels at association, mapping inputs to outputs based on statistical likelihood. However, it fails at the higher levels of Judea Pearl’s causal hierarchy: intervention and counterfactuals. Humans do not calculate probabilities when dodging a pen; they simulate the outcome using a causal model. LLMs lack this internal simulator. They operate within a fixed manifold of learned correlations and cannot generate new representations or manifolds to explain anomalous data. This limitation is why LLMs cannot derive new scientific theories, such as Einstein’s theory of relativity, from existing data. They can process the data, but they cannot reject the existing axioms to create a new, shorter, causal explanation.
Strategic Implications for the Path to AGI
The path to Artificial General Intelligence (AGI) requires solving two distinct problems: plasticity and causality. First, LLMs must move beyond frozen weights to implement continual learning, allowing them to retain knowledge across sessions without catastrophic forgetting. Second, they must transition from Shannon entropy (predicting the next token) to Kolmogorov complexity (finding the shortest program that explains the data). This shift requires new architectures that can build causal models and simulate interventions. For business leaders and investors, this suggests that the current race for scale is hitting a ceiling. The next breakthrough will not come from larger models, but from architectural innovations that enable causal reasoning and lifelong learning. Until then, LLMs remain powerful tools for association and retrieval, but not for true innovation or discovery.
Key insights
-
Transformers mathematically perform Bayesian updating, adjusting posterior probabilities in real-time as new evidence is presented in the context window. This has been proven through controlled experiments where models matched theoretical Bayesian solutions with high precision.
Impact: Validates the reliability of LLMs for probabilistic reasoning tasks and provides a clear mathematical basis for debugging and optimizing model behavior.
-
Current LLMs are limited to the first level of the causal hierarchy (association) and cannot perform intervention or counterfactual reasoning. They lack the internal causal models required to simulate outcomes or understand why events happen.
Impact: Explains why LLMs fail at complex planning, scientific discovery, and novel problem-solving, necessitating hybrid systems or new architectures for high-stakes decision-making.
-
LLM weights are frozen after training, preventing true plasticity. Unlike human brains, which remain plastic throughout life, LLMs forget all learning from previous sessions unless explicitly stored in external memory.
Impact: Highlights the need for continual learning frameworks in enterprise AI deployments to ensure models can adapt to changing business environments without retraining.
-
Scale alone is insufficient to achieve AGI. Increasing model size and training data will not bridge the gap between correlation and causation. A fundamental architectural shift is required to enable causal reasoning.
Impact: Redirects investment focus from brute-force scaling to architectural innovation, signaling a potential shift in the competitive landscape of AI development.
-
The "Einstein Test" serves as a practical benchmark for AGI: the ability to derive new theories from existing data. LLMs cannot pass this test because they cannot create new representations or manifolds to explain anomalous data.
Impact: Provides a clear, high-bar metric for evaluating AI progress, distinguishing between sophisticated pattern matching and true general intelligence.
Action items
-
Implement external memory systems to mitigate the lack of plasticity in LLMs. Use vector databases and retrieval-augmented generation to allow models to retain and access learning across sessions.
Impact: Enhances the adaptability of AI systems in dynamic business environments, reducing the need for frequent retraining and improving long-term value.
-
Design AI workflows that explicitly separate correlation-based tasks from causal reasoning tasks. Use LLMs for data retrieval and pattern recognition, but employ human experts or specialized causal models for decision-making and planning.
Impact: Reduces the risk of AI errors in high-stakes decisions by leveraging the strengths of LLMs while compensating for their lack of causal understanding.
-
Invest in research and development of continual learning architectures. Focus on methods that allow models to update their weights without catastrophic forgetting, enabling true lifelong learning.
Impact: Positions the organization at the forefront of the next wave of AI innovation, potentially leading to more robust and adaptable AI systems.
-
Adopt the "Einstein Test" as a benchmark for evaluating AI capabilities in scientific and research contexts. Assess whether AI systems can generate new hypotheses or theories from existing data, rather than just retrieving known information.
Impact: Provides a rigorous metric for AI performance in innovation-driven fields, helping to identify true breakthroughs versus incremental improvements.
-
Educate stakeholders on the distinction between correlation and causation in AI outputs. Train teams to interpret LLM responses as probabilistic associations rather than causal explanations, reducing over-reliance on AI for complex reasoning.
Impact: Improves the quality of human-AI collaboration by setting realistic expectations and ensuring that AI insights are used appropriately in decision-making processes.
Quotes
“To get to AGI, Misra argues, we need the ability to keep learning after training and the move from correlation to causation.”
“Scale will not solve everything. You you need a different kind of architecture.”
“You take an NLM and train it on pre-1916 or 1911 physics and see if it can come up with the theory of relativity. If it does, then we have AGI.”