LLMs constrained to average answers without external validation mechanisms

Large language models are capable of automating digital tasks where responses can be validated at scale, but they fundamentally output what most people would probably say, effectively providing an average of human thought. This creates a theoretical barrier to true creativity or novel scientific discovery, as models struggle to generate content that is 'different but good' without an internal scoring system to test new ideas. AlphaGo is cited as a counter-example because it could invent new moves and validate them via an internal scoring system, allowing it to deviate from standard patterns.

A recent paper attempted to determine if a model given pre-Einstein physics knowledge could make the jump to Einstein's theories, though the answer remains untested. The 'million interns' analogy describes using vast numbers of fast agents to brute-force solutions, such as in cybersecurity, rather than relying on singular cleverness. The core question for AI strategy is whether the desired outcome is the average answer or something novel that does not exist in the training data. Current LLMs are constrained to extrapolating from existing data and cannot easily evaluate novelty without a specific validation mechanism, distinguishing them from systems that can test new strategies against a defined metric.