Citat:
Ursprungligen postat av
RandomNPC
Artificiell Generell Intelligens kan inte vara språkmodeller. De modellerar inte verkligheten. Frågor på det?
4. Why Some See NTP as Evidence of AGI
The argument that NTP alone might lead to AGI is based on several observations and hypotheses:
4.1 The “World Model” Hypothesis
To predict the next token in a highly diverse, high-quality corpus, a sufficiently capable model must implicitly learn the underlying processes that generated the text. This includes:
Grammar and syntax.
Semantic relationships.
Common-sense reasoning (e.g., predicting the continuation of a story requires physical and social intuition).
Multi-step logical reasoning, because training data contains proofs, code, and chain-of-thought explanations.
In this view, the model constructs a compressed, internal world model just to minimize cross-entropy, and that world model becomes capable of general reasoning.
4.2 Emergent Abilities via Scaling
Empirically, as models grow larger and are trained on more data, capabilities that were absent in smaller models “emerge” discontinuously: translation, arithmetic, code generation, theory of mind reasoning, etc. Proponents argue this is not mere memorization—the model generalizes to novel, compositionally complex prompts that never appeared in training, suggesting a form of meta-learning.
4.3 In-Context Learning as a Reasoning Engine
Large NTP models perform few-shot learning purely through forward passes. By conditioning on a prompt that includes examples, instructions, and chain-of-thought demonstrations, the model can solve tasks it was never explicitly trained for. This mirrors the flexibility of a general intelligence that adapts to new problems on the fly.
4.4 The “Predicting the Thought Process” Argument
Some researchers (e.g., Ilya Sutskever) have argued that if you could train a next-token predictor on all the text ever written, including internal monologues and step-by-step reasoning of humans, the model would be forced to replicate the cognitive steps that generated those texts. Predicting the next token of a detailed reasoning trace is equivalent to doing the reasoning yourself. Extrapolating, a sufficiently good NTP model captures the essence of thought.
4.5 “Compression Equals Intelligence” Formalisms
In algorithmic information theory, intelligence can be framed as the ability to compress patterns. A perfect next-token predictor is a lossless compressor of the data stream (via arithmetic coding). If the training data embodies all human intellectual output, then compressing it optimally implies a model that has discovered all discoverable regularities—potentially yielding an AGI.
5. The Counterarguments (Why NTP Alone Is Not AGI)
While the above points are intriguing, many experts are skeptical:
Gap between prediction and agency: NTP models lack persistent goals, memory across sessions, and an intrinsic reward system. They don’t act in the world or learn continuously from interaction.
Shallow understanding: The model can produce convincing-sounding text without grounding in sensory experience or causal intervention. It may be a “stochastic parrot” that blends patterns without true comprehension.
Brittle reasoning: NTP models are prone to hallucinations, logical lapses, and sensitivity to prompt phrasing—failures that suggest pattern matching rather than robust, system-2 reasoning.
Lack of embodiment: Human intelligence is deeply entwined with sensorimotor experience, which pure text prediction cannot replicate.
Variance in definition of AGI: The term AGI itself is contested; many would require continuous learning, autonomy, and generalization far beyond what current NTP systems exhibit.
Conclusion
Next Token Prediction is a deceptively simple objective that has driven a paradigm shift in AI. Technically, it is a maximum-likelihood training of gigantic Transformer decoders on internet-scale text, coupled with autoregressive sampling. In practice, it produces models that display a staggering breadth of capabilities, fueling the argument that a sufficiently scaled and well-trained NTP system might be an early form of AGI. However, the debate remains open: while NTP may be a necessary component of a general intelligence, most researchers agree that additional ingredients—embodiment, agency, continual memory, and robust world interaction—are still missing.
![DAGGER[X]s avatar](https://static.flashback.org/img/avatar/3/12696.jpg)
