Idea #1
Naive fine-tuning — simply training a pre-trained model on a private corpus with next-token prediction — does not produce a model that can generalize over that corpus; it produces a model that memorizes and then collapses.
The intuition that you can ‘just fine-tune’ a model on, say, 10,000 internal financial reports and get a model that usefully ‘knows’ those documents is wrong in a subtle way. The model can be trained until loss approaches 0.0001 — apparently near-perfect — and yet when you sample from the resulting model or try to query it, generation collapses.
The model can only reproduce content that was literally verbatim in the training data, question and answer both. It cannot answer a question whose framing appears nowhere in the training corpus, even if the answer is clearly implied by the documents. The reason is a classical machine learning one: next-token prediction on a narrow, fixed dataset has no mechanism for building the generalizable internal representations that come from pre-training on diverse data.
Normal LLM pre-training works because the diversity of the corpus forces the model to build representations that transfer; training on a small homogenous corpus just overfits. Practitioners should treat this as a hard disqualification rather than a tuning problem: if the goal is ‘the model should answer novel questions about D,’ naive fine-tuning on D is not the right method, and adding more epochs or lowering the learning rate will not fix it.
Idea #2
The central limitation of current LLMs is not capability but personalization: models are trained on public data and therefore cannot, by construction, acquire depth in private or personal domains — and scaling (more data, more compute, larger models) only widens the breadth-vs-depth gap rather than closing it.
The three axes of scaling that have driven the deep learning revolution — more training data, more compute, larger models — are all constrained to publicly available information. This means every successive generation of frontier models gets better at things that are documented publicly (Wikipedia, GitHub, arXiv, Reddit) but makes zero progress on knowing anything about you, your organization, your private correspondence, or your domain-specific long-tail knowledge.
The breadth a model acquires is genuinely impressive: it can bridge connections across all public mathematical literature in a way no individual human could. But it lacks the kind of subconscious intuition that a graduate student builds after five years of focused practice in a narrow area — because that intuition is not encoded in any public text.
This matters practically for three distinct failure modes: (1) models have a training cutoff and cannot learn what happened since; (2) they are weak at long-tail skills that rarely appear in public data (e.g., writing AMD GPU kernels); (3) they know nothing about the private context of any user’s life, work, or writing style. The framing to internalize is: scaling is powerful but it is only ever applied to public data, so an AI that knows everything about the world in general will always know nothing about your company’s internal partnership agreements or the way you personally like to write.
Idea #3
Every approach for injecting private context into model weights eventually hits a ‘synthetic data wall’ — a natural saturation ceiling — because any fixed corpus D has a bounded amount of information that can be extracted from it in a single training pass. Breaking through requires recursive self-improvement, not just more compute.
The landscape of approaches for the problem (compress D into KV caches; use on-policy distillation so the model behaves as if D is in context; generate Q&A pairs conditioned on D then train on them à la the Cartridges ‘self-study’ technique; simulate continued pre-training with D-conditioned synthetic data) are all genuinely different in mechanism but share the same structural problem: they define a derived dataset from D, train the model on it, and then the model saturates.
This is not a bug in any specific technique; it is a classical ML result — once the model is no longer underparameterized relative to the synthetic dataset, it stops learning. The model will have acquired some knowledge of D but not everything, and adding more compute at that point yields no further improvement.
The only known way to escape it is the mechanism that made AlphaGo work: self-improvement, where getting better at the task causes the training signal itself to become harder, so the model is always learning against a target it cannot yet fully match. For compute-on-context, this means the synthetic data generation step must be conditioned on the current model’s capabilities, so that as the model improves it generates harder and richer training questions from the same D — converting what was a fixed ceiling into a continuing gradient. Without this recursive loop, ‘scaling compute on context’ is not actually a scaling axis; it is just a one-shot knowledge injection with diminishing returns.
Source: Scaling Compute on Context — Jack Morris, Engram (AI Engineer)
Text summarized and optimized using Anthropic’s models and reviewed by a human.