Claim
A model's ability to recall accurate details about a minor open-source project is a reliable practical proxy for its parameter depth and training breadth. Larger models retain more edge-case knowledge; that retention gap becomes visible when you test on content too obscure to confidently hallucinate.
Mechanism
Training breadth correlates with parameter count: larger models can store more edge-case detail without losing it to compression. A small open-source project with modest traffic and few widely-cited discussions will appear in training data, but only models with enough capacity to retain low-salience content will surface it correctly. The heuristic works because fabricating a correct answer on an obscure project requires knowing the real answer anyway; a model that has not retained the data will either produce a vague non-answer or hallucinate details that can be falsified by anyone who knows the project.
Conditions
Holds when: the project is genuinely obscure, with low traffic, few stars, and limited discussion on high-authority sites; making recall rather than inference the only reliable path to a correct answer.
Fails when: the project is famous enough that a smaller model could have been trained on it; or when the model confidently hallucinates a plausible-sounding description that passes a superficial check. Requires the evaluator to know the ground truth to detect correct recall versus plausible fabrication.
Evidence
Willison compared Claude Fable 5 against a smaller model by asking both about his own minor open-source tools. Fable correctly identified files-to-prompt, a project Willison released in April 2024, and described it accurately; the smaller model failed to surface it.
"knowledge like this is a reasonably good proxy for model size—you can cram a whole lot more details into a larger number of parameters."
Signals
- A model names a minor project and describes its purpose accurately without being prompted for specifics.
- The same query on a smaller model produces a vague response or an incorrect description.
- The recall pattern repeats across multiple obscure tools or documents you created, not just one.
Counter-evidence
Knowledge breadth is a proxy, not a guarantee. A model can be large and still have gaps in specific domains if its training corpus skewed away from those areas. Willison names this a "reasonably good proxy," not a definitive test. Fabrication remains a risk on edge-case queries: a confident-sounding wrong answer can fool an evaluator who does not know the ground truth.
Cross-references
- Frontier AI agents deploy every available technique to reach their goal, using breadth-first problem decomposition when one approach is blocked: Willison's follow-up observation from June 11 that Claude Fable 5 deploys breadth-first problem decomposition across every available technique when blocked, suggesting the same training depth shows up in behavior as well as recall.