I've been building a small language model from scratch. It has about 16.8 million parameters: six transformer layers and a vocabulary of 16,000 tokens. I pretrained it on my Mac using TinyStories, a dataset of simple children's stories, and it saw about 200 million tokens of them. Recently I started teaching it to answer questions. Part of that was deciding how it should respond when it can't answer. In my training data, I wrote the reply as one line:
ABSTAIN = "The story doesn't say."My setup makes this easy. I generate short stories ("Lily had a red ball. Lily put the ball in the box."), and every question is about a story I wrote. If the answer isn't in the story, the model should say so. I wrote every fact, so I know which questions can be answered.
When I tried to imagine the same step for a real model, the easy part went away. That raised the question this essay is about, and it's what got me reading the research.
Two teachers who disagree
To fine-tune a model to say "I don't know," someone has to choose which questions get that reply. The intuitive choice is questions that seem obscure or unanswerable. But whether a model knows something isn't a property of the question. It depends on what that particular model saw during pretraining, and the person writing the fine-tuning data can't know that for sure.
Suppose the model did learn the answer during pretraining, and during SFT I teach it to say "I don't know" to that question. The model gets two signals that disagree. The answer is in its weights, and the training target tells it to deny it. I think the model will get confused about what to answer.
"I don't know" is just one example of a wider problem. Say a model sees answer X to a question during pretraining, then sees a completely different answer Y to the same question during fine-tuning. My guess is that it won't cleanly replace X with Y. It will waver between the two, and sometimes it will answer with neither.
The closest evidence I found is Gekhman et al. (EMNLP 2024). They fine-tuned models on facts the models didn't already know. Those examples were learned much more slowly than ones that matched what the model knew. Once the model did learn them, its tendency to hallucinate rose roughly linearly. The authors concluded that models get their factual knowledge mostly from pretraining, and that fine-tuning mostly teaches them how to use it.
Their study is about new facts, not contradictory ones, so it doesn't test my X-versus-Y case directly. But it points the same way. When the fine-tuning target disagrees with what is already in the weights, the model doesn't simply learn the target. It learns something about how to behave when it isn't sure, and that behavior can be guessing.
If that's correct, the label has to come from the model, not only from the question. R-Tuning (Zhang et al., NAACL 2024) does this. It first compares the model's pretrained knowledge with the instruction-tuning data to find the questions the model is uncertain about, and builds refusal training data from those. In that sense, "I don't know" isn't a fixed fact about a question. It's a statement about one particular model, so it has to be measured for each model.
There's reason to think the model can be asked. In Language Models (Mostly) Know What They Know (Kadavath et al., 2022), models were trained to predict whether they could answer a question correctly, and they did this reasonably well. The word "mostly" in the title matters, though. The ability only partly carried over to new kinds of tasks. So the measurement is possible, but it isn't reliable everywhere.
A bare "I don't know" says too little
Suppose the labels were perfect. A TAIRC article on testing whether LLMs can admit they don't know convinced me that would still not be enough. A model that never answers never hallucinates, so a refusal can avoid a false answer and still fail the person who asked. A model might not know for several different reasons:
- It doesn't have enough data to answer.
- The question is too vague to answer in the first place.
- It could answer if it were given more information.
Each one tells the user something different: look elsewhere, rephrase, or supply the missing source. The article adds a case I hadn't thought of: the question rests on a false premise, and the right reply corrects the premise rather than declining. A reply that doesn't say which case applies leaves the user to guess.
The research literature makes a similar split. The survey Know Your Limits (Wen et al., TACL 2025) classifies abstention by whether the cause is in the query, in the model, or in human values. A vague question is a problem with the query. Missing knowledge is a problem with the model. Kadavath et al. also found that a model's estimate of whether it can answer rises when relevant source material is added to the prompt. That is the third case, and it can be measured.
My toy setup gets one part of this right by accident. "The story doesn't say" is a claim about a document the user can read and check. It says where the answer should have been. "I don't know" about the world doesn't point anywhere.
The incentive to guess
There's also a reason labs might not push hard on any of this. Kalai et al. (2025) argue that hallucination persists partly because of how models are scored. Most benchmarks give a point for a correct answer and nothing for either a wrong answer or "I don't know." Admitting uncertainty never earns anything, while a guess sometimes does, so models end up trained to guess. A model that learned when to abstain would rank lower on the leaderboards that decide which model looks best.
Why this is harder for a small model
I think models with limited parameters are much less capable than large ones. Large models pick up skills and behaviors partly because they have more parameters and see more data during training. A small model has less room, and 16.8 million parameters is not much. You have to be very careful about which data you choose and which skills you teach, or the model might catastrophically forget something it already had. When I fine-tuned mine on the question-answering data, I mixed stories in alongside it so the model wouldn't forget its primary task, which was storytelling.
That raises the cost of a wrong "I don't know" label for a small model. It can't absorb one bad lesson among millions of good ones. And the easiest way for any model to satisfy an abstention objective is to abstain on everything. That model would never be wrong, and it would never be useful.
What comes next
I haven't run these experiments yet. My small model is a good place to try them, for the same reason the abstention label was easy to write: I know exactly what data it was pretrained on. I can check what it knows before deciding what it should refuse, and I can watch what happens when I teach it an answer that contradicts what it learned in pretraining. I'll publish what I find in a follow-up post.
Until then, I keep coming back to the question in the title. Whoever writes the "I don't know" labels is making a claim about the inside of a model they can't see into.