Generalizing Generalization: When Neural Networks Learn to Predict – But Not What We Expected

Picture this: you're cramming for a math exam, solving problems from your textbook. You show up to the test – and the problems are completely different. But you nail it! That means you actually understood the math and didn't just memorize the examples, right?

Now imagine that on the next exam – same math, but with real-life word problems – you fail miserably. Question: did you really understand math, or did you just learn how to solve a specific type of problem?

This is exactly the issue large language models face. And this is exactly what a new study is about, asking the uncomfortable question: if a neural network generalizes well on one type of new data, does that mean it will generalize everywhere?

Spoiler alert: nope. And that’s a serious problem.

What is generalization and why does it matter more than you think?

When we talk about generalization in machine learning, we mean the model's ability to work with data it has never seen before. It’s the difference between a student who memorized the answer key and one who truly grasped the material and can apply that knowledge to any situation.

In the world of AI, this is called OOD generalization (out-of-distribution generalization) – the ability of a model to handle data that differs from its training set. And this isn't just an academic exercise. When you release a language model into the real world, it encounters all sorts of text: from formal legal documents to social media slang, from scientific papers to poetry. If the model can't generalize – it's useless 🤷‍♀️

But here’s the kicker: most studies check generalization on just one test set. That’s like checking a person’s driving skills only in an empty parking lot, and then unleashing them onto the highway during rush hour. It might work. Or it might... not.

Cat and mouse with neural networks: the history of generalization tests

The history of testing language models looks like an endless game of tag. Model creators think: «Aha! Our model handles this dataset with 95% accuracy! We won!» Then researchers create a new dataset – and the model faceplants again.

Here are a few examples of these «traps»:

HANS – a dataset specifically designed to reveal when models use primitive heuristics instead of real understanding. For example, if the premise contains the word «not», the model might automatically decide the hypothesis contradicts it, without even analyzing the meaning.

ANLI – a dataset collected in several rounds, where each round specifically targets the model's weaknesses. It’s like a coach who constantly finds new ways to test your stamina 💪

Synthetic datasets – artificially created examples that look like training data but contain tricky modifications.

Every new dataset shows: models have learned to handle specific tests, but haven't necessarily learned to understand language.

The Experiment: what if we check multiple datasets at once?

Researchers decided to run a simple but crucial experiment. Instead of evaluating the model on a single OOD dataset, they took seven different ones and tracked how the model handled each of them throughout the entire fine-tuning process.

The task was a classic one: NLI (Natural Language Inference) – determining logical relationships between two sentences. You get a premise and a hypothesis, and the model has to say: does the hypothesis follow from the premise (entailment), contradict it, or is it neutral?

For example:

• Premise: «A cat is sitting on the window».
• Hypothesis: «An animal is indoors.»
• Answer: «Entailment»

Or:

• Premise: «All students passed the exam.»
• Hypothesis: «Some students failed the exam.»
• Answer: «Contradiction»

Sounds simple, but this task requires logical reasoning and context understanding – which is exactly why it’s often used to check a model's ability to generalize.

Which models were tested?

Two families of language models participated in the experiment:

OPT – open models of various sizes created by Meta OLMo2 – modern models known for their training stability

The models were fine-tuned on tiny datasets (just 32, 64, or 128 examples!) using the LoRA method – an efficient technique that allows tuning large models without fully retraining all parameters.

The Datasets: seven ways to check generalization

Training was conducted on standard datasets SNLI (570k sentence pairs describing images) and MNLI (433k examples from ten different text genres).

But to check generalization, seven different datasets were used:

Standard:

• SciTail – scientific questions and answers
• WNLI – coreference resolution tasks (understanding what pronouns refer to)
• RTE – a classic textual entailment dataset

Adversarial (created specifically to expose weaknesses):

• HANS – traps based on simple heuristics
• PAWS – paraphrase traps (sentences with similar words but different meanings)
• ANLI – an advanced dataset with examples collected in multiple rounds

Each dataset tests different aspects of language comprehension. If the model really learned to generalize – it should handle them all. If not – we’ll see failures.

The Method: how to measure «pure» generalization

Here’s where it gets interesting. The problem is that when a model learns on training data, it simultaneously improves two metrics:

1. Quality on training data (this is natural)
2. Quality on OOD data (this is the actual generalization)

But how do we figure out how well the model actually generalizes, rather than just getting better overall?

The researchers used a clever trick – partial correlation. It works like this:

Step 1: At each stage of training, the model's quality is recorded on the training set and on all seven OOD datasets.

Step 2: A regression model is built to predict OOD quality based on training quality. Basically, it answers the question: «If the model scored X on training, what score do we expect on the OOD test»?

Step 3: The residuals are calculated – the difference between the real and expected quality. This is the «pure» generalization, unrelated to the model's general improvement.

Step 4: The correlation between the residuals of different OOD datasets is measured.

If two OOD datasets measure the same generalization ability, their residuals should correlate: when the model handles one better (above expectation), it should handle the other better too.

But if correlations are low or even negative – it means the datasets are measuring different things, and success on one doesn't guarantee success on the other.

The Results: welcome to the chaos 🎭

Brace yourselves: the results were... unexpected.

Discovery #1: Models generalize, but selectively

The good news: almost all models demonstrated the ability to generalize on at least some OOD datasets. Meaning, they weren't just memorizing examples, but truly extracting some patterns.

The bad news: no model showed universal generalization. For instance:

• OPT-30B, trained on MNLI, handled the MNLI test perfectly but failed miserably on PAWS
• OLMo2-32B in the SNLI experiment showed a wild scatter: on some datasets quality went up, on others – it went down

It’s like a student who brilliantly solves algebra problems but gets lost the moment they see geometry.

Discovery #2: Training is unstable

When analyzing how model quality changed during fine-tuning, it turned out that:

OPT showed sharp fluctuations – quality on OOD datasets would rise and fall unpredictably. This aligns with observations from previous studies: fine-tuning can both improve generalization and harm it.

OLMo2 trained much more stably (as expected from this model family), but even there, different OOD datasets showed different trends. Somewhere quality grew, somewhere it stayed flat, and somewhere it declined.

Conclusion: you cannot judge a model's general ability to generalize based on a single test. That’s like assessing a person's health solely by their body temperature.

Discovery #3 (The Main One): Correlations are total chaos

And here is where it gets really interesting. After the researchers eliminated the influence of general model improvement and looked at the «pure» correlations between OOD datasets, the picture became even more confusing.

Partial correlations between datasets:

• Show no general pattern whatsoever
• Change radically from model to model
• Depend on what data the model was trained on
• Can be positive for one model and negative for another

Concrete examples:

Two OOD datasets might correlate strongly (positive link) for the OPT-13B model, but demonstrate a negative correlation for OLMo2-32B. That is, if the first model improves generalization on dataset A, it simultaneously improves it on dataset B. But the second model, while improving A, actually worsens B!

Model size doesn't save the day either. One might assume that larger models with more parameters generalize better and yield more consistent results. But the data doesn't back this up: average correlations don't grow with increased size, and sometimes even become more negative.

What does this mean practically? Generalization is not a universal property of a model or a task. It is a unique combination of a specific model, a specific data distribution, and a specific type of shift.

Why this matters: from the lab to the real world

Let’s go back to the exam analogy. Imagine you are a company developing a language model for real-world application. You want your model to work well with:

• Official documents
• Spoken language
• Technical texts
• Social media
• Scientific papers
• News
• Creative writing

Each of these text types is its own OOD shift relative to the training data. And here is what the study shows: your model's success on documents does not guarantee success on social media. Moreover, improvement on one type of text might even worsen performance on another!

It’s like an experienced driver from Seoul getting confused on the mountain roads of Gangneung – despite having excellent basic driving skills. It’s not about the skills themselves, but the specifics of the situation.

What does this mean for model evaluation?

Current evaluation practice – using one or two OOD datasets – is clearly insufficient. It’s like diagnosing a patient by only measuring their weight and ignoring blood pressure, blood tests, pulse, and everything else.

The study shows: to adequately assess a model's generalization ability, it is necessary to use multiple OOD tests covering different types of shifts. And even that doesn't guarantee the model will handle a completely new type of data you haven't tested.

The Technical Side: how it worked

For those who want to understand the details of the experiment:

Pattern-based training

Researchers used a special training format that looks like this:

{premise} Question: {hypothesis} Yes or No? 

The model learned to answer with the tokens «_Yes» or «_No». This approach is called pattern-based fine-tuning and avoids problems associated with adding a new classification head to the model. Instead, it uses the same language head (LM-head) that was already trained during pre-training.

Why is this important? Because adding new layers can lead to feature degradation: the model might «forget» what it learned earlier while adjusting to the new task. The pattern-based approach avoids this.

Computing Resources

The experiment required serious computing power:

• Different types of GPUs were used (A5000, A6000, A100)
• Total cost – about 5500 GPU-hours
• Models were trained with varying numbers of examples (32, 64, 128) to check the impact of training set size

This serves as a reminder that AI research requires not just good ideas, but significant resources 💻

Data Processing

An interesting detail: researchers removed examples with a neutral label (when the hypothesis neither follows from nor contradicts the premise) from all datasets. Why?

Because different datasets interpret neutrality differently. What is considered neutral in SNLI might be interpreted differently in HANS. Removing these examples made the comparison fairer and eliminated an additional source of noise in the data.

What’s next: where generalization research is heading

This study opens up more questions than it answers. And that’s good – that’s exactly how science progresses.

Unsolved Questions

Why are the correlations so different? What exactly in the architecture of OPT or OLMo2 causes the same datasets to behave differently? This could be related to the pre-training process, data distribution, architectural features, or something else entirely.

Do universal principles of generalization exist? Or is every combination of model and task unique? If it’s the latter – that’s a serious problem for deploying AI systems.

How to predict which OOD data a model will handle? Can we analyze a model and say in advance where it will generalize well and where it will fail?

Practical Takeaways

For developers and researchers, this study means:

Test on multiple datasets. One OOD test is an illusion of safety. You need a full checkup, not just a temperature reading.

Be skeptical of generalization claims. If a paper or technical documentation claims a model «generalizes well», ask: on exactly which datasets? How many? What types of shifts do they cover?

Prepare for the unexpected. Even if your model works great on all test datasets, the real world can still offer surprises. Production quality monitoring is mandatory.

Use ensemble approaches. If one model is good on one type of data and another on a different type, maybe it's worth combining them or using different models for different tasks.

Conclusion: generalization as a multi-faceted crystal

So, do generalization results generalize? No.

This doesn't mean language models can't generalize – they can. But generalization is not a single universal property that a model either has or doesn't. It is a complex, multi-faceted ability that manifests differently depending on:

• The model architecture
• The data it was trained on
• The type of data shift
• The specific task

Imagine generalization as a multi-faceted crystal: from one angle it shimmers with all the colors of the rainbow, from another it looks dull, and from a third – completely opaque. And for every model, this crystal is turned its own way.

For the machine learning research community, this means: more complex, comprehensive evaluation methods are needed. For companies deploying AI: rigorous testing on the most diverse data possible is required. For all of us: honesty regarding the limitations of modern models is essential.

Because code is poetry, just in another language. And good poetry requires understanding all its nuances, not just the first line 📚✨