You know what an elephant and a neural network have in common? Both allegedly never forget. The only difference is that the elephant can at least pretend he forgot that whole peanut incident. But a neural network? It remembers everything. Absolutely everything. And that, oddly enough, is a huge problem.
When I first encountered the concept of «machine unlearning», my first reaction was something like: «Seriously? We spent years teaching these things to remember, and now we want them to forget»? But the deeper you dig, the clearer it becomes: the ability to forget is not a bug, it's a feature. And a critically important one at that.
Why Perfect Memory Is a Problem for AI
Why Perfect Memory Is a Problem
Imagine you remember absolutely every second of your life. Every conversation, every ad you glimpsed, every password someone accidentally typed next to you. Sounds like a superpower? In practice, it's hell.
Neural networks face the same problem, only on an industrial scale. GPT-4 was trained on terabytes of text. And hidden somewhere in those terabytes could be anything: people's personal data, medical records, confidential company information that accidentally ended up in the public domain. Credit card numbers, addresses, children's names – all of this could well have ended up in the training dataset.
And the model remembered it. Forever. Or almost forever.
That's where the fun begins. 🎭
GDPR and the Right to Be Forgotten
In Europe, there is a wonderful thing called GDPR – General Data Protection Regulation. Among other things, it gives people the «right to be forgotten». That is, if you want a company to delete your data, they are obligated to do so.
Sounds simple, right? Except, how do you delete data from an already trained neural network?
The classic approach: retrain the model from scratch, excluding that data. Great! Now take GPT-4. Training costs – tens of millions of dollars. Training time – months. And you're suggesting doing this every time someone in Barcelona decides to exercise their right to be forgotten?
I live in Barcelona. Trust me, people here really love their rights. 😏
Retraining the model every time is like demolishing the whole house just to replace one lightbulb. Technically possible, economically – absurd.
How Neural Networks Actually «Remember»
To understand how to make a neural network forget, you first need to figure out how it remembers in the first place.
Spoiler alert: a neural network doesn't store data like a database. It doesn't write down «Juan Garcia lives at 42 Carrer de Balmes». Instead, the information is smeared across billions of weights – numerical parameters that determine how the model reacts to input.
It's as if your memories weren't photos in an album, but changes in the structure of your brain's neurons. You can't just cut out one memory without affecting the others. They are all intertwined.
When a neural network learns, it adjusts its weights to give the correct answers based on the training data. If the data contained the phrase «The best paella in Barcelona is at Juan's», the model will tweak its weights so that mentioning Barcelona and paella triggers a connection to the name Juan.
The problem is that this information doesn't reside in a single weight. It is distributed across thousands, if not millions, of parameters. Finding and neatly removing it is a task harder than finding a needle in a haystack. It's like finding one specific atom inside that needle.
How Neural Networks Actually "Remember" Information
Machine Unlearning: The Art of Selective Amnesia
Researchers have come up with several approaches to the problem. And they all try to answer one question: how do we make the model behave as if it never saw specific data, but without retraining it from scratch?
Approach One: Reverse Fine-Tuning
The most obvious method is to take the data that needs to be forgotten and train the model to make mistakes on it. Literally. If the model answers the question about Juan's address correctly, we penalize it for that. We rub its nose in it: «No! Bad neural network! Don't know this»!
Sounds like bullying AI, and in a sense, it is. But the method works. Partially.
The problem is that if you overdo it, the model will forget not only about Juan. It might forget what Barcelona is, what paella is, and start generating complete gibberish. It's like trying to erase a single word from a book and accidentally smudging ink across the whole page.
Approach Two: Influence via Influence Functions
Here we dive into math. There is this thing called influence functions. They allow you to estimate how much a specific data point influenced the final model.
Imagine you were baking a cake, adding ingredients one by one. Influence functions are a way to understand how much each spoonful of flour affected the final taste. Knowing this, you can try to «subtract» the influence of a specific ingredient.
Sounds beautiful. In practice, calculating influence functions for a model with billions of parameters requires computing power comparable to the GDP of a small European country. And the results are still approximate.
Approach Three: Modular Architecture
What if we designed the model from the get-go to make forgetting easier? Instead of one giant neural network, we have a bunch of small modules, each with its own area of knowledge.
Need to forget medical data? Throw out the medical module, train a new one without the unwanted data, and plug it back in. Like replacing one Lego piece without taking apart the whole construction.
The problem? Such models usually perform worse than monolithic ones. It's the eternal trade-off between performance and flexibility. Want the best quality – you get a monolith that's impossible to edit. Want flexibility – you sacrifice accuracy.
Machine Unlearning: The Art of Selective Amnesia
Why Else We Need Forgetting (Besides the Law)
Okay, we sorted out GDPR. But there are other reasons why it is vital for neural networks to know how to forget.
Outdated Information
The world changes. What was true yesterday might be a museum exhibit today. Presidents change, companies close, scientific theories turn into pumpkins.
If a model is trained on data up to 2020, it doesn't know about the COVID-19 pandemic. If up to 2022 – it has no clue about the explosive popularity of ChatGPT. Data becomes obsolete, and the model must be able to update it – and for that, sometimes you need to forget the old stuff first.
Otherwise, you get cognitive dissonance: the model simultaneously «knows» that Twitter is called Twitter, and that it is also X. And it starts to hallucinate, trying to combine the incompatible.
Malicious Data
Imagine someone deliberately slipped harmful data into the training set. Data poisoning is a real threat. Attackers can inject bias, backdoors, or just pure nonsense.
Detected the problem? Great. Now you need to get rid of the consequences. And again: retraining the entire model is expensive and long. The ability to surgically remove the influence of specific data could save millions of euros and months of work.
Ethical Considerations
Sometimes data was obtained unethically. For example, image generation models were trained on millions of pictures from the internet – and artists aren't thrilled that their work was used without permission.
If an artist says, «I don't want my art to be used for AI training», it makes sense to provide a mechanism to remove that data from the model. This isn't just a legal issue, but a question of respect for content creators.
Why Else We Need AI Forgetting (Beyond the Law)
The Problem: What If It Forgets Too Much?
This is where it gets interesting. Forgetting is an operation that is difficult to control. When you adjust the model's weights so it forgets Juan's address, you might accidentally hit its knowledge about the geography of Barcelona, Spanish cuisine, or the principle of how addresses work in general.
This is called «catastrophic forgetting». And it can turn a smart model into a white noise generator.
Imagine teaching a neural network to play chess, and then deciding to teach it checkers. After learning checkers, it might completely forget the rules of chess. The human brain handles this better – we know how to learn new things without forgetting the old (well, theoretically). Neural networks can't do that. Yet.
There are techniques that help mitigate this problem. For example, elastic weight consolidation – a method that identifies which weights are important for old tasks and protects them from changes when learning new ones. But it's still a compromise: the more you protect old knowledge, the harder it is to add new stuff.
The Problem: What If AI Forgets Too Much?
Practical Experiments: How It Works in Reality
Researchers from various universities have conducted experiments on removing data from models. The results... let's just say, are mixed.
In one study, they tried to remove information about a specific person from a language model. After applying machine unlearning techniques, the model indeed stopped generating facts about this person. Success?
Not quite. It turned out that if you rephrase the question or come at it from a different angle, the model still coughs up bits of information. It's like a person with partial amnesia – they don't remember the event directly, but if you ask a trickier question, the memories start surfacing.
This is a fundamental problem: information in a neural network is distributed so widely that deleting it completely without breaking the model might simply be impossible. We can make it so it doesn't output this information explicitly, but we can't guarantee it has disappeared from all the weights.
Practical Experiments: How AI Forgetting Works in Reality
Alternative: What If We Don't Forget, But Don't Remember?
Some researchers suggest a different approach: let's just not remember what isn't needed in the first place. Differential privacy is a set of techniques that add noise to the training process so that the model learns general patterns but doesn't memorize specific examples.
It's like teaching a child traffic rules without showing them specific license plates. They understand they need to look both ways, but they don't memorize that a silver Seat with license plate 1234-ABC drove by at 15:32.
Sounds good, but there are nuances. Firstly, adding noise degrades the quality of the model. Secondly, this needs to be planned in advance – you can't retrofit differential privacy into an already trained model.
Plus, there is a fundamental paradox: for a model to be useful, it must learn from data. But the more it learns, the higher the risk it will remember something unnecessary. It's like trying to learn how to cook without tasting the food – theoretically possible, but the result is questionable.
Alternative: What If We Don't Forget, But Don't Remember?
The Future: Self-Learning Models with Selective Memory
What's next? Researchers are working on models that can update their knowledge in real time, forgetting the obsolete and incorrect, but keeping the important and relevant.
Imagine a neural network that works like the human brain: actively forgets details but remembers the essence. You don't remember every letter of the article you read yesterday, but you remember the main idea. Models could work the same way – keeping abstractions and patterns while discarding the specifics.
There is a direction called continual learning. Models that can learn constantly, adapting to new information without forgetting the old. So far, the results are modest, but there is progress.
Another direction is federated learning. The model trains on user devices without transmitting data to a server. If a user decides to delete their data, it gets deleted locally, and the model simply stops accounting for that user's contribution. Elegant, but not suitable for all tasks.
The Future: Self-Learning Models with Selective Memory
Philosophical Question: Should AI Even Forget?
Here we step outside technology and into philosophy. Human memory is imperfect – we forget, distort memories, reconstruct the past. And this is not a bug, but a feature of the psyche.
Forgetting helps us move on. To forgive. Not to fixate on mistakes. If we remembered every awkward situation with perfect accuracy, life would be unbearable.
Maybe AI needs this ability too? Not just to comply with GDPR, but to be more... human? To be able to let go of the unimportant and focus on what matters?
Or conversely – is AI's perfect memory an advantage that shouldn't be lost? After all, computers were created to do what people do poorly – remember accurately and forever.
I don't have the answer. And that's okay. We are creating tools whose consequences we don't fully understand yet. The history of humanity, basically.
Philosophical Question: Should AI Even Forget?
Practical Tips: What to Do Now
While researchers are struggling to find the ideal forgetting algorithm, developers have to work with what they have. A few practical recommendations:
Document your training data. If you don't know what the model was trained on, you won't be able to remove anything from it. Keep track of data sources, dataset versions, and metadata.
Design with forgetting in mind. If your system must comply with GDPR or similar requirements, build in the ability to delete data at the architectural stage. Modularity, model versioning, fast retraining on a subset of data.
Use differential privacy during training. Yes, it lowers quality. But sometimes it's better to have a slightly less accurate model that doesn't memorize personal data than a super-accurate one that has to be retrained every week.
Be transparent with users. If the model can memorize their data, say so. If deleting data will take time, explain why. People are more tolerant when they understand the constraints you are working with.
Prepare for stricter laws. GDPR is just the beginning. Other regions are introducing their own privacy requirements. The ability to quickly adapt models to new rules will become a competitive advantage.
Practical Tips: What to Do Now
Conclusion: Memory and Oblivion in the Age of AI
We live in paradoxical times. We create systems with perfect memory and simultaneously look for ways to teach them to forget. We build increasingly complex models and realize we can no longer fully control them.
The ability to forget is not a flaw in neural networks, but a necessary function that needs to be developed. And the sooner we do this, the better – for users, developers, and society.
Because an AI that remembers everything but doesn't know how to forget is not a smart AI. It's a legal bomb, an ethical dilemma, and a technical migraine rolled into one.
So yes, neural networks can forget. They must forget. And we are learning this – through trial and error, generously seasoned with math and philosophy.
In the meantime, I'm off to forget where I left my apartment keys. Some things humans do better than machines. 🔑
