When it comes to AI progress, we usually hear about benchmarks: tests for reading comprehension, solving math problems, or generating code. But there is a problem – these metrics don't always show how useful a model is in real work. One can excel at academic assignments while «spinning their wheels» on practical tasks that people solve every day.
Anthropic decided to approach AI evaluation from a different angle. The company launched the Anthropic Economic Index – an index that measures models' ability to perform specific work tasks, precisely the ones for which money is exchanged in the economy.
Что такое экономические примитивы как базовые действия ИИ
What Are Economic Primitives?
The essence of the approach is to break down complex work into basic actions – «economic primitives». These are the elementary tasks that constitute almost any profession: writing an email, analyzing data in a spreadsheet, finding information in a document, creating an action plan.
Anthropic identified eight such primitives and created tests for each. Simply put, instead of abstract questions like «does the model understand context», the company checks: can the AI, for example, read a long contract and extract key terms from it? Or take a spreadsheet with data and build a forecast based on it?
These are not just theoretical exercises. Each primitive corresponds to real actions people perform at work – from a data analyst to a project manager.
Восемь базовых навыков ИИ: описание и применение
Eight Basic Skills
Here are the tasks included in the index:
- Information Retrieval – finding necessary data in a large volume of text, like a corporate knowledge base or a set of documents.
- Classification – sorting information into categories: for example, categorizing customer inquiries by problem type.
- Data Analysis – working with spreadsheets: cleaning, aggregation, finding patterns.
- Summarization – condensing a large document into a brief summary without losing its essence.
- Planning – creating a sequence of actions to achieve a goal.
- Content Generation – writing text according to specified requirements: from an email to a report.
- Editing and Refinement – improving existing text: fixing errors, changing the tone, adding details.
- Coding – writing or debugging software code.
Each of these tasks is found in dozens of professions. And if a model handles them reliably, it means it can genuinely alleviate human workload or take over part of the routine.
Как измеряются навыки ИИ моделью Anthropic
How Is It Measured?
For each primitive, Anthropic assembled a set of tests that mimic real work situations. For instance, in an information retrieval task, the model receives a stack of documents and must quickly find the answer to a specific question. In a data analysis task – a spreadsheet with raw numbers and a request to generate statistics.
An important point: the tests are designed to reflect not just answer accuracy, but also reliability. If a model handles a task in 95% of cases, that's one thing. If it's 60% – that's quite another, because in real work, instability becomes a problem.
The index shows the level at which the model performs for each of the eight skills. This allows one to see not only general progress but also specific strengths and weaknesses.
Зачем необходим Anthropic Economic Index
Why Is This Needed?
Standard benchmarks help developers improve models, but they say little about what these models can do in business or productive work. The Anthropic Economic Index solves a different problem: it shows which real-world functions AI is already capable of performing, and where further development is still needed.
This is useful for companies adopting AI. Instead of vague promises, one can check: will the model handle processing customer requests? Can it help analysts with data preprocessing? Is it reliable enough to automate part of the document workflow?
This approach helps developers too. If it is evident that the model excels at text generation but «stalls» on other tasks, this provides insight into where to direct efforts.
Первые результаты тестирования моделей ИИ
What Do the First Results Show?
In the first report, Anthropic tested its Claude 3.7 Sonnet model. The results show that the model handles most primitives at a high level, but there are tasks where performance is lower.
For example, information retrieval and summarization tasks are performed consistently well – these are areas where language models have long shown strong results. However, planning and data analysis require more complex reasoning, and there is room for growth there.
It is important that the index will be updated regularly. Anthropic plans to track progress and show how models improve in performing real tasks. This is not a one-time snapshot, but a dynamic picture of development.
Ограничения подхода к оценке экономического индекса ИИ
Limitations of the Approach
It is clear that any index is a simplification. Real work is more complex than a set of isolated tasks. People often combine several skills simultaneously, work under conditions of uncertainty, and make decisions based on incomplete information.
Economic primitives do not cover the full diversity of professions. There are tasks that require creativity, empathy, the ability to read between the lines, or negotiation skills. These things are harder to formalize.
But even considering these limitations, the index provides a useful reference point. It shows the model's basic capacity to perform specific actions that make up a significant part of working time in many professions.
Куда движется развитие ИИ: выводы из нового индекса
Where Is This Leading?
The appearance of such an index signals a shift in focus within the AI industry. Previously, main attention was paid to making models smarter, faster, and better at tests. Now there is increasing interest in making them more useful – so they can be genuinely integrated into workflows.
The Anthropic Economic Index is an attempt to shift the conversation about AI from the language of technology to the language of economics and practical utility. If a model can take over part of routine tasks, it frees up time for people for more complex and creative work. If it does this reliably, it reduces costs and speeds up processes.
For now, this is the first step. Let's see how the index develops and what changes it shows in the coming months. But the very idea of measuring AI progress through the prism of real tasks looks logical and timely.