New Benchmark and Metric for Long Term AI Event Forecasting

Predicting the next step isn't all that difficult. It is far more challenging to foresee what will happen in a week or a month. This very distinction became the starting point for a study conducted by scientists at Sber's Center for Practical Artificial Intelligence.

Limitations of Current AI in Sequential Event Prediction

The Next Step Is Not Yet a Forecast

Every day, people leave digital footprints: paying for purchases, visiting websites, or booking doctor appointments. All of this forms sequences governed by their own logic. Modern AI systems are quite adept at guessing a single next action – for instance, that a person who bought a laptop will soon buy a mouse. However, business and medicine need something else: understanding not just what will happen, but when, while forecasting an entire chain of events rather than a single one.

The problem is that previously, researchers lacked a unified method to verify how well a particular model could construct such long-term forecasts. Every team measured quality in its own way, making it nearly impossible to compare results.

HoTPP Benchmark and T-mAP Metric for Model Evaluation

A Yardstick That Didn't Exist Before

To remedy this, Sber researchers developed a benchmark – a standardized set of tests – called HoTPP (Horizon Temporal Point Process). This is an open platform: any team in the world can use it to test their model under unified rules.

The platform works with data from various fields: finance, e-commerce, and medicine. This is crucial, as an effective forecasting tool should not be limited to a single narrow niche.

Along with the benchmark, the authors proposed a new metric – T-mAP (Temporal mean Average Precision). Simply put, it evaluates a forecast based on two parameters simultaneously: whether the model correctly identified the type of event and whether it accurately guessed the time of its occurrence. Previously, these aspects were often evaluated separately, providing an incomplete picture.

Efficiency of Simple Statistical Methods vs Complex Neural Networks

More Complex Doesn't Mean Better

One of the study's most interesting results served as a sort of warning for the entire industry. It turned out that in long-term forecasting tasks, complex neural network models sometimes perform no better than simple statistical methods. In other words, increasing the number of parameters and complicating the architecture does not solve the problem by itself.

Another issue identified by the researchers is the so-called «collapse» of predictions. Complex models occasionally begin to output repetitive forecasts, ignoring rare but significant events. This is akin to a weather forecaster who promises «cloudy, no precipitation» every day: formally, they will be right in most cases, but they will miss critical weather anomalies.

As Andrey Savchenko, the center's Scientific Director, noted:

"Our benchmark and metric allow for an objective assessment of which AI model truly 'sees' the future and which merely makes a lucky guess at the next step. It is particularly important that we identified the problem of prediction 'collapse': complex models sometimes produce uniform forecasts, ignoring rare events. This discovery sets the stage for new research."

An additional result was a significant boost in computing speed: algorithmic optimization allowed for training and model operation to be accelerated tenfold. This is a vital practical bonus: researchers will be able to conduct experiments faster, and companies will receive results more promptly.

Applications of Temporal Point Process in Finance and Medicine

Where This Will Be Useful

The applications for such tools are quite diverse. Banks and fintech companies will be able to more accurately predict when and what transactions customers will make. Retailers and logisticians can plan inventory more effectively by understanding not just demand, but its temporal structure. In healthcare, analyzing sequences of doctor visits will aid in the early diagnosis of diseases.

The paper based on the study's results has been accepted for publication in Neurocomputing – a prestigious journal in the field of neural networks, ranked in the first quartile (Q1) of scientific journals in its domain.

The authors hope that HoTPP will become a global standard for researchers – a tool that enables progress toward creating AI capable of truly understanding the uncertainty and complexity of the real world, rather than just guessing the next obvious event.