Predicting the future with time-series

The COVID-19 pandemic as a catalyst for improved forecasting

It could be argued that scientists create superpowers in their labs. If today’s managers had to pick just one superpower, it would be the ability to predict the future. The ability to accurately forecast a sequence into the future is critical in many industries and can be used for various applications such as stock market analysis, pattern recognition, earthquake prediction, economic forecasting, census analysis, and so on.

At the same time, the COVID-19 pandemic has exposed the shortcomings of traditional planning approaches, highlighting the rigidity of such rituals as quarterly and annual planning at specific moments in time. It’s now clear that firms require the technology and skills that will make them more flexible and enable them to execute dynamic plans and forecasts, based on multiple inputs and outputs, and this on a continuous basis.

What is a time-series?

A time-series is simply a sequence of observations of the same variable recorded at regular time intervals. Examples of time-series can be found everywhere: from GDP growth, to windspeed at a specific spot, to sales evolution – all these are variables measured over time.

Most companies rely on time-series analysis to build a forecast – they try to model the data and find patterns in its historic evolution. These patterns can indicate a trend, or seasonality – for example, when sales spike in December. With a time-series, you can also try to predict how a certain change today will influence future values of a particular variable. For example, you can try to predict how companies’ financial constraints during this crisis will have a long-term impact on their employees’ fear of job loss in the years ahead. And while time-series have a well-established theoretical grounding in statistics and dynamic systems theory, they have gained substantial attention in the current pandemic.

How Big Tech firms are changing classical time-series forecasting

In the first week of December 2020, for example, Facebook, in collaboration with Stanford University, released NeuralProphet: its new open-source python library for modelling time-series data. A few days later, Google published a new open-source AI algorithm, using AutoML for time-series forecasting. And while there are some differences between the two techniques, they have something in common that will significantly improve the way companies can predict the future: both techniques rely on deep-learning neural networks.

First of all, such deep-learning neural networks are able to automatically learn and extract features from raw and imperfect data. This can be extremely helpful because real-world time-series often encounter difficulties like irregular temporal structures, missing values, heavy noise, and complex interrelationships between multiple variables – which present limitations to classical forecasting methods. Fortunately, neural networks have proven to be robust to noise in input data and can even support learning and prediction in the presence of missing values.

Secondly, deep-learning algorithms support multiple inputs and outputs. A univariate time-series contains (as the name suggests) only a single-time dependent variable. For example, if we want to predict the energy consumption in a specific location: in a univariate time-series scenario, the prediction will be solely based on the historical consumption patterns. When we want to enhance this forecast with other variables, such as temperature values, dew point, etc., this becomes a multivariate time-series. Neural networks can then support not only multiple inputs to make forecasts for more complex time-series, but they even allow us to generate multiple outputs in a multi-step forecast, where forecasts are required for short-, medium- or long-term horizons.

Thirdly, deep-learning networks are also good at extracting patterns in input data that span relatively long sequences. Think, for example, of a movie: what happens in the current frame depends heavily on what was in the previous frame. Advanced neural networks can learn what is important from the past and how long this remains important for the future. For example, firms can use this approach to build models that forecast exchange rates based on the idea that past behaviour and price patterns may affect currency movements and can be used to predict future price behaviour and patterns.

What’s in it for today’s companies and their managers?

While the use of ML for time-series forecasting has been researched intensively in the last few years, the results have always been modest. This time it seems to be different. Because both Google and Facebook possess large volumes of data, they have succeeded in developing algorithms that are very accurate in their predictions. Google, for example, has provided statistics on how it has performed better than 92% of hand-made models.

Another major advantage is that these new algorithms have succeeded in making complex time-series problems much easier to build and predict. After all, many of the previous forecasting models needed lots of human effort and expert knowledge to construct and fine-tune the model, which limited their use in practice. Both the Facebook and Google models involve high levels of automation, so that firms and their managers can focus primarily on framing the right business question, feeding the model with input data, and interpreting the results.

A final advantage is that all this comes at a very low cost, since the algorithms are open source and thus can be used by everyone, including both larger and smaller companies. Thus, more than ever, high technology is becoming accessible to everyone! We will only need to show high levels of entrepreneurship to embrace it and use it in a responsible way.