Consider a retail company with stores in several states of the USA. Four teams in the company forecast the same sales, each at its own level. The finance team forecasts total sales of the whole chain to set the budget. Regional managers forecast sales in each state. The store operations team forecasts each store to plan staff. The replenishment team forecasts each product in each store to place orders. When each team forecasts its level separately, the numbers do not match. The sum of the store forecasts differs from the state forecast, and the sum of the product forecasts differs from the chain forecast. Figure 1 shows this for a chain with three states and ten stores.
Finance plans for 1,280k units. The product forecasts behind the orders sum to 1,140k, so 140k units of the budget have no order behind them.
−9.6%
−16.4%
−9.4%
−19.5%
Forecasts that disagree
Coherent forecasts: every level is consistent with the others.
A set of forecasts is said to be coherent when every level is consistent with the others. The forecast for a store equals the sum of the forecasts for its products. The forecast for a state equals the sum of the forecasts for its stores, and so on up to the chain. Turning separate forecasts into a coherent set is called hierarchical forecast reconciliation.
When forecasts are not coherent, each team plans with a different number. Finance approves a budget that does not match the orders placed by the stores. A regional manager sees a gap between the state forecast and the sum of the store forecasts and has to decide which one to use. When actual sales come in, it is hard to tell whether a miss came from demand or from the forecasts disagreeing with each other.
An old question
Economists asked in 1960 whether a model of the total does better than the sum of models of its parts, and forecasters asked the same question about regional forecasts in the 1970s. The topic now even has its own chapter in the standard forecasting textbook. Two longstanding approaches make the levels agree. Figure 2 shows how they move numbers through a small hierarchy.
- Bottom-up. Forecast every product at the bottom level, for instance in every store, then sum. The levels are consistent, but the series of single products in single stores are sparse and noisy, and their errors accumulate in the sum.
- Top-down. Forecast the total, then split it with historical shares. The levels are consistent too, but a fixed split cannot follow a store or a product whose sales change differently from the rest.
These approaches were designed for point forecasts. With probabilistic forecasts we cannot sum quantiles, as a quantile of a sum is not the sum of the lower-level quantiles. We thus draw samples from the marginal forecasts, with control over how they relate to one another.
Both of the above methods are usually less accurate than forecasting each level separately, which is why many planning teams keep forecasts that disagree. These are not the only reconciliation methods that exist.
The sum of the forecasts should equal the forecast of the sum.
S: which stores belong to each level
B: the four store forecasts
Bottom-up adds the four store forecasts to get state and chain totals.
The M5 competition
M5: Walmart daily sales, 12 levels, 42,840 series
One of the best-known benchmarks for this forecast reconciliation problem is the M5 competition. It is a Kaggle competition organised in 2020 by the University of Nicosia with data from Walmart. It asks for 28 days of daily unit sales for 3,049 products in 10 stores in California, Texas and Wisconsin, at 12 levels. The levels are the chain, each state, store, category and department, their combinations, each product, and each product in each store. That makes 42,840 series. Both tracks give every level the same weight in the score.
- Accuracy track (5,507 teams, scored with WRMSSE). Teams submitted one forecast per product and store, and the organisers computed every other level by summing those forecasts. Every scored submission is therefore coherent, built bottom-up.
- Uncertainty track (892 teams, scored with WSPL). Teams submitted nine percentiles for each of the 42,840 series separately. This track does not require coherent forecasts.
The winner of the Accuracy track combined 220 gradient-boosted tree models built for this dataset, one per store, per store and category and per store and department, each in two variants. Our post on the history of forecasting methods explains why gradient-boosted trees dominated M5.
What forecasting models do today
Foundation models for time series, including Chronos-2, Timestral and the public release of our own t0-gamma, are used today to forecast each level of a hierarchy independently, and their forecasts are not coherent.
Using public t0-gamma as released, we forecast each of the 42,840 M5 series independently and compared the forecast for each store with the sum of the forecasts for its products (Figure 3). In every store the two differ by 9% or more, and the forecast for the chain differs from the sum of its product forecasts by more than 10%.
A planning team that wants one set of numbers then has two options. It can keep the separate forecast for each level and accept the gaps, or sum the product forecasts bottom-up and lose accuracy at every level above them. On M5, these bottom-up forecasts score poorly. Public t0-gamma’s bottom-up forecast falls outside the top 50 solutions on the Uncertainty leaderboard. Chronos-2’s bottom-up forecast is even worse than the ARIMA baseline set by the organisers. ARIMA scores 0.2046. Chronos-2 scores 0.3240. For these bottom-up comparisons, we sum mean forecasts for point accuracy. We draw from marginal forecasts to estimate uncertainty because quantiles cannot be added to give the quantiles of the total.
Coherent forecasts from t0
We have developed an internal version of t0-gamma with support for hierarchical forecast reconciliation in mind. It also supports static covariates. Its forecasts are coherent at every level, and so are its uncertainty ranges, from one product in one store to the whole chain. Figure 4 compares public t0-gamma, forecasting each level independently as released, with the internal coherent version against actual daily sales for the chain, three states and ten stores. Figure 4 lets you inspect the forecasts for every node of the chain–state–store hierarchy shown. The comparison includes improvements to the model as well as reconciliation.
Average daily error, indexed points (lower is better): public 9.5 · internal coherent 4.2.
Daily values for WI store 2
| Date | Actual | Public | Internal |
|---|---|---|---|
| 2016-05-23 | 81.5 | 85.7 | 83.4 |
| 2016-05-24 | 72 | 79.9 | 77.4 |
| 2016-05-25 | 77 | 75.5 | 75.6 |
| 2016-05-26 | 78.2 | 76.8 | 75.6 |
| 2016-05-27 | 84.8 | 82 | 85.8 |
| 2016-05-28 | 92.3 | 93.9 | 90 |
| 2016-05-29 | 83.5 | 102.6 | 81.2 |
| 2016-05-30 | 71.6 | 94.2 | 86.7 |
| 2016-05-31 | 70.7 | 99.1 | 82.8 |
| 2016-06-01 | 104.5 | 108 | 95 |
| 2016-06-02 | 114.2 | 118.2 | 113.2 |
| 2016-06-03 | 130.5 | 110.9 | 135.1 |
| 2016-06-04 | 118 | 124.6 | 116.1 |
| 2016-06-05 | 124.8 | 130.7 | 127.1 |
| 2016-06-06 | 120.1 | 112.8 | 125.2 |
| 2016-06-07 | 93.4 | 110 | 99 |
| 2016-06-08 | 105.2 | 109.7 | 113.1 |
| 2016-06-09 | 118.5 | 105.7 | 116.5 |
| 2016-06-10 | 98 | 113.6 | 102.7 |
| 2016-06-11 | 128.4 | 133.5 | 128.5 |
| 2016-06-12 | 131.4 | 126.8 | 136.1 |
| 2016-06-13 | 93.1 | 114.8 | 100.1 |
| 2016-06-14 | 115.3 | 110.4 | 113.6 |
| 2016-06-15 | 113 | 98.1 | 116.8 |
| 2016-06-16 | 97.2 | 94.8 | 94.1 |
| 2016-06-17 | 96.3 | 100.4 | 97.6 |
| 2016-06-18 | 100.3 | 105.3 | 104.3 |
| 2016-06-19 | 86.1 | 104.5 | 89.2 |
Internal t0-gamma scores better than the coherent bottom-up forecasts of every public model we tested (Figure 5).
The same internal t0-gamma would have placed 1st on both the Accuracy and Uncertainty tracks.
Results on M5
Internal t0-gamma scores 0.5169 on the Accuracy track and 0.1468 on the Uncertainty track. On the published leaderboards that is 1st of 5,507 teams, ahead of the winner at 0.5204, and 1st of 892, ahead of the winner at 0.1542. The public models are far behind when their forecasts are coherent (Figure 5).
M5 ran in 2020, and the field has learned from it since, so the comparison with the competitors of the time favours a model built today. Internal t0-gamma has not seen any data from the test window, 23 May to 19 June 2016.
What coherence means for a planning team
Today many planning teams forecast each level independently and live with the gaps. Others make the levels consistent with bottom-up or top-down, and accept a less accurate forecast at some levels. With internal t0-gamma, the budget, the regional plans, the store schedules and the orders come from one coherent set of forecasts, and the safety stock held in each store is consistent with the risk expected for the chain. When actual sales differ from the plan, the difference comes from demand, not from forecasts that disagree.