A retailer may need a forecast for every product in every store. Testing a forecasting system adds another dimension: what would it have predicted at each earlier point in time?
Those are two different GPU workloads. Production inference asks one question across many series. Historical evaluation asks many questions of each series, with a little more history revealed every time.
We optimized t0-alpha, our first forecasting foundation model, for both. On one NVIDIA H100, its model core produces 235,000 full-horizon probabilistic forecasts a second. For historical evaluation, it reaches 1.87 million forecasts a second across multiple series and evaluation points. Our end-to-end pipeline, including storage read and result materialization, sustains 197,000 forecasts a second.1
We have since released t0-beta, our latest forecasting foundation model. We’ll update this post with t0-beta benchmark results when they’re ready.
One forecast includes the stated horizon and all five quantiles. Both rates time warm, synchronized calls with GPU-resident inputs on one H100. Historical evaluation uses expanding contexts across 512 series.
Why speed matters in forecasting
A retailer’s forecast may cover millions of product–store combinations. An energy operator may need to update forecasts across meters, substations and regions as weather assumptions change. These runs have deadlines: a forecast that arrives after the replenishment order or the trading decision is not much use.
Speed also changes how you work on the model. A backtest usually raises another question. Would weather have helped? Does a shorter history work better? Was the improvement confined to one region? In the API or Retrocast, fast evaluation lets you follow those questions while you are still working on the problem.
The automated forecaster we’re building follows the same idea: run an experiment, inspect the result, choose what to try next. It can do more of that work if it spends less time waiting for the model.
Making one forecast fast
Our open tfc-t0 package is the starting point. Using bfloat16 (bf16, short for “brain floating point”), a 16-bit number format, it processes about 17,000 forecasts a second on a workload with varying batch sizes and history lengths. Adding PyTorch’s compiler, torch.compile, raises that to 35,000. We use the same model weights in both versions and in our optimized engine.
A GPU can do an enormous amount of arithmetic quickly. Keeping it busy with useful work is another matter. Making a forecast also involves preparing the data, starting operations on the GPU, and moving intermediate results through memory. Those costs add up, especially when the individual operations are small.
Skipping unnecessary setup
The general model code has to handle many kinds of request: incomplete histories, related series forecast together, and predictions far into the future. A batch of independent products with complete sales histories is a simpler job. We give it a shorter route through the code.
Before forecasting, the model rescales each series so that differences in magnitude are easier to work with. For independent series, we can calculate those scaling statistics directly on the GPU, using only the history available at each point. There is no need to assemble groups of related inputs first.
Likewise, t0-alpha can predict many future steps in one pass. If that already covers the requested forecast length, we skip the code that manages repeated calls for longer forecasts.
Choosing where to use lower precision
Using fewer bits to represent a number can make GPU calculations much faster. The catch is rounding: an error in a scaling calculation can affect everything that follows it.
Both bf16 and fp16 use 16 bits, but divide them differently. Compared with fp16, bf16 trades precision for range: eight exponent bits instead of five, and seven fraction bits instead of ten. It covers roughly the same range as fp32, but with fewer significant digits.
We use lower precision for the large matrix multiplications, where much of the computation happens. We keep more precision for scaling statistics, sums, and the updates that carry information between model layers. This lets us accelerate the expensive operations without applying the same rounding everywhere.
For workloads that need closer agreement with 32-bit floating-point execution, or fp32, we also have a slower mode. It splits values into high and low components and accumulates their contributions separately. Both modes use the same model weights.
On identical test inputs, our throughput mode’s largest output difference from fp32 was 1.296e-3, versus 4.114e-2 for stock bf16. That is 31.7× less maximum numerical deviation while running substantially faster. This measures fidelity to fp32 execution, not predictive accuracy.
Shorter bars mean less deviation from fp32 execution, not better predictive accuracy. Throughput and deviation were measured in the same five-quantile suite.
Avoiding padding
A long-lived product may have years of history beside a new SKU with six weeks. To process them together, a common approach is to pad the shorter series until both have the same length. That makes the six-week SKU pay the compute bill for the three-year one.
Instead, we group histories by length and pack their actual observations together. Variable-length attention lets the model process those histories without extending every one to the longest length in the batch. We put the forecasts back in their original order afterwards, so callers do not have to manage the grouping.
We cache the packing plan when the same lengths recur, but always use the new data. In our retail serving test, a mixed-length batch and a uniform batch took effectively the same time.
Fusing GPU operations
Imagine calculating an intermediate result, writing it to GPU memory, then immediately reading it back for the next calculation. A model does this many times in a single forecast. For small operations, starting the next piece of work and moving the data can cost as much as the calculation.
We combine operations that repeatedly run together into a single GPU function, or kernel. The intermediate result can then stay close to the computation instead of making another round trip through memory. For example, we fuse projection work with query/key normalization and rotary position transforms, and combine the first MLP projection with its SwiGLU activation.
We still use vendor libraries where they are faster. The engine chooses between implementations for each operation and input shape.
Keep compilation off the request path
One request might contain a few series with short histories; the next might contain thousands with years of data. These are different input shapes. A compiler can generate efficient code for them, but generating that code takes time.
We compile dynamic execution classes at startup, rather than compiling every combination of batch size and history length separately. In our dense-serving tests, previously unseen shapes then run without the multi-second compilation pauses. We finish this warmup before routing traffic to a worker.
Repeated shapes become candidates for CUDA-graph replay: recording a sequence of GPU operations and replaying it without dispatching each operation separately. These recordings are cached separately for each concrete request shape and configuration. Each recorded plan has its own input and output buffers, so alternating between plans does not overwrite another plan’s data.
Preparing a worker from a cold compiler cache took about eight minutes on H100; its first subsequent request took 151 ms. The compiled reference took about 110 seconds across construction and its first request. We pay the setup cost before accepting traffic, and measure throughput once the workers are warm.2
How fast does it get?
Together, these changes bring the model core to 235,000 full-horizon forecasts a second at the fixed serving shape—8.2× the open package with bf16 and torch.compile enabled.
To test changing shapes, we interleave nine batch/context configurations, spanning batches of 256–2,048 series and histories of 160–448 steps. All shapes are warm. The optimized engine reaches 179,000 forecasts a second: 10.4× the open package’s native bf16 path, and 5.1× with torch.compile added.
6,784 series/pass · context 160–448 · horizon 52 · batch 256–2,048 · five quantiles. Minimum of 12 complete warm passes per configuration, each in a fresh H100 container with the same checkpoint. The package and engine are separate implementation paths; step gains below the arrow compare successive engine configurations. Both graph-replay layers are checked.
On a 1,024-series request with 160 steps of history, the engine without compilation runs in 11.80 ms, versus 26.38 ms for the compiled reference. That is 2.2× faster despite roughly the same kernel count, 669 versus 691. And the custom kernels still leave plenty for the compiler to do: compilation cuts the engine’s kernel count to 224 and brings the request down to 7.50 ms.
Across the mixed workload, compilation adds 1.52× over the eager engine, and CUDA-graph replay adds 1.79× on top. The benefit of replay changes considerably with batch size. In a separate sweep, it improved performance by 10% at batch 4,096. At batch 16, it made the request 7.24× faster. Dispatch overhead takes up much more of a small call, which is why we test small requests as well as large batches.
The final engine took 37.89–38.63 ms per complete pass across all 12 repetitions. Its median was 38.11 ms, or about 178,000 forecasts a second—close to its best run.
One pass, many historical cutoffs
For backtests, we can go further. Here we need forecasts for the same series at many earlier points in time, or cutoffs.
The straightforward way to run a backtest is to call the model once for each historical date. Slice the series at that date, run the model, then repeat. A 64-cutoff backtest can make the transformer reread the beginning of the same series 64 times.
t0 was designed from the beginning to learn from forecasts made at many points within each training series, not just at its end. This is forking-sequence training, introduced by Wen et al. in their multi-horizon quantile recurrent forecaster.
Think of the conventional backtest as rereading a book from page one for every bookmark. t0-alpha reads history in 32-point chunks, or patches, and builds an internal representation at the end of each one. Its attention is causal: a representation can use earlier observations, but cannot look ahead. The scaling statistics follow the same rule. Each of these points can therefore serve as the starting point for a forecast.
An ordinary prediction returns a forecast from the end of the history. The forking interface returns forecasts from the earlier points we ask for as well. Its output has four dimensions:
(series, cutoffs, horizon, quantiles)We keep the requested historical representations through the final decoding layer and rescale each forecast using statistics from its own history. The path uses the same compilation and graph machinery as ordinary serving, with a separate warmup.
Every later cutoff sees a superset of the earlier history. This is expanding-context evaluation, not a fixed sliding window.
An earlier forecast must not benefit from seeing what happened later. We checked this by rewriting the final input patch and comparing earlier forecasts. Their maximum change was 1.344e-3 in throughput mode and 2.9e-6 on the fp32 reference path. These small differences are consistent with numerical rounding, though the test alone cannot prove that earlier forecasts are unaffected by later data.
Asking for 65 historical forecasts instead of two adds just 0.22 ms in our cutoff sweep. At 4,096 steps of context and batch 128, the two calls take 10.28 ms and 10.50 ms: more than 32× as many forecasts for roughly 2% more time, with memory flat at 3.50 GB. Most of the work is reading the history. Once those states are available, decoding more of them is cheap.
Against a conventional prefix-by-prefix evaluation, the advantage grows with history length. Each independent request reads its prefix again; the forking path reads the full span once. At 160 steps of context and five cutoffs, forking is 6× faster. At 2,048 steps and 64 cutoffs, it is 71× faster, producing 1.87 million cutoff forecasts a second.
The bars share one linear time axis. These are application-level production routes; the prefix arm also includes the serving stack's one-patch eager route.
What if we want a forecast between patch boundaries? We shift the input and run another pass, giving us a second set of historical points. Two passes, offset by half a patch, cover half-patch spacing; 32 passes cover every individual step. This does more work, but returns proportionally more forecasts. Moving from patch boundaries to every step changes throughput from 1.285 million to 1.261 million forecasts a second—a two-percent difference per forecast.
This fast path currently handles independent series without missing values, the model’s five native quantiles, and a 32-step forecast at each cutoff. It uses expanding history: later forecasts see more of the past, rather than a fixed-width sliding window. Additional inputs such as weather, joint forecasting of related series, and other horizons are outside this optimized path today.3
The GPU still has to send those extra forecasts back. From two cutoffs to 64, compute stays near 10.5 ms while output grows from 0.3 MB to 10.5 MB. Copying the results to CPU memory then accounts for about 10% of compute-plus-copy time, up from 1%.
If an experiment only needs aggregate scores, we may not need to copy every forecast back to the host. Reducing the results on the GPU would avoid moving data we are about to discard.
35× more output, nearly flat compute. Durable storage and metric reduction are not included.
What about KV caching?
We tested a KV-cache prototype on the reference model, but it is not in the optimized engine. Long-horizon production rollout uses a sliding context window. As the window moves, the causal scaler recomputes statistics, changing retained positions’ normalized embeddings and invalidating their cached keys and values.
The live path reuses work within each rollout step instead. t0 already emits several forecast patches per forward pass. When another step is needed, its quantile trajectories share a context prefix: we compute that prefix once per series, then separate the path-specific suffixes. No K/V is persisted across steps.
What this changes
At the measured end-to-end rate, 500 million target-only forecasts would take about 42 minutes on one H100. Adding covariates would increase the work, since each adds another input series to process. For faster results, we can split independent series across multiple GPUs.
For evaluation, we can check many historical dates without paying for a separate model call at each one. If a candidate looks good overall, we can examine where it fails: a particular product group, a seasonal peak, a period of unusual demand. Those checks can inform the next experiment while the work is still in progress.
This is how we want the automated forecaster to work too: test a candidate, look closely at the errors, and use them to choose the next experiment. Fast backtests make those iterations much more practical.
Using the optimized engine
Both t0-alpha and our latest model, t0-beta, have open weights. These benchmarks use a serving checkpoint of t0-alpha, newer than its original public checkpoint, with identical weights in the reference and optimized implementations. We have also taken t0-alpha in the other direction: running a forecasting foundation model entirely in a browser tab.
We already use the optimized engine in dedicated deployments and custom customer workloads, and are bringing it to the general API and Retrocast. If you have a large forecasting job or a backtest that takes too long, we’d like to hear about it.
We are hiring engineers and researchers to work on forecasting models, GPU execution and automated experimentation. If those problems sound interesting, we’d like to hear from you.
Benchmark notes
-
A forecast here includes the whole requested horizon and five quantiles, not one scalar prediction. The 235,284/s result uses batch 4,096, context 160 and horizon 52. The 1,873,574/s result uses 512 series × 64 historical cutoffs, context 2,048 and horizon 32. Both time warm, synchronized predictor calls with GPU-resident inputs on one H100. The 197,015/s storage-to-output result is a separate pipeline benchmark; its difference from the core result is not a matched estimate of pipeline overhead.
-
Compute tests use the same t0-alpha serving checkpoint in both implementations, the optimized engine’s throughput mode, five quantiles, PyTorch 2.12.1 and Triton 3.7.1. No t0-beta measurements are included. The original precision gate used PyTorch 2.11.0 / Triton 3.6.0; the numerical checks reported here were rerun on the article stack. Chart throughput uses minimum warm timings. Each mixed-shape configuration uses a fresh H100 container, four warmup passes and 12 complete timed passes. This tests batch/context variation, not arbitrary unseen shapes or production tail latency. Cold-cache construction plus first call totals 501.85 seconds for the optimized engine and 110.22 seconds for the compiled reference; imports, process startup and GPU input allocation are excluded.
-
Forking uses patch-aligned contexts in throughput mode, with shifted passes for off-grid cutoffs. The 6×–71× comparisons replace repeated production
predict()calls with the forking interface. The single-prefix route also uses a small-horizon eager path that forking bypasses, so those ratios include routing differences as well as shared computation. Output-copy timings exclude durable storage and metric reduction.