In a 2024 NeurIPS study, Mingtian Tan and colleagues tested three forecasting systems built by other research teams. Each system used a language model trained on text to predict future measurements, such as hourly electricity use.
To find out whether the language model improved those forecasts, Tan’s team tested each system with the language model in place, with it removed, and with two smaller replacements. The simpler versions often made more accurate forecasts.
Four things to know
1. Tan’s team tested each forecasting system with its language model removed and with two smaller replacements.
Consider how one system, Time-LLM, would forecast electricity use. It groups past meter readings into short segments and converts each segment into a form its language model, LLaMA, can process. A final layer uses LLaMA’s output to predict future readings.
Mingtian Tan and colleagues trained one version of Time-LLM that sent the prepared segments straight to the final layer, skipping LLaMA. They also trained two smaller replacements for LLaMA. One was a single multi-head self-attention layer, which learns how much weight to give different segments. The other was a single transformer block, the kind of unit that language models stack in layers. The team compared all three versions with full Time-LLM, then repeated the comparison on CALF and OneFitsAll, two forecasting systems that use GPT-2.
2. On every dataset, at least one simpler version of Time-LLM beat the researchers’ reproduction of the full system.
The team scored forecasts on 13 datasets using mean absolute error, the average size of a miss, and mean squared error, which gives large misses more weight. Averaging each score across forecast horizons gave them 26 comparisons. A simpler version won each time, though the winner varied. On taxi demand, the transformer block beat full Time-LLM on both measures. On that same dataset, CALF and OneFitsAll each did best with GPT-2 in place.
3. On one weather task, a single attention layer cut Time-LLM’s training time from 50 hours to just over two minutes.
The task was to predict the next 96 weather readings. Full Time-LLM had a mean absolute error of 0.199. Removing LLaMA raised the error to 0.213. Replacing it with an attention layer that learned how much weight to give each segment of past readings brought the error down to 0.189. The researchers timed both runs on an NVIDIA A100 GPU and adjusted learning rates or batch sizes for some smaller models.
In partnership with
The ice cream shop that makes money when it's cold
28 Wishes sells ice cream in Los Angeles. Below 70°F, sales fall about 20%. The weather is out of their hands. Rent isn't.
So the owners started putting about $20 a day into Kalshi weather markets, taking the cold side. The days that keep customers away now pay something back.
This is hedging. Big companies have done it for decades, buying protection against bad weather, fuel spikes and rising rates. It used to take a broker, a trading desk, and an order size no corner shop could meet.
Kalshi opens it up. Contracts on weather, fuel prices, inflation, tariffs and regulation, starting at a few dollars. Take a position on the outcome that would hurt you. If it hits, the payout softens it. If it doesn't, the contract expires and the good month was the point.
4. Starting with a model trained on text gave CALF no consistent advantage.
The team kept GPT-2’s design but varied what it had learned before it saw the forecasting data. In one version, they assigned GPT-2’s weights at random, then trained it on the forecasting task. The authors credited that version with eight wins in 16 comparisons, including both measures of error for electricity use. Standard CALF, which adapts a pretrained GPT-2 to the task, won three comparisons. A version that used the pretrained GPT-2 without adapting it won the other five.
