Timer-XL

Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [Paper], [Slides], [Poster], [Intro]

🚩 News (2025.01) Timer-XL has been accepted as ICLR 2025. See you in Singapore :)

🚩 News (2024.12) Released a univariate pre-trained model [HuggingFace]. A quickstart usage is provided here.

🚩 News (2024.10) Model checkpoint, training script, and pre-training dataset are released in [OpenLTM].

Usage

Zero-Shot Forecasting

For users interested in zero-shot forecasting, we release a HuggingFace model as a out-of-box forecaster.

Model Adaptation

For developers interested in fine-tuning or training on customized datasets, please refer to OpenLTM.

Introduction

Timer-XL is a decoder-only Transformer for time series forecasting. It can be used for task-specific training or scalable pre-training, handling arbitrary-length and any-variable time series.

💪 We observe performance degradation of encoder-only Transformers on long-context time series.

💡 We propose multivariate next token prediction, a paradigm to uniformly predict univariate and multivariate time series with decoder-only Transformers.

🌟 We pre-train Timer-XL, a long-context version of time-series Transformers (Timer), for zero-shot forecasting.

🏆 Timer-XL achieves state-of-the-art performance as for time series forecasting: [Univariate], [Multivariate], [Covariate], [Zero-shot].

What is New

For our previous work, please refer to Time-Series-Transformer (Timer)

In a word, we generalize next token prediction from 1D sequences to 2D time series.

Comparison

Time-Series Transformers	PatchTST	iTransformer	TimeXer	UniTST	Moirai	Timer	Timer-XL (Ours)
Intra-Series Modeling	Yes	No	Yes	Yes	Yes	Yes	Yes
Inter-Series Modeling	No	Yes	Yes	Yes	Yes	No	Yes
Causal Transformer	No	No	No	No	No	Yes	Yes
Pre-Trained	No	No	No	No	Yes	Yes	Yes

Multivariate Next Token Prediction

We generalize next-token prediction for multivariate time series. Each prediction is made based on tokens of the previous time from multiple variables:

Universal TimeAttention

We design TimeAttention, a causal self-attention allowing intra- and inter-series modeling while maintaining the causality and flexibility of decoder-only Transformers. It can be applied to univariate and covariate-informed contexts, enabling unified time series forecasting.

Main Results

Citation

If you find this repo helpful, please cite our paper.

@article{liu2024timer,
  title={Timer-XL: Long-Context Transformers for Unified Time Series Forecasting},
  author={Liu, Yong and Qin, Guo and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  journal={arXiv preprint arXiv:2410.04803},
  year={2024}
}

Acknowledgment

We appreciate the following GitHub repos a lot for their valuable code and efforts:

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)
• Large-Time-Series-Model (https://github.com/thuml/Large-Time-Series-Model)
• AutoTimes (https://github.com/thuml/AutoTimes)

Contact

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Guo Qin (qinguo24@mails.tsinghua.edu.cn)

License

This model is licensed under the Apache-2.0 License.