【TimeXer】论文研读，时序模型

前言

虽然读研以来一直想做nlp，但由于小导不是这个方向。我们折中了一下选择了时序，最起码还能蹭一蹭Transformer。

TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables

来源：arXiv:2402.19072v1 @ TimeXer

ECL (Li et al., 2019) includes hourly electricity consumption data from 321 clients. We take the electricity consumption of the last client as an endogenous variable and other clients as exogenous variables.
Weather (Zhou et al., 2021) records 21 meteorological factors collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020. In our experiment, we use the Wet Bulb factor as the endogenous variable to be predicted and the other indicators as exogenous variables.
ETT (Zhou et al., 2021) contains four subsets where ETTh1 and ETTh2 are hourly recorded, and ETTm1 and ETTm2 are recorded every 15 minutes. The endogenous variable is the oil temperature and the exogenous variables are 6 power load features.
Traffic (Wu et al., 2023a) records hourly road occupancy rates measured by 862 sensors of San Francisco Bay area freeways. We take the measurement of the last sensor as an endogenous variable and others as exogenous variables.

Contains five datasets representing five different day-ahead electricity markets spanning six years each.

NP represents The Nord Pool electricity market, recording the hourly electricity price, and corresponding grid load and wind power forecast from 2013-01-01 to 2018-12-24.
PJM represents the Pennsylvania-New Jersey-Maryland market, which contains the zonal electricity price in the Commonwealth Edison (COMED), and corresponding System load and COMED load forecast from 2013-01-01 to 2018-12-24.
BE represents Belgium’s electricity market, recording the hourly electricity price, load forecast in Belgium, and generation forecast in France from 2011-01-09 to 2016-12-31.
FR represents the electricity market in France, recording the hourly prices, and corresponding load and generation forecast from 2012-01-09 to 2017-12-31.
DE represents the German electricity market, recording the hourly prices, the zonal load forecast in the TSO Amprion zone, and the wind and solar generation forecasts from 2012-01-09 to 2017-12-31.

(a) Both patch embedding and variate embedding are utilized for the endogenous variables to obtain multiple temporal tokens and a variate token respectively.
(b) Each exogenous variable is embedded as a variate token through variate embedding.
(c) Self-attention is applied to the endogenous temporal tokens to capture patch-wise dependencies.
(d) Cross-attention is adopted to model the series-level dependencies over endogenous and exogenous variables.

(a) 内生变量的embedding与以往常见的point-wise分割方式不同，每个token中包含多个time step，这种分割方式为patch-wise。作者保证了每个patch互不重叠。同时这层还将整个内生变量的数据打包成了一个series-wise的变量。
(b) 外生变量的embedding是按变量不同进行分割，每个token包括一个变量的series-wise数据。
(c) 内生变量样本经过self-attention模块，主要是为了探索内生变量token之间的时间依赖性。
(d) cross-attention用来探索内生和外生变量的之间series级别的依赖性。

Transformer-based model:
1. iTransformer (Liu et al., 2023)
2. PatchTST (Nie et al., 2022)
3. Crossformer (Zhang & Yan, 2022)
4. Stationary (Liu et al., 2022b)
5. Autoformer (Wu et al., 2021)
CNN-based model:
1. TimesNet (Wu et al., 2023a)
2. SCINet (Liu et al., 2022a)
Linear-based model:
1. RLinear (Li et al., 2023)
2. DLiear (Zeng et al., 2023)
3. TiDE (Das et al., 2023). Among these models, TiDE is an advanced recent forecaster elaborated for exogenous variables.

–有空再写

外生变量在现实世界预测场景中的是普遍存在的，过去的Transformer-based model往往忽略这一点，普遍是单变量预测单变量、多变量预测多变量。作者提出了用多变量数据预测单一变量，在不改变Transformer架构的情况下赋予了它处理外生变量和内生变量的信息的能力。
凭借设计巧妙的embedding模块，TimeXer能够捕获内生变量的时间依赖性以及内生变量和外生变量之间的多变量相关性。
实验结果表明，TimeXer在具有外生变量的短期和长期预测任务中都实现了最先进的性能。TimeXer在各种包含外源变量的复杂现实世界预测场景中展示了其潜力，包括数据质量低和series之间错位等挑战。