🛒MART🛒
MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction

Seongju Lee · Junseok Lee · Yeonguk Yu · Taeri Kim · Kyoobin Lee
ECCV 2024

ECCV Paper Arxiv Poster Source Code Cite MART

---

This repo is the official implementation of "MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction (ECCV 2024)"

📢 Updates

• (2024.09.19) Official repository of 🛒MART🛒 is released
• (2024.09.30) Update ECCV poster
• (2024.11.21) Train and evaluation code for ETH-UCY dataset is uploaded
• (2024.11.22) Train and evaluation code for SDD dataset is uploaded
• (2025.06.24) Minor bug in PRT is fixed
• (2025.xx.xx) Source code for convert SDD dataset from PECNet is uploaded
• (2025.xx.xx) Source code for visualization is uploaded

🖼️ ECCV Poster

🚀 Getting Started

Environment Setup

1. Set up a python environment

conda create -n mart python=3.8
conda activate mart

2. Install requirements using the following command.

pip install -r requirements.txt

🚂 Train & Evaluation

🏀 NBA Dataset

• The dataset is included in ./datasets/nba/

• Train MART on the NBA dataset

  python main_nba.py --config ./configs/mart_nba.yaml --gpu $GPU_ID
  

• Test MART on the NBA dataset after training

  python main_nba.py --config ./configs/mart_nba.yaml --gpu $GPU_ID --test
  

🚶 ETH-UCY Dataset

• The dataset is included in ./datasets/ethucy/

• Train MART on the ETH-UCY dataset

  chmod +x ./scripts/train_eth_all.sh
  ./scripts/train_eth_all.sh ./configs/mart_eth.yaml $GPU_ID
  

• Test MART on the ETH-UCY dataset after training

  chmod +x ./scripts/test_eth_all.sh
  ./scripts/test_eth_all.sh ./configs/mart_eth.yaml $GPU_ID
  

🚁 SDD Dataset

• The dataset is included in ./datasets/stanford/

• Train MART on the SDD dataset

  python main_sdd.py --config ./configs/mart_sdd.yaml --gpu $GPU_ID
  

• Test MART on the SDD dataset after training

  python main_sdd.py --config ./configs/mart_sdd.yaml --gpu $GPU_ID --test
  

📊 Main Results

🏀 NBA Dataset

• Version with minor bug fixes

  minADE (4.0s): 0.728
  minFDE (4.0s): 0.902
  

• In paper

  minADE (4.0s): 0.727
  minFDE (4.0s): 0.903
  

🚶 ETH-UCY Dataset

minADE Table
       ETH    HOTEL    UNIV    ZARA1    ZARA2    AVG
       0.35    0.14    0.25    0.17    0.13    0.21    

minFDE Table
       ETH    HOTEL    UNIV    ZARA1    ZARA2    AVG
       0.47    0.22    0.45    0.29    0.22    0.33    

🚁 SDD Dataset

minADE: 7.43
minFDE: 11.82

🐣 How to reproduce results

🏀 NBA Dataset

• The checkpoint is included in ./checkpoints/mart_nba_reproduce/

  python main_nba.py --config ./configs/mart_nba_reproduce.yaml --gpu $GPU_ID --test
  

• The results will be saved in ./results/nba_result.csv

🚶 ETH-UCY Dataset

• The checkpoints are included in ./checkpoints/mart_eth_reproduce/

  ./scripts/test_eth_all.sh ./configs/mart_eth_reproduce.yaml $GPU_ID
  

• The results will be saved in ./results/$SUBSET-NAME_result.csv

🚁 SDD Dataset

• The checkpoint is included in ./checkpoints/mart_sdd_reproduce/

  python main_sdd.py --config ./configs/mart_sdd_reproduce.yaml --gpu $GPU_ID --test
  

• The results will be saved in ./results/sdd_result.csv

📝 Citation

@inproceedings{lee2024mart,
  title={MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction},
  author={Lee, Seongju and Lee, Junseok and Yu, Yeonguk and Kim, Taeri and Lee, Kyoobin},
  booktitle={European Conference on Computer Vision},
  pages={89--107},
  year={2024},
  organization={Springer}
}

🤗 Acknowledgement

• The part of the code about the feature initialization is adapted from (GroupNet).
• Thanks for sharing the preprocessed NBA dataset and dataloader (LED).
• Thanks for sharing the ETH-UCY dataloader (SGCN).
• Thanks for sharing the training code of ETH-UCY (NPSN).
• Thanks for sharing the preprocessed SDD dataset (PECNet).
• Thanks for providing the code of the Relational Transformer (RT). We implemented the RT from jax to pytorch.