Identifying Spatio-Temporal Drivers of Extreme Events (IDEE)

Computer Vision Group, Institute of Computer Science III, University of Bonn.

This is the code to reproduce the results presented in the paper: "Identifying Spatio-Temporal Drivers of Extreme Events" by Mohamad Hakam Shams Eddin, and Juergen Gall. Accepted at NeurIPS'24.

The code for generating the synthetic data is available at Synthetic_Multivariate_Anomalies.

Website | Paper

The spatio-temporal relations of extreme events impacts and their drivers in climate data are not fully understood and there is a need of machine learning approaches to identify such spatio-temporal relations from data. The task, however, is very challenging since there are time delays between extremes and their drivers, and the spatial response of such drivers is inhomogeneous. In this work, we propose a first approach and benchmarks to tackle this challenge. Our approach is trained end-to-end to predict spatio-temporally extremes and spatio-temporally drivers in the physical input variables jointly. We assume that there exist precursor drivers, primarily as anomalies in assimilated land surface and atmospheric data, for every observable impact of extremes. By enforcing the network to predict extremes from spatio-temporal binary masks of identified drivers, the network successfully identifies drivers that are correlated with extremes. We evaluate our approach on three newly created synthetic benchmarks where two of them are based on remote sensing or reanalysis climate data and on two real-world reanalysis datasets.

Poster

Setup

For pip with a virtual environment:

virtualenv <env_name>
source <env_name>/bin/activate
pip install -r requirements.txt

Pytorch has to be updated to be computable with your GPU driver.

If you want to use Mamba:

pip install mamba-ssm
pip install mamba-ssm[causal-conv1d]

Code

The code has been tested under Pytorch 1.12.1 and Python 3.10.6 on Ubuntu 20.04.5 LTS with NVIDIA A100 & A40 GPUs and GeForce RTX 3090 GPU.

Configuration

The main config file config.py includes the parameters for training and testing and models.

To train on the synthetic data:

  train_synthetic.py

For testing on the synthetic data:

  test_synthetic.py

Similarly, to train on the real-world data i.e., ERA5 Land:

  train_ERA5_Land.py

and for testing on ERA5 Land:

  test_ERA5_Land.py

The code is implemented with a simple DataParallel for multi-GPUs. Training on real-world different than the one in the paper, requires fine-tuning for the hyper-parameters and weighting.

Backbones

• Video Swin Transformer
• Vision Mamba
• 3D CNN

Baselines

• Multiple instance learning:
  • DeepMIL
  • ARNet
  • RTFM
  • MGFN
• One Class:
  • SimpleNet
• Reconstruction based:
  • UniAD
  • STEALNET

Pretrained models

Synthetic CERRA Reanalysis

name	backbone	resolution	config	model
deepmil_exp3	Video Swin Transformer	200x200	config	checkpoint
arnet_exp3	Video Swin Transformer	200x200	config	checkpoint
rtfm_exp3	Video Swin Transformer	200x200	config	checkpoint
simplenet_exp3	Video Swin Transformer	200x200	config	checkpoint / backbone
stealnet_exp3	Video Swin Transformer	200x200	config	checkpoint
uniad_exp3	Video Swin Transformer	200x200	config	checkpoint
ide_swin_exp3	Video Swin Transformer	200x200	config	checkpoint
ide_mamba_exp3	Vision Mamba	200x200	config	checkpoint

Real-world CERRA Reanalysis

name	backbone	region	resolution	config	model
ide_CERRA	Video Swin Transformer	Europe	512x832	config	checkpoint

Real-world ERA5-Land Reanalysis

name	backbone	region	cordex	resolution	config	model
ide_AFR	Video Swin Transformer	Africa	AFR-11	804x776	config	checkpoint
ide_CAS	Video Swin Transformer	Central Asia	CAS-11	400x612	config	checkpoint
ide_EAS	Video Swin Transformer	East Asia	EAS-11	668x812	config	checkpoint
ide_EUR	Video Swin Transformer	Europe	EUR-11	412x424	config	checkpoint
ide_NAM	Video Swin Transformer	North America	NAM-11	520x620	config	checkpoint
ide_SAM	Video Swin Transformer	South America	SAM-11	668x584	config	checkpoint

Structure

├── Baselines_MIL
│   ├── config.py
│   ├── dataset
│   │   └── Synthetic_dataset.py
│   ├── log
│   ├── models
│   │   ├── agent
│   │   │   └── Swin_3D.py
│   │   ├── build_arnet.py
│   │   ├── build_deepmil.py
│   │   ├── build_mgfn.py
│   │   ├── build_rtfm.py
│   │   ├── classifier
│   │   │   ├── ARNet.py
│   │   │   ├── DeepMIL.py
│   │   │   ├── MGFN.py
│   │   │   └── RTFM.py
│   │   ├── encoder
│   │   │   ├── CNN_3D.py
│   │   │   ├── Mamba.py
│   │   │   └── Swin_3D.py
│   │   └── losses.py
│   ├── test_mil_synthetic.py
│   ├── train_arnet_synthetic.py
│   ├── train_deepmil_synthetic.py
│   ├── train_mgfn_synthetic.py
│   ├── train_rtfm_synthetic.py
│   └── utils
│       └── utils_train.py
├── Baselines_OneClass
│   ├── config.py
│   ├── dataset
│   │   └── Synthetic_dataset.py
│   ├── log
│   ├── models
│   │   ├── build_simplenet.py
│   │   ├── encoder
│   │   │   ├── CNN_3D.py
│   │   │   ├── Mamba.py
│   │   │   └── Swin_3D.py
│   │   └── losses.py
│   ├── test_simplenet_synthetic.py
│   ├── train_simplenet_synthetic.py
│   └── utils
│       └── utils_train.py
├── Baselines_Reconstruction
│   ├── config.py
│   ├── dataset
│   │   └── Synthetic_dataset.py
│   ├── log
│   ├── models
│   │   ├── build_steal.py
│   │   ├── build_uniad.py
│   │   ├── initializer.py
│   │   └── losses.py
│   ├── test_steal_synthetic.py
│   ├── test_uniad_synthetic.py
│   ├── train_steal_synthetic.py
│   ├── train_uniad_synthetic.py
│   └── utils
│       └── utils_train.py
├── config.py
├── dataset
│   ├── CERRA_dataset.py
│   ├── ERA5_Land_dataset.py
│   └── Synthetic_dataset.py
├── docs
│   ├── images
│   │   ├── era5.png
│   │   ├── NeurIPS-logo.svg
│   │   ├── neurips-navbar-logo.svg
│   │   └── video_cover_ide.jpg
│   └── poster
│       └── Shams_Gall.png
├── log
├── models
│   ├── build.py
│   ├── classifier
│   │   └── CNN_3D.py
│   ├── codebook
│   │   ├── FSQ.py
│   │   ├── LatentQuantize.py
│   │   ├── LFQ.py
│   │   ├── Random_VQ.py
│   │   └── VQ.py
│   ├── encoder
│   │   ├── CNN_3D.py
│   │   ├── Mamba.py
│   │   └── Swin_3D.py
│   └── losses.py
├── README.md
├── scripts
│   ├── download_cerra.sh
│   ├── download_era5_land.sh
│   ├── download_noaa_cerra.sh
│   ├── download_noaa_era5_land.sh
│   └── download_synthetic.sh
├── test_CERRA.py
├── test_ERA5_Land.py
├── test_synthetic.py
├── train_CERRA.py
├── train_ERA5_Land.py
├── train_synthetic.py
├── utils
│   └── utils_train.py
└── vis
    ├── visualize_CERRA_data.py
    ├── visualize_ERA5-Land_data.py
    ├── visualize_NOAA_data.py
    └── visualize_synthetic_data.py

Dataset

• The full data set can be obtained from https://doi.org/10.60507/FK2/RD9E33 (~ 1.1 TB after decompression).

• The data can be also downloaded via scripts found in scripts i.e., you can download the synthetic data via script/download_synthetic.sh (~46 GB):

   wget --continue  https://bonndata.uni-bonn.de/api/access/datafile/7506 -O Synthetic.7z
  

  To extract the files you need the 7-Zip packge:

   sudo apt update
   sudo apt install p7zip-full
  

  and to extract:

   7za x Synthetic.7z
  

  CERRA files are large, so they are split into two files. To extract CERRA files just run:

   7za x CERRA.7z.001
  

  7z will find CERRA.7z.002 automatically.

• You can visualize the data using the scripts in vis.

Citation

If you find our work useful, please cite:

@inproceedings{
eddin2024identifying,
title={Identifying Spatio-Temporal Drivers of Extreme Events},
author={Mohamad Hakam Shams Eddin and Juergen Gall},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=DdKdr4kqxh}
}

Acknowledgments

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within the Collaborative Research Centre SFB 1502/1–2022 - DETECT - D05 and by the Federal Ministry of Education and Research (BMBF) under grant no. 01IS24075C RAINA.

License

The code is released under MIT License. See the LICENSE file for details.