Event Camera Data Pre-training
_{Official PyTorch Implementation of the ICCV 2023 Paper
Paper | Supp | Video}

Getting Started

1. Dependency

• python 3.8.13
• numpy 1.24.1
• torch 1.11.0
• torchvision 0.10.1
• pytorch_lightning 1.6.4
• einops 0.4.0
• clip
• timm 0.9.2
• pyyaml 6.0
• flatten_dict 0.4.2
• torch_scatter 2.0.9

2. Datasets

• N-ImageNet dataset. Please follow here for downloading and organizing the dataset. The expected file sturctures are shown below.

└───N_Imagenet
    └───/extracted_train/n******/***.npz
    │
    └───/extracted_val/n******/***.npz

• ImageNet dataset. After obtaining the dataset, please follow here to extract the dataset. The expected file sturctures are shown below.

 ___/train/n******/***.JPEG
│
└───/val/n******/***.JPEG

3. Pre-training

• extract clip features by

python3 extract_clip_feature.py --batch 128 --source_dir #where you store ImageNet --save_dir #where you store the CLIP features

• download a MoCoV3 checkpoint pre-trained on ImageNet to slighly improve the model performance, and store it in ./checkpoints/ folder.
• Change path for files in config folder accordingly.
• Run the following code:

python3 train_bert.py --opt config/ours/pr_vits.yml --gpus 4 --num_nodes 4

4. Linear probing / Fine-tuing

• Change path for files in config folder accordingly.
• Perform learn probing before fune-tuning will shorten the training epochs and usually improve the model performance.
• Run the following code:

python3 train_bert.py --opt config/ours/lr_vits.yml --gpus 4 --num_nodes 4
python3 train_bert.py --opt config/ours/ft_vits.yml --gpus 4 --num_nodes 4

Pre-trained models

Find pre-trained models here.

Contact

If you have any questions, please drop me an email.

Acknowledgement

Most of the code is borrowed from:

• https://github.com/82magnolia/n_imagenet
• https://github.com/swz30/Restormer/tree/main
• https://github.com/facebookresearch/moco-v3
• https://github.com/facebookresearch/mae
• https://github.com/TimoStoff/event_utils

Citation

If you find this code useful, please consider citing:

@article{Yang_2023,
  author       = {Yan Yang and
                  Liyuan Pan and
                  Liu Liu},
  title        = {Event Camera Data Pre-training},
  journal      = {CoRR},
  volume       = {abs/2301.01928},
  year         = {2023},
  url          = {https://doi.org/10.48550/arXiv.2301.01928},
  doi          = {10.48550/arXiv.2301.01928},
  eprinttype    = {arXiv},
  eprint       = {2301.01928},
}