Skip to content

Latest commit

 

History

History
164 lines (143 loc) · 5.76 KB

README.md

File metadata and controls

164 lines (143 loc) · 5.76 KB
Raw

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks by Bohdan Ivaniuk-Skulskyi, Andrew Garrett Kurbis, Alex Mihailidis, and Brokoslaw Laschowski

News

10/07/2024

  • Our absrtact accepted to Toronto Robotics Conference

10/06/2024

  • Our paper accepted to IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob) 2024

05/05/2023

  • Our absrtact accepted to ICRA 2023 Computer Vision for Wearable Robotics Workshop

19/04/2023

  • Our poster accepted to ICAIR 2023

Installation

Create virtual environment and run requirements.txt

python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/Atze00/MoViNet-pytorch.git
pip install -r requirements.txt

Install StairNet dataset and run preprocessing file

python data_preprocessing/dataset_preprocessing.py --data_folder /path-to-dataset-dir/

Define os environment variable pointing to it

export DATASET=/path-to-preprocessed-dataset-dir/
  • unzip data_splits/train.txt file.

Model Zoo

Name Parameters GFLOPs Resolution Accuracy F1-score Download
MoViNet 4.03M 2.5 5x3x224x224 0.983 0.982 model config
MobileViT-LSTM 3.36M 9.84 5x3x224x224 0.970 0.968 model config
MobileNet-LSTM 6.08M 53.96 5x3x224x224 0.973 0.970 model config
MobileNet-LSTM (seq2seq) 5.93M 50.97 5x3x224x224 0.707 0.799 model config
Baseline (Kurbis et al.) 2.26M 0.61 3x224x224 0.972 0.972 - -

Inference

Download one checkpoint with its configuration file and run the following command

python test.py --experiment_cfg CONFIG.yaml \
  --dataset_folder $DATASET \
  --val_samples_file data_splits/validation.txt \
  --test_samples_file data_splits/test.txt \ 
  --checkpoint_path CHECKPOINT.pth

Train

python train.py --experiment_cfg CONFIG.yaml \
  --dataset_folder $DATASET \
  --train_samples_file data_splits/train.txt \
  --val_samples_file data_splits/validation.txt \
  --test_samples_file data_splits/test.txt

Confusion matrices

BibTex

Sequential image classification of human-robot walking environments using temporal neural networks

@article {Ivanyuk-Skulskiy2023.11.10.566555,
  author       = {Ivanyuk-Skulskiy, Bogdan and Kurbis, Andrew Garrett and Mihailidis, Alex and Laschowski, Brokoslaw},
  title        = {Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks},
  elocation-id = {2023.11.10.566555},
  year         = {2023},
  doi          = {10.1101/2023.11.10.566555},
  publisher    = {Cold Spring Harbor Laboratory},
  URL          = {https://www.biorxiv.org/content/early/2023/11/13/2023.11.10.566555},
  eprint       = {https://www.biorxiv.org/content/early/2023/11/13/2023.11.10.566555.full.pdf},
  journal      = {bioRxiv}
}

StairNet: visual recognition of stairs for human–robot locomotion

@article{Kurbis2024,
  author       = {Andrew Garrett Kurbis and Dmytro Kuzmenko and Bogdan Ivanyuk-Skulskiy and Alex Mihailidis and Brokoslaw Laschowski},
  title        = {StairNet: visual recognition of stairs for human–robot locomotion},
  journal      = {BioMedical Engineering OnLine},
  year         = {2024},
  volume       = {23},
  number       = {1},
  pages        = {20},
  doi          = {10.1186/s12938-024-01216-0},
  url          = {https://doi.org/10.1186/s12938-024-01216-0},
  issn         = {1475-925X}
}

Acknowledgements

The visual encoder models are taken from timm library and the MoViNet is based on this implementation