Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA

Official Pytorch Implementation of Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA (IEEE Access)

---

Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA
Won Jo (Sejong Univ.), Geuntaek Lim (Sejong Univ.), Joonsoo Kim (ETRI), Joungil Yun (ETRI), and Yukyung Choi (Sejong Univ.)

Paper: Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA

Abstract: As the demand for large-scale video analysis increases, video retrieval research is also becoming more active. In 2014, ISO/IEC MPEG began standardizing compact descriptors for video analysis, known as CDVA, and it is now adopted as a standard. However, the standardized CDVA is not easily compared to other methods because the MPEG-CDVA dataset used for performance verification is not disclosed, despite the fact that follow-up studies are underway with multiple versions of the CDVA experimental model. In addition, analyses of modules constituting the CDVA framework are insufficient in previous studies. Therefore, we conduct self-evaluations of CDVA to analyze the impact of each module on the retrieval task. Furthermore, to overcome the obstacles identified through these self-evaluations, we suggest temporal nested invariance pooling, abbreviated as TNIP, which implies temporal robustness realized by improving nested invariance pooling, abbreviated as NIP, one of the features in CDVA. Finally, benchmarks of the existing CDVA and the proposed approach are provided on several public datasets. Through this, we show that the CDVA framework is capable of boosting the retrieval performance if utilizing the proposed approach.

Prerequisites

Recommended Environment

• Python 3.7
• Pytorch 1.1

Depencencies

You can set up the environments by using dockerfile

$ make docker-base.

$ make docker-run.

Data Preparation

FIVR

Fine grained incident Video Retrieval dataset used in our work can be downloaded from the FIVR-200K. The experiment was conducted using FIVR-5K disclosed by the author of the ViSiL. The data should be located like the structure below.

├── dataset
   └── FIVR
       ├── video
         ├── video_1
         ├── video_2
         └── ...
       └── missing_video
         ├── missing_video_1
         ├── missing_video_1
         └── ...

CC_WEB_VIVDEO

Near duplicate video retrieval dataset used in our work can be downloaded from the CC_WEB. The data should be located like the structure below.

├── dataset
   └── CC_WEB
       ├── 1
         ├── 1_1_Y.flv
         ├── 1_2_Y.flv
         └── ...
       └── 2
         ├── 2_1_Y.flv
         ├── 2_2_Y.flv
         └── ...
       └── ...

Usage

Extract TNIP Feature

You can easily extract the TNIP feature.

$ bash TNIP_FIVR5K.sh
$ bash TNIP_CC_WEB.sh

If you want to try other extract options, please refer to args.py.

Evaluate CDVA Retreival

Data Preparing

Annotation files can be download from

1. FIVR Annotation

2. CC_WEB Annotation

To check out experiments, you can evaluate our retrieval csv file.

• FIVR5K
• FIVR200K
• CC_WEB

$ python calculate_performance_fivr.py
$ python calculate_performance_cc_web_video.py

Experiments

FIVR5K

method	DSVR	CSVR	ISVR
TCAc	0.609	0.617	0.578
ViSiLf	0.838	0.832	0.739
ViSiLsym	0.830	0.823	0.731
ViSiLv	0.880	0.869	0.777
TCAf	0.844	0.834	0.763
TCAsym	0.763	0.766	0.711
SCFV+NIP256	0.813	0.781	0.673
SCFV+TNIP256	0.880	0.862	0.744

• Result file

FIVR200K (Additional Benchmark)

method	DSVR	CSVR	ISVR
TCAc	0.570	0.553	0.473
ViSiLf	0.843	0.797	0.660
ViSiLsym	0.833	0.792	0.654
ViSiLv	0.892	0.841	0.702
TCAf	0.877	0.830	0.703
TCAsym	0.728	0.698	0.592
SCFV+NIP256	0.819	0.764	0.622
SCFV+TNIP256	0.896	0.833	0.674

• Result file

CC_WEB_VIDEO

method	cc_web	cc_web*	cc_webc	cc_webc*
DML	0.971	0.941	0.979	0.959
TCAc	0.973	0.949	0.983	0.965
DP	0.975	0.958	0.990	0.982
TN	0.978	0.965	0.991	0.987
ViSiLf	0.984	0.969	0.993	0.987
ViSiLsym	0.982	0.969	0.991	0.988
ViSiLv	0.985	0.971	0.996	0.993
TCAf	0.983	0.969	0.994	0.990
TCAsym	0.982	0.962	0.992	0.981
SCFV+NIP256	0.973	0.953	0.976	0.959
SCFV+TNIP256	0.978	0.969	0.983	0.975

• Result file

References

We referenced the repos below for the code.

• ViSiL
• FIVR
• CC_WEB

Contact

If you have any question or comment, please contact using the issue.

Citation

@article{jo2022exploring,
  title={Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA},
  author={Jo, Won and Lim, Guentaek and Kim, Joonsoo and Yun, Joungil and Choi, Yukyung},
  journal={IEEE Access},
  volume={10},
  pages={38973--38981},
  year={2022},
  publisher={IEEE}
}