when this 1st training phase ie "Train the captioning module" will stop?

[harshadl]

Hello Vladimir,
Thank you so much for wonderful documentation..
I am trying to execute your code on my machine. I am novice to Deep Learning. With some research i reached upto 1st training phase ie "Train the captioning module". Training is going on since last 8 hours.
My machine is Ubuntu 18.04 with RTX2060
Here is output log:-

The best metrict was unchanged for 0 epochs.
Expected early stop @ 37
Started @ 210531174920; Current timer: 4.57
0531174920: train 7 @ cuda:0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1068/1068 [04:09<00:00, 4.28it/s]
0531174920: val_1 7 @ cuda:0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 249/249 [00:54<00:00, 4.56it/s]
0531174920: val_2 7 @ cuda:0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 243/243 [00:53<00:00, 4.58it/s]
0531174920: val_1 1by1 7 @ cuda:0: 100%|█

My question is, when this 1st training phase ie "Train the captioning module" will stop?

Also will you please tell about stopping condition for "Train proposal generation module"

[v-iashin]

Hi, I am glad you liked the documentation.

In our paper, on page 18, we have an extensive Implementation Details section that can advise you on these questions and more.

The captioning module was trained until METEOR on the validation set has not im- proved for 30 epochs while the proposal generator is trained for 70 epochs at most. In our experiments, the training of the final captioning module reaches the peak performance at 26th epoch while the proposal generator achieves the highest F1-score on the validation set at 17th epoch.

[harshadl]

Thank you for reply,
I'll go through the paper carefully.
So, according to epoch conditions, will it take 8 - 10 days to train on my Core i7 , 16 GB RAM, RTX2060 single GPU?
If i need to train the system, on my customized video dataset, which only focused on only 10 video scenarios,

1. I am interested to work only on scene specific videos wrt "railway stations" and "shopping malls"

then how to generate video dataset? and what are steps i need to follow?

[v-iashin]

1068/1068 [04:09<00:00, 4.28it/s]

This suggests to me that it wouldn't take that long. Am I missing something?

One speculation, by the way. The longest time takes the '1-by-1 sampling' after every epoch. It is especially problematic during the early stages of training. Just by tweaking

BMT/main.py

Lines 80 to 81 in d45ad8f

	parser.add_argument('--one_by_one_starts_at', type=int, default=1,
	help='# of epochs to skip before starting 1-by-1 validation (saves time)')

you can speed it up significantly. However, this produces the metric which we use to track the progress. So, considering you have a smaller batch size and, therefore, slower training dynamics, an increase to 5 or 10 should do.

how to generate a video dataset? and what are the steps I need to follow?
Good question, but I don't have an answer for it. If I were you, I would try to "mimic" the organization of ActivityNet Captions dataset – e.g. how caption-video pairs are organized. Then, you can use our code-base for feature extraction (video_features) which we used for this project and MDVC. However, please, keep in mind the branch as the current main branch of video_features is ahead of the one we used for BMT (https://github.com/v-iashin/BMT#details-on-feature-extraction) but it shouldn't matter that much considering you want to use it on another dataset.