Training time

[sharonapa]

Hi, can you approximate train time of 100k examples on Gtx 1080? I started it, seems very slow. Thanks.

[Belval]

The current pretrained model (used to make the predictions in the README) took about 24-48 hours to train on a GTX1080Ti. You'd be looking at 72 hours-ish for a GTX1080.

If you feel adventurous, I am pretty sure that there are a lot of low-hanging fruits in the training code that could speed up the whole thing. Current data loading is not optimal for example.

[sharonapa]

Thanks, I've noticed it's taking some time to process one batch. I've few more questions : 1) where can I see your 200k data set? What was the parameters you ran the training? 2) the loss value I saw started from 6900 and dropping. Is this normal value? 3) my parameters were: 100k data set, batch size 32, 100k iterations (too much?) 4) can I speed up the lstm cells(or other ways to speed up) ? I've read that cudnnlstm does support variants sequence length. Thanks again. בתאריך יום א׳, 31 במרץ 2019, 16:05, מאת Edouard Belval ‏< notifications@github.com>:

…

The current pretrained model (used to make the predictions in the README) took about 24-48 hours to train on a GTX1080Ti. You'd be looking at 72 hours-ish for a GTX1080. If you feel adventurous, I am pretty sure that there are a lot of low-hanging fruits in the training code that could speed up the whole thing. Current data loading is not optimal for example. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKS0brt1zJF7si7Rt5O_cKNLvKRXuWKAks5vcLKmgaJpZM4cULKs> .