[HW4] Problem 2.2

[yoonheeh]

Hello!

Problem 2.2 asks to compare the results between BERT and LSTM from assignment 1, but unfortunately, I have not done assignment 1. Could someone share what accuracy results they got for LSTM on 1) accuracy and 2) training time?
It would be very much appreciated, and thank you in advance!

Best,
YH

[emjay73]

with LSTM ( Assignment1, Prob5.1 )
Validation set Accuracy : 0.78 (@epoch 23)
Train phase 1 epoch time : min 36.22s / max 37.44s ( 28 epoch )

[yoonheeh]

Thank you!!! 😊

[emjay73]

Also, you can find the assignment1 solution from the material list below.
materials : https://seominjoon.github.io/kaist-ai605/schedule.html
solution : https://colab.research.google.com/drive/1Mq_ZUPis2F8xuH9a2u9FzA_hEdMzFYeW
I didn't try it, but you can use this result instead of mine.
I wonder what others' result was like though.

[emjay73]

Well, reported figures are measured from my local computer, and I realize that the runtime from the colab is quite different. Values above are measured from my local computer (with 2080ti) and measurement from the colab shows the following results.
(I used minibatch size = 8 for both cases)
=> Train Set Sample Accuracy : 0.87500
=> Train Epoch Time : min 17.74s / max 18.37s
=> Dev Set Best Accuracy : 0.78565 (@epoch 30)
=> Test Set Accuracy/Precision/Recall : 0.797 / 0.765 / 0.835

[yoonheeh]

🤔 hmm... since I got 0.822 for BERT, compared to the results you shared, the accuracy from BERT seems to be better than that of LSTM (which is expected). Thanks for sharing such detailed comments!