Question about loss

[hw-choo]

Thank you for your work and please understand that it is not a direct question about the code. Is there any reason the loss function does not include the classification error term? Some models that perform reconstruction and imputation are include classification error in the loss function. Have you ever trained models in this way? If so, please let me know what the results were like.

[WenjieDu]

Hi there,

Thank you so much for your attention to SAITS! If you find SAITS is helpful to your work, please star⭐️ this repository. Your star is your recognition, which can let others notice SAITS. It matters and is definitely a kind of contribution.

I have received your message and will respond ASAP. Thank you again for your patience! 😃

Best,
Wenjie

[WenjieDu]

Thank you for raising this issue to discuss. You can append a classification task to train SAITS (just like BRITS does) if your work needs it. Actually, I did think about this when I was designing SAITS. We can extend the jointly-optimization training approach into a multi-task one including classification. But finally, I didn't, and the reasons are listed below.

Firstly, as the title illustrated, this paper focuses mainly on the imputation problem. So we try to solve it well to surpass the state-of-the-art. As you can see in the experiment 'Downstream Classification Task', our SAITS with a two-stage method (impute then classify) achieves comparable performance to the end-to-end model BRITS in ¹. This qualitative evaluation together with the quantitative imputation accuracy evaluation in the paper demonstrates the quality of imputation from SAITS.

Secondly, as I said above, if one's work needs SAITS to perform end-to-end classification, one can add additional NN modules and append the classification task in the training to see its performance. This is very intuitive and natural, and cannot be claimed as a novelty in the paper. Different from ORT+MIT utilized to solve the optimization problem of imputation for self-attention models, appending a classification task here is more like applying SAITS structure to solve a classification task, this is an application. Also, the classification task may disturb the optimization of the imputation problem (in my model training, I observed imputation accuracy degraded). Therefore, if we append the classification task, it will be kind of redundant.

If you are interested in this topic, you can give it a try. And if you can report your investigation here, I can pin this issue to let others notice your work and continue the discussion. This won't be a problem.

Footnotes

1. Cao, W., Wang, D., Li, J., Zhou, H., Li, L., & Li, Y. (2018). BRITS: Bidirectional Recurrent Imputation for Time Series. NeurIPS 2018. ↩

[hw-choo]

Thank you for your kind reply. I will try and share it if there are interesting results.