THE Deep Learning Benchmarks
See: soumith/convnet-benchmarks#101
Come back here on June 15th, 2016.
A bit delayed due to y'know -- a lot of co-ordination among groups.
- InceptionV3-batchnorm (http://arxiv.org/abs/1512.00567 , https://gist.github.com/soumith/e18a56b847a8905a84fffb1f32b06db8, https://github.com/Moodstocks/inception-v3.torch)
- Alexnet-OWT
- VGG-D
- ResNet-50 ( http://arxiv.org/abs/1512.03385 , https://github.com/facebook/fb.resnet.torch )
- C3D - A vgg-style 3D net ( http://vlg.cs.dartmouth.edu/c3d/ )
- DeepSpeech2 - Convnet + RNN + FC ( http://arxiv.org/abs/1512.02595 )
- MSR's 5 layer FC net ( https://github.com/Alexey-Kamenev/Benchmarks )
- Small RNN LSTM ( https://github.com/karpathy/char-rnn/blob/master/train.lua#L38-L48 )
- Large RNN LSTM ( BIG-LSTM in http://arxiv.org/abs/1602.02410 )
- Initially multi-GPU with (1 to 4 titan-X cards)
- However, multi-machine, custom hardware, other GPU cards such as AMD, CPUs etc. can and should be accommodated, we will work this out after the initial push.
- Round-trip time for 1 epoch of training (will define an epoch size separately for each network)
- Maximum batch-size that fits (to show and focus on the extra memory consumption that the framework uses)
Everyone who wants to join-in, but I thought an initial set that is important to cover would be:
- Caffe
- Chainer
- CNTK
- MXNet
- Neon
- Theano
- TensorFlow
- Torch
- Emit JSON output (so that the README -- or jekyll website can be auto-generated, similar to http://autumnai.com/deep-learning-benchmarks )
- I will personally to the best of my abilities make sure that the benchmarking is fair and unbiased. The hope is that the community at large will watch these and point-out / fix mistakes.
- The benchmarks will be placed at https://github.com/DeepMark/deepmark and other key community members / organizations who want ownership will be welcome to join in proposing new benchmarks that get relevant as the field progresses.