How to tune hyperparameters

[liqi0126]

Thanks for your great work! This repo includes many state-of-the-art methods and is easy to reproduce. So I would like to ask for some advice about insight into tuning hyperparameters. There are many hyperparameters given in the script of training, and I'm wondering how did you dig them out. How should I adjust them if I am training a model of my own?

[Kshitij09]

@rwightman I'm also stumbled upon same question. How could I know what are the hyperparameters that I'm able to tune? I agree it varies depending on the model, but I'm looking for some documentation or a place to dig into. Particularly, I'm interested in customizing activation_layer, attention_layer, anit-aliasing layer

[rwightman]

@liqi17thu @Kshitij09 this reply has been a long time coming, but going to move to discussions as it seems a good thread to leave open.

There's no magic to hparam tuning, just persistence and spending enough time with your models and datasets to get a feel for what works and what doesn't. With more experience you'll get better at finding starting points but will still have to search for the optimal values. If you've got enough compute, just doing exhaustive sweeps can be a good strategy, especially if your dataset isn't large.

The starting points, and large improvement jumps in hparams often comes from ideas put forward in various papers and their combinations. My current 'best' ImageNet hparams are based on a blend of ideas in EfficientNet, RandAugment, and 'Bag-of-Tricks' papers + experience based additions which include turning up the augmentation and regularization significantly. This won't necessarily work as well on a different dataset, especially smaller ones, or for fine-tuning instead of training from scratch, etc.

Fine-tuning, I often find a simple combo of Adam + Plateau LR schedule, or Momentum/Nesterov (SGD) w/ cosine decay and lower LR than you'd use for training from scratch is a good starting point. Also less augmentation is usually a good starting point.