Hyperparameter tuning

Once there is support for preserving and comparing the data from >1 training run across >1 session it would be very useful to be able to set up (rummages through vocabulary for collective noun not already in use) heats, i.e. a batch job of training runs that would allow parameters of (at least) the training node to be varied.

Running the textile demo on my setup takes a while but with GPU running I could do several runs overnight… being able to run competitive heats between hyperparameters would be useful for that.

Hi @JulianSMoore,
Absolutely agree, HP tuning is something we have had our eyes on to implement for a long while.

Followup question, since HP tuning and NAS both create variations of a model (some more drastic than others), which model would you want to be represented on the Workspace?
Or would you prefer to have a “blueprint” model which contains HP tuning/NAS elements and then be able to create a copy of the best model it generates into another model?

Oops - missed the questions.

I would prefer to have a blueprint model with Hyperparameter tuning/network architecture search elements and then to be able to create a new model from any of the variations. “Best” should be determined by the user based on clearly presented metrics: loss/accuracy/batch size/…, but by all means indicate what seems best based on standard criteria,

I would also like to ensure that HP tuning/NAS elements are also in the derived model: searches might start coarse and then be refined with scaled/transformed ranges.

It should a also be possible to remove the HP/NAS elements with a single action if they impose any performance penalties.

1 Like