Suppose I have a viable model, is it/will it be possible to
- Resume training from data saved to disk (i.e. potentially in another session, on another machine…)
- Do more training, i.e. if I had specified 10 epochs and training has completed, continue training from epoch 11 as though 10+n had initially been specified but training stopped/paused
Use case: suppose (following the hyperparameters question in Issues/Feedback) I have trained a model in several ways with different hyperparameters and have finally identified the best way to train…
I’ve already invested time & energy (literally, running the processor) and don’t want to re-run the best candidate just for the sake of reproducing a state I already have.
Serious question ends…
Bonus Question for the Theoretically Inclined: can trained parameters from different runs be combined in any way? Thinking: I’ve run, say, 10 epochs of the same model with different randomisation… to what extent (thinks, hand-waving appeal to Central Limit Theorem?) could the trained parameters be considered population samples that could be combined by e.g. taking means?
Superbonus Question: how could one apply genetic algorithms to combine parameters from different runs. Would one have to be able to “slice” the parameters by processing dependency (“top” of image through convolution etc. chain) or could one just consider insert-other-model-values in place of simple dropout. Literally copy values for dropout from the same location in another run.