V0.12.3 Various III

TL;DR “You only run once” You can run twice (yay! well done) but PL forgets the results (nay :frowning: )

Also

  • sometimes GPU not used??
  • need copy/paste for text fields in UI
  • rename folders on model rename please

Short Points

  • Model Hub, data wizard, etc. don’t seem to support copy/paste on text such as model name - please fix, it will smooth the experience

Long Points

I created mnust_140k_85_5_10_No_BOM_3 (note the typo… I can rename the model in the model hub but the model folder also needs to be renamed)

Reason: I re-ran a model from yesterday and I swear it didn’t use the GPU properly - run time was 2x the nominal

So I restarted PL and it looked as though the model was not trained

So I ran it again

mnust_140k_1st__training_after_restart.txt (25.8 KB)

Notice the difference in CPU & GPU usage.

Then I ran it again without restarting PL congratulations! the memory leak seems to have been fixed - it ran fine!

Unfortunately, on restarting PL we see this: it looks fine but that is the timing for the 1st training run! And looking in the checkpoints folder nothing had been added for the second run

Hope that helps.

Thanks as always @JulianSMoore!
We will look into it. Some parts of it is our current frontend storage method which needs to change, but the part of the GPU seems strange.
The checkpoints are currently also overwriting rather than saving twice. Something we will tackle as soon as we look at the first step of version control in the tool. Expect to see that one sometime this summer :slight_smile:

1 Like