V0.12.3 Various II

So… running the full set of mnist pngs (140k) split 85:5:10 with the recommended model.

1st thoughts

Excellent!

I can use the model hub, and inspect the model while training proceeds and everything seems responsive. For example, clicks on the > of the convolution preview changes feature immediately.

Observations

  • In Statistics view, previews never appear on the map, even when training has been completed.
  • Weights on convolution only show one set even though there are 8 features specified by the recommended model
  • Dense_1 (128 neurons) vs Dense_2 (10): output, weights & biases graphs oddly inconsistent. Output of Dense_2 is a (widely spaced) histogram, but smooth line graph for Dense_1. I thought - aha! - # neurons, but Weights and Biases are graphed as line graphs for both
  • Tab selection in Viewbox resets when component selection changes in the Map - I can understand why but it’s a bit annoying if I am trying to compare e.g. gradients across components that I have to move mouse, click, move back for every selection.
  • Looking at the Output tab of the Statistics view, I still don’t know how to read the Prediction vs Ground Truth panels. Why is there only one Ground Truth bar? Shouldn’t the One be split across all the categories (totalling One of course)?
  • Accuracy over all epochs (there was only 1 on this occasion) is empty
  • Why the strange dip in Accuracy during one epoch at the start of validation - it looks too large to be real (eyeball SD & probability estimation :wink:
  • Run tests - already noted elsewhere, but nothing happens even though server restarted, model rebuilt, etc. Possibly the “dataset too large” error that kills the kernel.
  • Restart server, re-open model, (but same browser, no cache clear) alas no previews on the components in modelling tool, but they are visible in the Preview area in the RH panel.
  • Annoying: it no longer recognises that the model has been trained once - status is Waiting in the hub, but a checkpoint was saved.

Minor Oddities

In Statistics view, with focus on a convolution the “squares” of the Weights in the Viewbox cease to be square when the window is moved to another monitor whose dimensions in pixel is different, and the lower value of the scale bar is lost. They return to square when the windows is thrown back to the original monitor.

Squares not square on monitor 2

Generally some parts scale better with window resizing than others (while on the second monitor, so squares not square to begin with)

That’s enough for now :slight_smile:

Thanks a ton for the feedback @JulianSMoore! :pray:
I’ve added those we didn’t already have to our “polishing” stack.

To answer some questions:

Looking at the Output tab of the Statistics view, I still don’t know how to read the Prediction vs Ground Truth panels. Why is there only one Ground Truth bar? Shouldn’t the One be split across all the categories (totalling One of course)?

Prediction vs Ground Truth is showing just for one sample (the same sample you are seeing in the input etc.), this means that there only will be one prediction and one label in this bar plot -> one Ground Truth bar (the label for that single sample).

  • Why the strange dip in Accuracy during one epoch at the start of validation - it looks too large to be real (eyeball SD & probability estimation :wink:

This is something we need to make a bit prettier.
What happens is that the validation starts at 0 before any validation samples have been evaluated, which leads to the huge dip until you get to the validation phase.
There was a difficulty with keeping the length of the lines different in the framework we are using for visualizing the graphs, so this was the lowest hanging solution. Looking for a nicer way to do it though, maybe just defaulting it to the previous validation value rather than 0 would help :thinking:

Hi @robertl

Re validation value: it may actually be 0 but I think it is conceptually NAN - why not “fill down” :wink: the last training value if you can’t skip it for the plot? Some plotting (matplotlib??) can omit NAN, but not sure how that is handled in your case.

Re Prediction vs Ground Truth: so, I should read this specific example as saying - the real digit was 5, but in this batch the model found it hard to distinguish 5 from 3, but over the epoch it was was improving because 5 was favoured over 3, 5 was never confused with 1 and five looks only a bit like 8… wait! What?

Eh? I just noticed the prediction values can be -ve; if this is a classification task then the outputs are (I thought) interpreted as probabilities: how can there be -ve values?

Is there some blending of regression/classification going on here? Again, IIRC, the loss was Quadratic which would tend to be more for regression and would admit -ve values. If that is the case, why is Quadratic the default from the recommendation engine (rather than, say, cross entropy)?

@robertl Here’s the mnist with cross-entropy…

If the statistics view is for the single output as described earlier, what’s going on here? Ground truth is not a single value in the batch.

(Also: ground truth should be full height if there is 1 truth, it’s just a marker it seems)

Hi @JulianSMoore,

Re Prediction vs Ground Truth: so, I should read this specific example as saying - the real digit was 5, but in this batch the model found it hard to distinguish 5 from 3, but over the epoch it was was improving because 5 was favoured over 3, 5 was never confused with 1 and five looks only a bit like 8… wait! What?

Haha, the Batch Prediction vs Ground Truth will soon be replaced by a confusion matrix to make the model predictions a bit easier to understand.
The (Sample) Prediction vs Ground Truth just says “The real digit was 5, but the model predicted it to be 3 because it has difficulties distinguish 5 from 3”.

Eh? I just noticed the prediction values can be -ve; if this is a classification task then the outputs are (I thought) interpreted as probabilities: how can there be -ve values?

By “-ve” you mean negative values?
It depends on what activation function you have in the last component. If running with the cross-entropy loss you likely want a softmax as activation on that last component of your model to prevent this.

Is there some blending of regression/classification going on here? Again, IIRC, the loss was Quadratic which would tend to be more for regression and would admit -ve values. If that is the case, why is Quadratic the default from the recommendation engine (rather than, say, cross entropy)?

This will be fixed soon, the recommendation engine is still fairly MVP :slight_smile:

1 Like

@robertl PS yes “-ve” = negative; “softmax to prevent” negative values with cross-entropy? Um. I was reporting what was happening with Quadratic, so shouldn’t the recommender also use an appropriate activation fn?

Because it was broken I just rebuilt it and confirmed (after fixing the Merge and linking the 2nd input) activation on the Dense is None

Tried running it, but same error of course…

Internal error in asyncio.events:88: Trainer raised an error on validation: InvalidArgumentError()
Traceback (most recent call last):
  File "perceptilabs\coreInterface.py", line 262, in perceptilabs.coreInterface.coreLogic._validate_trainer
  File "perceptilabs\trainer\base.py", line 63, in perceptilabs.trainer.base.Trainer.validate
  File "perceptilabs\trainer\base.py", line 218, in perceptilabs.trainer.base.Trainer._compute_total_loss
  File "c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\ops\array_ops.py", line 195, in reshape
    result = gen_array_ops.reshape(tensor, shape, name)
  File "c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 8372, in reshape
    tensor, shape, name=name, ctx=_ctx)
  File "c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 8397, in reshape_eager_fallback
    ctx=ctx, name=name)
  File "c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 1 values, but the requested shape has 2 [Op:Reshape]

so shouldn’t the recommender also use an appropriate activation fn?

It absolutely should, but it doesn’t quite yet :slight_smile:
For the error, I wrote in the other thread about that one.