Why the different previews using the perceptilabs

I tried to build the PilotNet two times following the video tutorials . However, the model previews and training previews are not the same on these two tried? I don’t know why the happened?
The training previews are empty on the upper four windows.

Hi @wucc

That’s a nice question - and I’m sure one of the PL guys will give you a definitive answer but I just wanted to join in to check my own thinking about this.

Firstly, I believe the content of the previews reflect the current state of the model, so what is seen will be different at different times (you have 3% and 15% here and I don’t know how significant that might be!)

Beyond that…

It is very difficult to guarantee reproducibility in TensorFlow end-end when training: even with careful control of the several randomisers likely to be in use (I know PL has some control over this, but not the details) if a GPU is in use the parallel processing can introduce varying results.

However, things should look very similar at the end of training - though not necessarily exactly the same… any variation in initialisations will lead to variation in backprop and although the weights etc. should converge, they will probably not converge to exactly the same numbers - just to values sufficient for the required accuracy/loss etc.

You don’t seem to be using GPU for training & CPU training is generally recommended for reproducibility, but there is a small GPU bump in the graph… can you confirm you are training with CPU only?

How were these two models in overall performance after all training had been completed? I’m very interested in hearing about your results :slight_smile:

Hi @wucc,
Nice to see that you got the PilotNet running!

Hmm, since it’s the older version, it’s a bit hard to say straight off the bat why the top part is not visible. If you run the tool as “perceptilabs -v=3” we can take a look and see if there is any clue.

For why you are seeing different things in the previews I defer to @JulianSMoore’s great answer about the reproducibility in TF. Each component is initialized with random weights, so the initial preview may very well look quite different if that’s what’s happening.
I’m very curious as well about the end performance as he points out :slight_smile: