Concatenation Puzzle

[JulianSMoore]

I wanted to make a 3-channel mnist digit so, since I couldn’t turn off shuffle and take three separate inputs, tried to use the same input three times with two Merge components set to concatenate

Observations & Questions

• Merge_3 shows the same pattern twice - this is correct/expected
• Merge_4 shows all three channels different - this is (very) unexpected
• Q1 - is this unexpected-to-me behaviour by design?
• I could just about expect two different patterns if input1 to Merge_4 was a different sample to inputs 1 & 2 of Merge_3 (though I’d still call that unfortunate design), but… all 3 different?
• Reshape_2 shows 3x3 multicoloured pixelated digits
• Q2 - why is it 3x3 grid? Confirm not greyscale because of the unexpected concatenation behaviour?
• Q3 - am I doing something wrong with my Merge components
• Q4 - how could I use components and this data set to create the 3-channel data I wanted for convolution experiments

Suggestion: why not allow Merge to accept as many inputs as one drags onto it for concatenation. Consider the concatenations of Inception… lots of separate components to build it, then to repeat.

Suggestion: when updating component settings, don’t update the component/model until the user has hit “Apply” (new feature I suppose); the model updates on every change, creating errors due inconsistency that is purely ephemeral (I noticed this when changing the shape of the reshape from 28x28x3 to 3x28x28 to see what happened.)

And finally, but less definitively, why is the result different when the operations are ordered differently like this

[robertl]

Hi @JulianSMoore,
Great questions! Let me take them one by one.

Q1 - is this unexpected-to-me behaviour by design?

I think this is a rendering bug where it’s not able to visualize all datapoints and some distinct ones are lost in the image. We will soon have a zoom functionality for the graphs which would allow you to zoom into each of them and inspect them separately, which should show that they are the same.

Q2 - why is it 3x3 grid? Confirm not greyscale because of the unexpected concatenation behaviour?

Sometimes tensorflow reshapes it a little bit funny, this can be countered by transposing the image together with reshaping it. You can transpose it by pressing the blue arrows in the Reshape component. Here it looks somewhat better (although still on its side):

Q3 - am I doing something wrong with my Merge components
Q4 - how could I use components and this data set to create the 3-channel data I wanted for convolution experiments

I think you are doing it correct

Suggestion: why not allow Merge to accept as many inputs as one drags onto it for concatenation. Consider the concatenations of Inception… lots of separate components to build it, then to repeat.

Agreed with this, that’s how it used to work.
We had to change it now that we got more rigid input and outputs, so first we need to make sure that it’s easier to create more inputs to a component (currently hidden behind a right click, and then the new input does not automatically have any functionality but is just accessible in the custom code).
Hoping we can push out a new update with in not too long. With all the other things going on though it’s a little bit lower on our roadmap.

Suggestion: when updating component settings, don’t update the component/model until the user has hit “Apply” (new feature I suppose); the model updates on every change, creating errors due inconsistency that is purely ephemeral (I noticed this when changing the shape of the reshape from 28x28x3 to 3x28x28 to see what happened.)

We used to have an Apply button, but went away from it cause it felt a lot less reactive.
Of course, we don’t want to go too far the other way where it feels clumsy either. Maybe there is some happy middleground where only the component preview changes but not the rest of the model?

And finally, but less definitively, why is the result different when the operations are ordered differently like this

Tensorflow handling reshape a bit weird… I’ll dig into it and see if we can do anything more on our side though

[JulianSMoore]

@robertl “Rendering bug” - ah, I see what you mean… aliasing variation owing to rounding I expect. I’d have to say, I wouldn’t call that a bug as such - and it would be very hard to eliminate. But, if zooming in also zoomed the little graphs rather than just magnifying them that would mitigate the issue into avoidance at some appropriate scale.

I did the transposing and obtain the result you showed, which is good. What does that do to my downstream 1x1 convolution? Look forward to news on reshaping later.

Hmmm. Reactivity. Yes, some things need to respond immediately and others don’t. Within the settings panel, I think any dependencies should be ~“actioned” immediately, but the model update could be deferred.

Suggestion: (when you have user prefs available) - restore the apply button & add preference to “Apply changes immediately” (in which case all change events trigger the Apply action) or “Apply Changes on Demand”. Then since it is always the user’s choice the user can never complain that you are assuming you always know best (as I regularly complain about MS).

[robertl]

I like that suggestion, thanks!

The downstream components should not be affected more than their input looking similar to how it does in the Reshape component, or did you have something specific in mind?

[JulianSMoore]

Convolution was the initial concern: the dimensions the kernel would be slid along… if the reshape transposes, how do I make sure my convolution e.g. 2D is on the 28 x 28 “face”?

And I’m even more confused now I see that in “fixing” the reshape to eliminate the 3 x 3 grid, the labels of the dimensions are now Z, Y, X with sizes 3, 28, 28 so given that the permutation has an effect in one place, but the “dimensions” look the same, I don’t know what to expect elsewhere.

The inputs of the two reshapes are connected to the same output (of Merge_4)

[robertl]

What exactly happens with the permutation may be easier to understand by looking at the custom code.
It is moving the dimensions around, but it’s also reshaping them in a different order before that so that the output dimension will be the same.
Maybe there is a better way to represent this setting
We don’t do any magic on the dimensions above the component or the previews (besides showing just one channel if there are too many), so what you see there is exactly what will go into the Conv.

Does that clear it up a bit?

[JulianSMoore]

@robertl I’ll give an explanation shortly, but first to note that if you look carefully at the image in post 5 you will see that the digit is rotated and mirrored. This is probably partly due to the shape dimension sequence which is Z Y X, where the Y-X is an inversion of the “natural” order X-Y (but the details escape me )

Unfortunately, if one then swaps X &Y one gets this:

This looks like a bug to me. Is it?

As to the other part (neglecting dimension 0 which is used for batches): if I concatenate three 784 element lists (each of which is in fact a linearised 28 x 28 array I have [[1…784], [1…784], [1…784]] which has shape [3, 784]. I am trying to reshape this to give me a conceptual pseudo rgb 28 x 28 x 3, which must in fact be a tensor of shape [3, 28, 28].

Now, I then tried something pretty pointless, but it broke something. In the experiments above, I was concatenating the mnist digit 3 times, then attempting to reshape. This time I reshaped immediately, and then concatenated the 28 x 28 shapes to get a shape labelled 28 x28 x 3 (this is where labelling conflicts may be causing confusion) and then I asked perceptilable to reshape that to… X, Y, Z = 28, 28, 3 - and this happened, despite the fact that setting Z = 3 caused percepilabs to say it would no longer auto update settings.

Where and why 4 x 7 x 1? (yes, 4 x 7 = 28, I get that, but…) This looks like a different bug. Is it?

It’s late now - apologies for my own inevitable errors and omissions…

[robertl]

Thanks a lot as always for digging into it so much!

The first one with the strange preview may be a bug, this entire concatenating and reshaping has made me question a bit if we do it in the best way, if nothing else we could probably do it more intuitive.
I’ve added a task to look into the reshape some extra. Hopefully we can get to it before the next release and then I will have some more details for you there as well.

For the second one, you are saying that it auto-updated to 4x7x1, even though you had manually changed something?
That is wrong at two levels then and definitely a bug
I’ve added it to our bug list.

[JulianSMoore]

Yes, I’m pretty sure there was updating when I had manually changed something, but only (?) in the edge case where there was a shape mismatch and it was in fact impossible to reshape as specified… I think

[JulianSMoore]

@robertl Here’s a reproducible way to see reshape settings being overwritten even when they were set manually (and this time I am using perceptilabs-gpu 0.11.7)

• Get local data & use the object recognition dataset 224 x 224 x 3
• Set up a reshape - it defaults to 28 x 28 x 3 so setting to 224 x 224 x 3 is a manual update
• (Link the reshape to the local data)
• Copy & paste the reshape - note that the x, y, z values are still 224 x 224 x 3
• Link output of local data to the pasted reshape input - x, y, z become 14 x 16 x 1 (factorisation of 224)

[Side note: mnist digits have (0, 0) at the top left - but not all graphics formats do. Is percepilabs “format aware” so that within it, images are the “right way” up? I believe PNG and JPEG are different, for example]

[robertl]

That’s fantastic, thank you!