Notebook, Graph - request for code to complete the notebook

V0.11.15

I previously asked (January) about the missing graph in jupyter notebook export and so did somebody else back in December

Until such time as notebook code is complete, would it be possible to provide the necessary code separately so that it is possible to paste it into a notebook to run a model?

Then, for example, I could run batches of models with variations in parameters quite simply.

You can build the graph by adding:

graph_builder = GraphBuilder()
graph = graph_builder.build(layer_classes,edges)

and adding brackets to the layer_classes e.g.:

layer_classes = {
    'TrainRegression_Regression_1': TrainRegression_Regression_1(),
    'DeepLearningFC_Dense_1': DeepLearningFC_Dense_1(),}

This will still not work for me. It looks like the graph is still not correctly set up, leading to empty input dicts for the layers.

This might have something to do with the conn_info dict that I couldn’t find a use for or simply that I have been too negligent with version compatibility while setting up perceptilabs.

@Marc

Thanks; I’m getting the same result - there’s no ‘input’ key - but that is a step forward!


KeyError                                  Traceback (most recent call last)
<ipython-input-13-88d0146715ac> in <module>
     38 sentinel = object()
     39 while result is not sentinel:
---> 40     result = next(iterator, sentinel)

<ipython-input-8-91d92389f8da> in run(self, graph, mode)
    392         """  
    393 
--> 394         self.init_layer(graph, mode)
    395         self._variables = {k: v for k, v in locals().items() if can_serialize(v)}
    396 

<ipython-input-8-91d92389f8da> in init_layer(self, graph, mode)
    158             return layer_output_tensors
    159 
--> 160         layer_output_tensors = build_graph(input_tensor, label_tensor)
    161 
    162         output_tensor = None

<ipython-input-8-91d92389f8da> in build_graph(input_tensor, label_tensor)
    152                 y = dst_node.layer_instance(
    153                     inputs,
--> 154                     is_training=is_training
    155                 )
    156                 layer_output_tensors[dst_node.layer_id] = y

<ipython-input-3-5ca818f68db9> in __call__(self, inputs, is_training)
     14                 permutation.append(i)
     15 
---> 16         x = inputs['input']
     17         input_shape = x.get_shape().as_list()
     18         new_shape = [input_shape[0] if input_shape[0] is not None else -1] + shape

KeyError: 'input'

From what I can tell the remaining error comes from the graph.get_input_connections() function not doing what it should. This helper function does what I think the original function was intended for. At least replacing them allows me to run the notebook without errors.

def get_input_connections(graph,dst_node,conn_info):
    output = []

    #iterate over input nodes of dst_node to get [src_node, src_var, dst_var] for every connection
    for node in graph.get_input_nodes(dst_node):
        src_node = node
        dict_key = f"{node.layer_id}:{dst_node.layer_id}"
        src_var,dst_var = conn_info[dict_key][0]

        output.append([src_node,src_var,dst_var])

    return output

But there is still no training happening. As far as I can tell there is no code that actually changes the mode to training or testing. At this point this doesn’t feel like a ‘Quickfix’ anymore, so I guess there’s no help waiting until the devs fix code generation.

But this does make me interested in what is actually different in the tool compared to the module import since this seems more than just a few missing lines.

Maybe I need to try exporting a model again: there is no def get_input_connections in the notebooks I have looked at. I see calls to graph.get_input_connections but not the definition.

(I also note that I find no get_input_nodes either, and there’s probably a get_output_nodes to be found too; & interesting observation about mode changing.)

Which version did you use for export? I’m on 0.11.15 but I suspect my ipynb exports are from 0.11.13 (but not certain - could be even earlier)

I’m keen to have working notebook exports too since with viable base code one can do a lot of things that the tool does not yet support/support easily.

Perhaps @robertl could give us some insight into the way things are and whether 0.12 will resolve this.

Sorry for the confusion I wasn’t clear enough on that. The code block with def get_input_connections is written by me as a quick replacement for the graph.get_input_connections in the exported code. So it is not surprising that it is not in your notebook if you haven’t added it yourself :slight_smile: .

The Graph class and its functions graph.get_input_connections, graph.get_input_nodes are imported from the perceptilabs module so they are not defined in the notebook itself. Im on 0.11.15.

:rofl: Yep, that would account for it :smiley:

From the PL module in site-packages? I thought I’d looked for insight there and not found any readable code… thx… I’ll look again.

I’d be very surprised if there were non compiled files in the in-site package. I’m just guesstimating what their code is supposed to do and where the error might be, but haven’t seen any actual source code from the package so don’t go on a wild goose chase on my behalf :smile:.

Can you clarify how def get_input_connections works?

It looks to me as though it should be a method of graph, but I don’t see how the helper function (defined in isolation) becomes a method of graph, as in (from the training component)

for src_node, src_var, dst_var in graph.get_input_connections(graph.active_training_node):

(graph is created by calling GraphBuilder()… What’s the source of that GraphBuilder? Have you defined the whole class?)

The function is not a method of graph nor does it become one. I wrote it as a replacement for the original function of the graph class, which did not return anything. (Hence the missing ‘input’ key error). I then replaced the original function in the code with my function(2 occurences for me). E.g.


The comment is the original generated code by perceptilabs.

The idea of the helper function was to return an iterable containing the input node and the two keys provided by the conn_info dictionary. This way I can run the notebook without errors, but I do not know for sure that this was the intended functionality, since I don’t know the source code.

GraphBuilder is part of the perceptilabs package I have nothing to do with this.

Ah, OK. I see what you did there… this has the pleasingly engaging touch of a detective story about it… I get very puzzled, a new clue is revealed in a timely manner. :smiley: Now I get what you’ve been doing I probably won’t be so mystified - and I shall have another look myself in the next few days. I have some catching up to do!

Small advance, small update

I didn’t like the idea of having to edit the code to call your function that should be a method of graph so I made it a method of graph. (Because: less typo prone, more reusable, etc.)

After the layer-classes, etc, definitions (and import types of course)

graph_builder = GraphBuilder()
graph = graph_builder.build(layer_classes,edges)

# make get_input_connections a method of the graph instance, see https://stackoverflow.com/questions/972/adding-a-method-to-an-existing-object-instance
graph.get_input_connections = types.MethodType(get_input_connections, graph)

We then get lots of TF warnings…

2021-05-04 08:11:00,319 - WARNING - From c:\users\julian\anaconda3\envs\pl_tf2_main\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py:2561: calling DatasetV2.from_generator (from tensorflow.python.data.ops.dataset_ops) with output_types is deprecated and will be removed in a future version.
Instructions for updating:
Use output_signature instead

and the issue at the end is:

> <ipython-input-8-91d92389f8da> in build_graph(input_tensor, label_tensor)
>     148                 inputs = {
>     149                     dst_var: layer_output_tensors[src_node.layer_id][src_var]
> --> 150                     for src_node, src_var, dst_var in graph.get_input_connections(dst_node)
>     151                 }
>     152                 y = dst_node.layer_instance(
> 
> TypeError: get_input_connections() missing 1 required positional argument: 'conn_info'

No doubt one of us will look at get_input_connections and the calls… :wink:

Of course, get_input_connections should no longer have graph as an argument…

NB I have belatedly realised why conn_info is required: I/O is to named pins, and not only can inputs be other than ‘input’, outputs can be other than ‘output’ (search the forum for “y_before” for examples)