This article continues our discussion of the source structure for Keras.

In the first source note we looked at how Layer, Tensor and Node are coupled together, but in this series we’ll focus on a directed acyclic graph (DAG) of a multi-layer network. The main file involved is keras/engine/topology.py, and the class to look at is Container.

ContainerObject: Topology prototype of the DAG

In the first episode we said that the enhanced \_keras_history property in Keras Tensor allows us to build the whole thing just by taking the input and output Tensor. The Container object does just that.

Computational graph construction

DAG diagram construction is completed when the Container object is instantiated. It consists of the following operations:

1) Record connection information about containers

def __init__(self, inputs, outputs, name=None):
  for x in self.outputs:
      layer, node_index, tensor_index = x._keras_history
      self.output_layers.append(layer)
      self.output_layers_node_indices.append(node_index)
      self.output_layers_tensor_indices.append(tensor_index)

  for x in self.inputs:
      layer, node_index, tensor_index = x._keras_history
      self.input_layers.append(layer)
      self.input_layers_node_indices.append(node_index)
      self.input_layers_tensor_indices.append(tensor_index)Copy the code

2) Recursively construct the calculation diagram from output_tensors, using the width-first criterion. The key of this step is to construct the queue nodes_in_decreasing_depth. The connection information and depth information contained in these nodes will be the basis for subsequent forward propagation and reverse training of calculation execution order.

def build_map_of_graph(tensor, finished_nodes, nodes_in_progress): layer, node_index, Tensor_index = tensor._keras_history node = layer.inbound_nodes[node_index] nodes_in_progress.add(node) # breadth-first search for I in range(len(node.inbound_layers)): x = node.input_tensors[i] layer = node.inbound_layers[i] node_index = node.node_indices[i] tensor_index = Build_map_of_graph (x, finished_nodes, nodes_in_progress, layer, node_index, Tensor_index) # Maintain two queues finished_nodes.add(node) nodes_in_progress.remove(node) nodes_in_decreasing_depth. Append (node) # Outputs: build_map_of_graph(x, finished_nodes, nodes_in_progress)Copy the code

3) Calculate the depth of each node and calibrate the position of the node in DAG according to the depth

For node in reversed(nodes_in_decreasing_depth): depth = nodes_depths.setdefault(node, 0) previous_depth = layers_depths.get(node.outbound_layer, 0) depth = max(depth, previous_depth) layers_depths[node.outbound_layer] = depth nodes_depths[node] = depth for i in range(len(node.inbound_layers)): inbound_layer = node.inbound_layers[i] node_index = node.node_indices[i] inbound_node = inbound_layer.inbound_nodes[node_index] previous_depth = nodes_depths.get(inbound_node, 0) nodes_depth [inbound_node] = Max (depth + 1, previous_depth) # nodes_by_depth = {} for node, p = 0) NOdes_depth (depth + 1, previous_depth) depth in nodes_depths.items(): if depth not in nodes_by_depth: Nodes_by_depth [depth] = [] nodes_by_depth[depth]. Append (node) nodes_by_depth = {} depth in layers_depths.items(): if depth not in layers_by_depth: layers_by_depth[depth] = [] layers_by_depth[depth].append(layer) self.layers_by_depth = layers_by_depth self.nodes_by_depth = nodes_by_depthCopy the code

4) Merge the entire Container into Node for compatibility

  self.outbound_nodes = []
  self.inbound_nodes = []
  Node(outbound_layer=self,
       inbound_layers=[],
       node_indices=[],
       tensor_indices=[],
       input_tensors=self.inputs,
       output_tensors=self.outputs,
       ...)Copy the code

Calculate the calculation in the diagram

The calculation is done in the call() method of the Container object, which in turn relies on the internal method run_internal_graph().

def run_internal_graph(self, inputs, masks=None): Depth_keys = list(self.nodes_by_depth. Keys ()) depth_keys.sort(reverse=True) # for depth in depth_keys: Nodes = self.nodes_by_depth[depth] # for nodes in nodes: Layer = node.outbound_layer # node Reference_input_tensors = node.input_tensors sors reference_output_tensors = node.output_tensors computed_data = [] if len(computed_data) == len(reference_input_tensors): With k.nam_scope (layer.name): if len(computed_data) == 1: computed_tensor, computed_mask = computed_data[0] output_tensors = _to_list(layer.call(computed_tensor, **kwargs)) computed_tensors = [computed_tensor] else: computed_tensors = [x[0] for x in computed_data] output_tensors = _to_list(layer.call(computed_tensors, **kwargs)) output_tensors = [] output_masks = [] for x in self.outputs: tensor, mask = tensor_map[str(id(x))] output_tensors.append(tensor) output_masks.append(mask) return output_tensors, output_masksCopy the code

As you can see from the code above, the calculations are based on depth, and the entire sequence is traversed by updating computed_data and output_tensor variables.

Continue reading article 3 of this series: Source Notes: Keras’s Model for source analysis

@ddlee

@ddlee

This article followsCreative Commons Attribution-ShareAlike 4.0 International License.

This means that you may reprint this article by name and attach this agreement.

If you want regular updates on my blog posts, feel free to subscribeDongdong monthly report.

Links to this article: Blog. Ddlee. Cn/posts/ba611…


Related articles

  • Keras source code analysis Model

  • Keras source code analysis Layer, Tensor, and Node

  • “Westworld” Rhapsody (with spoilers)

  • Some thoughts and puzzles about deep learning

  • Weight attenuation in deep learning