TensorFlowAlthough the most popular neural network framework, it has a reputation for being “hard to get started” (Jeff Dean: Blame me). Sometimes, we need succinct code to point the way. Recently, Junho Kim from NCsoft’s AI research department released a set of such TensorFlow code. It is similar to a mini version of Keras, but the source code is much easier to read because of its simplicity.

Projects link: https://github.com/taki0112/Tensorflow-Cookbook

In this project, the authors highlight an easy-to-use set of TensorFlow code that includes common regularization, convolution operations, and architecture modules. In fact, when we build our own model or system, we just copy and paste the code. They define different functional modules in a canonical form, so with a few parameters and code changes, they fit perfectly into our project.

The current project includes code required for general deep learning architectures, such as initialization and regularization, various convolution operations, basic network architectures and modules, loss functions, and other data preprocessing processes. In addition, the support for GAN is especially added, which is mainly reflected in the loss function, where the generator loss and discriminator loss can use bulldozer distance, least square distance and KL divergence, etc.

Method of use

There are actually two ways to use it. First, we can copy and paste the code, which is very helpful for module customization. Second, we can call operations and modules directly as if we were using the API. This method makes the model very concise and the imported source code is easy to understand. First of all, for the second direct import method, we can import the model operation part and image preprocessing process from the ops.py and utils.py files respectively.

  • from ops import *

  • from utils import *

from ops import convx = conv(x, channels=64, kernel=3, stride=2, pad=1, pad_type='reflect', use_bias=True, sn=True, scope='conv')Copy the code

For the first copy-and-paste, we might modify some of the parameters and structure as needed, but this is much easier than writing from scratch. As shown below, for a general neural network, it will adopt the following structural template:

def network(x, is_training=True, reuse=False, scope="network"): with tf.variable_scope(scope, reuse=reuse): x = conv(...) . return logitCopy the code

Deep neural networks are like building blocks. We stack the different modules of OPs.py on top of each other to get the full forward propagation.

Code set directory

Project page: www.notion.so/Simple-Tens…

The project currently consists of 20 code blocks that can be used to quickly model deep learning:

Code sample

Here are some code examples, including the most common convolution operations and residual modules. Each code example can be called apI-style or copy-pasted, so not only are they quick to use, but they’re also a great resource to learn how to implement the various operations.

convolution

I believe you are familiar with the principle of convolution, so let’s directly look at the call code:

x = conv(x, channels=64, kernel=3, stride=2, pad=1, pad_type='reflect', use_bias=True, sn=True, scope='conv')Copy the code

The code that implements the above API is shown below. It’s also good to know how to hand out the image padding zero instead of using the padding=’SAME’ directly. In addition, this code has been embedded with spectral_normalization/ SN, and we can even take a little bit of it and embed it in our own code.

# padding='SAME' ======> pad = ceil[ (kernel - stride) / 2 ]def conv(x, channels, kernel=4, stride=2, pad=0, pad_type='zero', use_bias=True, sn=False, scope='conv_0'): with tf.variable_scope(scope): if pad > 0: h = x.get_shape().as_list()[1] if h % stride == 0: pad = pad * 2 else: pad = max(kernel - (h % stride), 0) pad_top = pad // 2 pad_bottom = pad - pad_top pad_left = pad // 2 pad_right = pad - pad_left if pad_type == 'zero': x = tf.pad(x, [[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]) if pad_type == 'reflect': x = tf.pad(x, [[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]], mode='REFLECT') if sn: w = tf.get_variable("kernel", shape=[kernel, kernel, x.get_shape()[-1], channels], initializer=weight_init, regularizer=weight_regularizer) x = tf.nn.conv2d(input=x, filter=spectral_norm(w), strides=[1, stride, stride, 1], padding='VALID') if use_bias: Bias = tf.get_variable("bias", [channels], initializer=tf.constant_initializer(0.0)) x = tf.nn. Bias_add (x, bias) else: x = tf.layers.conv2d(inputs=x, filters=channels, kernel_size=kernel, kernel_initializer=weight_init, kernel_regularizer=weight_regularizer, strides=stride, use_bias=use_bias) return xCopy the code

Partial Convolution

Partial convolution is a convolution operation introduced by Nvidia for image repair, which enables the model to repair arbitrary non-central, irregular regions. In the paper Image Inpainting for Irregular Holes Using Partial Convolutions, the realization of Partial Convolutions is very critical, and the simple invocation process is shown as follows:

x = partial_conv(x, channels=64, kernel=3, stride=2, use_bias=True, padding='SAME', sn=True, scope='partial_conv')        Copy the code

The following code defines PConv for implementation information:

def partial_conv(x, channels, kernel=3, stride=2, use_bias=True, padding='SAME', sn=False, scope='conv_0'):    with tf.variable_scope(scope):        if padding.lower() == 'SAME'.lower():            with tf.variable_scope('mask') : _, h, w, _ = x.get_shape().as_list() slide_window = kernel * kernel mask = tf.ones(shape=[1, h, w, 1]) update_mask = tf.layers.conv2d(mask, filters=1, kernel_size=kernel, Kernel_initializer = tf. Constant_initializer (1.0), strides = stride, padding = padding, use_bias = False, Trainable =False) mask_ratio = slide_window/(update_mask + 1e-8) update_mask = tf.clip_by_value(update_mask, 0.0, Mask_ratio = mask_ratio * update_mask with tf.variable_scope('x') :if sn:                    w = tf.get_variable("kernel", shape=[kernel, kernel, x.get_shape()[-1], channels],                                        initializer=weight_init, regularizer=weight_regularizer)                    x = tf.nn.conv2d(input=x, filter=spectral_norm(w), strides=[1, stride, stride, 1], padding=padding)                else:                    x = tf.layers.conv2d(x, filters=channels,                                         kernel_size=kernel, kernel_initializer=weight_init,                                         kernel_regularizer=weight_regularizer,                                         strides=stride, padding=padding, use_bias=False)                x = x * mask_ratio                if use_bias:                    bias = tf.get_variable("bias", [channels], initializer= tf.constant_Initializer (0.0)) x = tf.nn. Bias_add (x, bias) x = x * update_maskElse:if sn:                w = tf.get_variable("kernel", shape=[kernel, kernel, x.get_shape()[-1], channels],                                    initializer=weight_init, regularizer=weight_regularizer)                x = tf.nn.conv2d(input=x, filter=spectral_norm(w), strides=[1, stride, stride, 1], padding=padding)                if use_bias:                    bias = tf.get_variable("bias", [channels], initializer=tf.constant_initializer(0.0)) x = tf.nn. Bias_add (x, bias)else:                x = tf.layers.conv2d(x, filters=channels,                                     kernel_size=kernel, kernel_initializer=weight_init,                                     kernel_regularizer=weight_regularizer,                                     strides=stride, padding=padding, use_bias=use_bias)        return x   Copy the code

Residual module

ResNet’s greatest feature is that it solves the problem of gradient disappearance during backpropagation, so it can train very deep networks without having to add a classification network in the middle to provide additional gradients as GoogLeNet does. ResNet is stacked with residual modules, and several different residual modules can be defined as required:

x = resblock(x, channels=64, is_training=is_training, use_bias=True, sn=True, scope='residual_block')x = resblock_down(x, channels=64, is_training=is_training, use_bias=True, sn=True, scope='residual_block_down')x = resblock_up(x, channels=64, is_training=is_training, use_bias=True, sn=True, scope='residual_block_up')Copy the code

Three types of residual modules are shown above, in which down represents down-sampling and the length and width of the input feature graph are halved. While up represents ascending sampling, the length and width of the input feature graph are doubled. On each residual module, the residual link directly adds the inputs and outputs of that module. Therefore, in the back propagation, the gradient transmitted by the residual connection does not go through multiple convolution layers inside the residual module, so enough gradient information can be reserved for the previous layer.

The following is a simple definition of resBlock in general and resblock_UP with ascending sampling, because the functions they call, such as conv(), deconv(), and batch_norm(), are different calculation modules defined earlier, so the overall code looks very clean.

def resblock(x_init, channels, use_bias=True, is_training=True, sn=False, scope='resblock'):    with tf.variable_scope(scope):        with tf.variable_scope('res1') : x = conv(x_init, channels, kernel=3, stride=1, pad=1, use_bias=use_bias, sn=sn) x = batch_norm(x, is_training) x = relu(x) with tf.variable_scope('res2'):            x = conv(x, channels, kernel=3, stride=1, pad=1, use_bias=use_bias, sn=sn)            x = batch_norm(x, is_training)        return x + x_initdef resblock_up(x_init, channels, use_bias=True, is_training=True, sn=False, scope='resblock_up'):    with tf.variable_scope(scope):        with tf.variable_scope('res1') : x = deconv(x_init, channels, kernel=3, stride=2, use_bias=use_bias, sn=sn) x = batch_norm(x, is_training) x = relu(x) with tf.variable_scope('res2') :            x = deconv(x, channels, kernel=3, stride=1, use_bias=use_bias, sn=sn)            x = batch_norm(x, is_training)        with tf.variable_scope('skip') :            x_init = deconv(x_init, channels, kernel=3, stride=2, use_bias=use_bias, sn=sn)Copy the code

The code implementation for only three functional blocks is shown here, and we might feel that the project is similar to a mini Keras. But because the operations implemented in this project are relatively simple and common, the source code will be much easier to read than a large library like Keras, which is an advantage for embedding and learning.