“This is the 10th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”

preface

Improvements in normalization techniques are one of the many improvements in Generative Adversarial Networks (GAN). This article introduces Pixel normalization, which is commonly used in current GAN. Known as pixel normalization) and Spectral normalization, both techniques have been used extensively in high-definition image generation, culminating in pixel normalization and Spectral normalization using Tensorflow2.

Pixel normalization

Detail of pixel normalization

Pixel normalization was proposed in the ProGAN model, where the authors of ProGAN abandoned batch normalization and used custom normalization for generators, i.e., pixel normalization.

The purpose of normalization in ProGAN is to limit the weight value to prevent it from growing exponentially. Larger weights may increase signal amplitude and lead to vicious competition between generators and discriminators. Pixel normalization Normalizes the features of each pixel position (H, W) in the channel size. If the tensor is a batch RGB image of size (N, H, W, C), the size of the RGB vector of any pixel after pixel normalization will be 1.

Pixel normalization implementation

In Tensorflow2, pixel normalization can be achieved using a custom layer:

from tensorflow.keras.layers import Layer
class PixelNorm(Layer) :
    def __init__(self, epsilon=1e-8) :
        super(PixelNorm, self).__init__()
        self.epsilon = epsilon
    
    def call(self, input_tensor) :
        return input_tensor / tf.math.sqrt(tf.reduce_mean(input_tensor ** 2, axis=-1, keepdims=True) + self.epsilon)
Copy the code

Unlike other normalization, pixel normalization does not have any parameters to learn. It consists only of simple arithmetic operations, so it is very efficient.

Spectrum normalization

Spectrum normalization detailed solution

In order to explain spectral normalization, we first need to review linear algebra to explain roughly what a spectral norm is.

Firstly, the eigenvalues and eigenvectors in matrix theory are discussed:


A v = Lambda. v Av=\lambda v

Where AAA is a square matrix, VVV is an eigenvector, and λ\lambdaλ is its eigenvalue.

We’ll use a simple example to understand these terms. Suppose VVV is a vector about position (x,y)(x, y)(x,y) and AAA is a linear transformation:


A = ( a b c d ) . v = ( x y ) A=\begin{pmatrix} a & b\\ c & d\\ \end{pmatrix},v=\begin{pmatrix} x \\ y \\ \end{pmatrix}

If we multiply AAA times VVV, we get a new position with the direction changed as follows:


A v = ( a b c d ) x ( x y ) = ( a x + b y c x + d y ) Av=\begin{pmatrix} a & b\\ c & d\\ \end{pmatrix}\times \begin{pmatrix} x \\ y \\ \end{pmatrix}=\begin{pmatrix} ax + by\\ cx + dy\\ \end{pmatrix}

Eigenvectors are vectors that don’t change direction when you apply A to vectors. Instead, they can be scaled only by scalar eigenvalues λ\lambdaλ. There can be multiple eigenvector – eigenvalue pairs. The square root of the maximum eigenvalue is the spectral norm of the matrix. For a non-square matrix, we will need to use a mathematical algorithm (such as Singular Value Decomposition (SVD)) to compute the eigenvalues, which can be computationally expensive.

Therefore, the power iteration method can speed up the computation and make it feasible for neural network training. Next, spectrum normalization is implemented in TensorFlow as a weight constraint.

Spectrum normalization implementation

The mathematical algorithm for spectral normalization may seem complicated. Often, however, the algorithmic implementation is simpler than it looks mathematically.

Here are the steps to perform spectrum normalization:

  1. The weight in the convolution layer is a 4-dimensional tensor, so the first step is to reshape it into a 2D matrix, where we keep the last dimension of the weight. After remolding, the shape of the weight is (H×W, C).

  2. Initialize vector uuu with N(0,1)N(0,1)N(0,1).

  3. In the for loop, the following is evaluated:

    A) Calculate v=(wT)uv =(w^T)uv=(wT)u using matrix transpose and matrix multiplication.

    B) use its normalization VVV L2L_2L2 norm, namely the v = v / ∣ ∣ n ∣ ∣ 2 v = v / | | v | | _2v = v / ∣ ∣ v ∣ ∣ 2.

    C) Calculate u= wVU =wvu =wv.

    D) use L2L_2L2 norm normalized uuu, u = u / ∣ ∣ u ∣ ∣ 2 u = u / | | u | | _2u = u / ∣ ∣ u ∣ ∣ 2.

  4. The spectrum norm is calculated as uTwvu^TwvuTwv.

  5. Finally, divide the weight by the spectral norm.

The complete code is as follows:

import tensorflow as tf
class SpectralNorm(tf.keras.constraints.Constraint) :
    def __init__(self, n_iter=5) :
        self.n_iter = n_iter
    def call(self, input_weights) :
        w = tf.reshape(input_weights, (-1, input_weights.shape[-1]))
        u = tf.random.normal((w.shape[0].1))
        for _ in range(self.n_iter):
            v = tf.matmul(w, u, transpose_a=True)
            v /= tf.norm(v)
            u = tf.matmul(w, v)
            u /= tf.norm(u)
        spec_norm = tf.matmul(u, tf.matmul(w, v),    transpose_a=True)
        return input_weights/spec_norm
Copy the code

The number of iterations is a hyperparameter, and generally 5 iterations is sufficient. Spectral normalization can also be implemented with a variable to hold the vector UUu instead of starting with random values. This reduces the number of iterations to 1. To achieve spectral normalization, we can apply spectral normalization by using it as a convolution kernel constraint, such as:

Conv2D(3.1,kernel_constraint = SpectralNorm())
Copy the code