Today I will introduce a Github project, the project address is as follows, it realizes the automatic coloring function of line draft, the effect is very good, let’s take a look.

Github.com/SerialLain3…

Introduction to the

This project mainly realized the automatic line draft into color picture function. Of course, we can train the neural network to handle the line draft only, but in practice we need the ability to color the line draft with the specified color in advance. There are many ways to implement coloring, including hints.

  • Don’t take tips
    • Coloring method without hints
    • Input: line draft only
  • Atari
    • A method of coloring with a hint, usually a line of desired color in a particular area (e.g. PaintsChainer)
    • Input: Line draft and Atari
  • The label
    • The hint is how the label is colored
    • Input: Line draft and label
  • reference
    • Paints using reference images as hints (e.g. Style2paints V1)
    • Input: line draft and reference image

Line extraction method

There are many improved versions of online extraction methods such as **XDoG ** or SketchKeras. However, if the model is only trained on one type of line draft, the model will overfit this type of line draft, so it cannot realize automatic coloring function for other types of line draft. Therefore, like Tag2Pix, a variety of different line drafts are used as training data for network training.

The following three types of lines are used:

  • XDoG: Line extraction is performed using the difference from two Gaussian distributions to standard deviation;
  • SketchKeras: Line extraction using UNet. The lines drawn in this way will look like pencil sketches;
  • Simplified version of Sketch: Continuous line extraction via full convolutional network. This method results in a similar number sketch.

The extraction results of the above three methods are shown below:

In addition, I also consider three data enhancement methods for line draft to prevent overfitting.

  • Increased strength;
  • Random morphological transformation deals with lines of different widths;
  • Random selection of RGB values to handle lines of different depths;

Experiments without hints

motivation

First, I needed to confirm that the neural network-based approach could be used to color accurately and variously without prompting. The difficulty is mainly in mapping from lines to color images, because colors vary. So, without being prompted, I think the neural network eventually learns to show a single color in any given region. To avoid falling into local minima, IN addition to content loss, I also add adversarial loss, because adversarial learning trains the neural network to color to more accurately match the distribution of data.

methods

  • pix2pix
  • Pix2pix-gp (Pix2PIx plus centrosymmetric gradient penalty)
  • pix2pixHD

The results of

  • pix2pix

  • pix2pix-gp

  • pix2pixHD

Experiments using Atari

motivation

Observing the above results, it is found that the neural network seems to fall into the local minimum even when the counter loss is added. Although there are varying degrees of color variation, the neural network can only learn to match a single color on a single character in any region. It was difficult to train line mapping to color images without hints, so I decided to add hints, atari, as input to the network (ps. As shown in the figure below, on the basis of the original line draft, a line with a specified color is added to a specific area to indicate the color needed in this part of the network.

methods

Added hints

The results of

Experiments using reference images

motivation

I also considered using reference images as hints to feed into the neural network. First, I tried to implement style2paints V1. However, due to the collapse of training, IT was difficult for me to reproduce the original experimental results. So I decided to look for an alternative to style2paints V1.

methods

  • style2paints

The results of

Video coloring experiment

The results of

Finally, give the address of the project again:

Github.com/SerialLain3…

Welcome to follow my public account – the growth of algorithmic ape, exchange and learn together.