1/ XOR Problem

The xOR problem is that if two inputs have the same Boolean value, the output is True (represented by 1), and if two inputs have the same Boolean value, the output is False (represented by 0).

Multilayer perceptrons (MLPS) are said to be difficult to deal with xOR problems, such as the one below.

First, we will introduce a question from The fourth chapter of The book neural Networks and Deep Learning by Professor Qiu Xipeng, exercise 4-2:

Problem 4-2 Tries to design a feedforward neural network to solve the XOR problem. It requires that the feedforward neural network has two hidden neurons and one output neuron, and uses ReLU as the activation function.

One possible outcome is the following,


W ( 1 ) = [ 1 1 1 1 ] . b ( 1 ) = [ 0 – 1 ] w ( 2 ) = [ 1 – 2 ] . b ( 2 ) = [ 0 ] (1) \boldsymbol{W}^{(1)}=\left[\begin{array}{ll}1 & 1 \\ 1 & 1\end{array}\right], \boldsymbol{b}^{(1)}=\left[\begin{array}{c}0 \\ -1\end{array}\right]\\ \boldsymbol{w}^{(2)}=\left[\begin{array}{c}1 \\ -2\end{array}\right], b^{(2)}=\left[0\right] \tag{1}

Therefore, the calculation of the entire network is:


y = ( w ( 2 ) ) T ( ReLU ⁡ ( ( W ( 1 ) ) T X + b ( 1 ) ) ) + b ( 2 ) \boldsymbol{y}=\left(\boldsymbol{w}^{(2)}\right)^{\mathrm{T}}\left(\operatorname{ReLU}\left(\left(\boldsymbol{W}^{(1)}\r ight)^{\mathrm{T}} \boldsymbol{X}+\boldsymbol{b}^{(1)}\right)\right)+b^{(2)}

Substitution:


X = [ 0 0 1 1 0 1 0 1 ] \boldsymbol{X}=\left[\begin{array}{llll}0 & 0 & 1 & 1 \\ 0 & 1 & 0 & 1\end{array}\right]

\left[\begin{array}{LLLL} 0&1&1&0 \end{array}\right]y=[0110] \boldsymbol{y}=\left[\begin{array}{LLLL} 0&1&1&0 \end{array}\right]y=[0110]

2/ Use Flux training

2.1/ Two implicit neurons

In fact, if such a network structure is trained in the way of random initialization, it is not good to train. The reason is that a ReLU activation function is required in the middle, which can be trained well if other activation functions are used.

Use the following code:

using Flux

functionLoss () ŷ = MLP (data) Flux. Mse (ŷ, Y)end

cb = function ()
    println(loss())
end

data = Array([[0 1 0 1];
              [0 0 1 1]]);
y = Array([[0 1 1 0]; ] ); mlp = Chain(Dense(2.2, relu), Dense(2.1));
ps = Flux.params(mlp);

opt = ADAM(0.01)
@timeFlux.train! (loss, ps, Iterators.repeated((),1000), opt, cb=cb)
Copy the code

After the training, the losses would still be high and the output would all be close to 0.50.50.5.

If we set the parameters of the hidden layer to the result of equation (1), and then train only the weight of the output layer w(2)\ boldSymbol {w}^{(2)}w(2), we get the same result:

# Custom weight, initialize the weight to all 1 matrix
mlp = Chain(Dense(2.2, relu, bias=[0; -1], init=ones),
            Dense(2.1, bias=zeros(1), init=ones))
Just take out the third parameter, the weight training of the output layer
ps = Flux.params(Flux.params(mlp)[3])
opt = ADAM(0.1)
@timeFlux.train! (loss, ps, Iterators.repeated((),1000), opt, cb=cb)
Copy the code

The result is w(2)=[0.9999…−1.9999…] ^ \ boldsymbol {w} {} (2) = / 0.9999-1.9999… W (2) = (0.9999-1.9999…) It is the same as the problem.

2.2/ Three implicit neurons

But actually, if we use three neurons on the hidden layer, we can solve this problem. This proves that the power of the model is not enough to use only two implicit neurons, and can be solved by adding one more:

mlp = Chain(Dense(2.3, relu), Dense(3.1));
ps = Flux.params(mlp)
opt = ADAM(0.01)
@timeFlux.train! (loss, ps, Iterators.repeated((),1000), opt, cb=cb)
# loss = 0.22230548
# loss = 0.21818444
#...
# loss = 0.0ŷ = MLP (data)Copy the code

The final solution is correct, as you can see by looking at the parameters of each layer


W ( 1 ) = [ 0.574309 – 0.574309 0.92754 – 0.966212 1.12378 – 1.12138 ] . b ( 1 ) = [ 0.5743128 – 0.00046141824 – 0.0034916808 ] w ( 2 ) = [ – 1.74122 0.60053 1.2883 ] . b ( 2 ) = [ 1.0000067 ] ^ \ boldsymbol {W} {} (1) = \ left [\ begin {array} {ll} 0.574309 & 0.574309 \ \ 1.12378 & 0.92754 & 0.966212 \ \ 1.12138] {array} \ \ end right, \ boldsymbol ^ {b} {} (1) = \ left [\ begin {array} {c} \ \ 0.00046141824 \ \ 0.0034916808\0.5743128 end] {array} \ right \ \ \ boldsymbol ^ {w} {} (2) = \ left [\ begin {array} {c} – 1.74122 \ \ \ \ 1.2883\0.60053 end {array} \ right]. B ^ {} (2) = \ left 1.0000067 \ [right]

When two implicit neurons are used, it is very difficult to solve the training by randomly initializing the weight, but if three implicit neurons are used, the power is enough to solve the xOR problem.

However, in fact, when two hidden neurons are used, instead of using ReLU as the activation function, Sigmoid function is used at the hidden layer, so the xOR problem can be solved even when only two hidden neurons are used.

The main problem should be that ReLU will directly truncate the part less than zero, which means that the neuron is not activated, which will easily cause the neuron to “die” and cannot continue training.