An overview of the

In short, neural networks are functions: input data, output results.

function

We take MNIST handwritten digital image recognition as an example to define the corresponding functional form:

  • Task type: Image classification
  • Input: An image contains 28 x 28=784 pixels, each pixel represented by a real number
  • Output: 0-9
  • Task description: Identify unique numbers from an image sheet
  • The function definitions

y=f(x1,x2,… ,x784)y = f(x_1, x_2, … , x_{784})y=f(x1,x2,… Xi, x784) ∈ R, I = 1,… ,784x_i \in \mathbb R, i = 1, … , 784 xi ∈ R, I = 1,… , 784 y ∈ {0, 1,… ,9}y \in \{0, 1, … , 9 \} y ∈ {0, 1,… 9},

This is an entry-level application of neural networks. The input is a low resolution (28 x 28) black and white image, corresponding to a function with 784 input variables. For a megapixel color image, there are 3 million input variables.

Thus, the neural network is used to solve some complex problems, the corresponding is also complex functions. The implementation of the algorithm is to construct the corresponding function.

How do you construct such a complex function? We can start with simple functions, and the simplest and most successful example is digital circuits.

The digital circuit

Digital circuits are the building blocks of computers, building our vast digital world. But its core is made up of and, or, non – logic gates.

What is a logic gate? It’s just a function. And these are the simplest functions.

Logic gate expression Functional form
With the doorAND
x Sunday afternoon y x \land y
z = Sunday afternoon ( x . y ) z = \land(x, y)
Or gateOR
x ∨ y x \lor y
z = ∨ ( x . y ) z = \lor(x, y)
gateNOT
such x \lnot x
y = such ( x ) y = \lnot(x)
  • {T,F}\{T, F \} {T,F \} {T,F \} {T,F \} {T,F}.
  • Number of variables: a function of one or two variables, also the simplest form of a function
  • Function representation: Use the truth table for description. Why not use images? Because it’s a discrete function, it’s just isolated points on the graph, which is not very nice.

Gate NOT


x x
such x \lnot x

T T
F F

F F
T T

The door OR the door


x x
y y
x Sunday afternoon y x \land y
x ∨ y x \lor y

T T
T T
T T
T T

T T
F F
F F
T T

F F
T T
F F
T T

F F
F F
F F
F F

With the door image

Or gate image

Combined logic gate

Simple logic gates can be combined to produce more powerful functions.

Construct a new binary function:

  • Xor: the radius x = y (Sunday afternoon x y) ∨ (such Sunday afternoon y x) x \ oplus y = (x/land/lnot y) \ lor (\ lnot x \ land y) radius x = y (Sunday afternoon x y) ∨ (such Sunday afternoon x y)

  • With or: ⊖ y = x (x Sunday afternoon y) ∨ (such Sunday afternoon x y) \ ominus y = x (x \ land y) \ lor (\ \ land \ lnot y lnot x) x ⊖ y = (x Sunday afternoon y) ∨ (such Sunday afternoon x y)

To construct a function of several variables:

  • And three: f (x, y, z) = x Sunday afternoon Sunday afternoon z y f (x, y, z) = x \ land y \ land government (x, y, z) = x Sunday afternoon Sunday afternoon y z

  • 8 bit adder: f(x1,… ,x8,y1,… ,y8)f(x_1, … , x_8, y_1, … , y_8)f(x1,… ,x8,y1,… ,y8)

    • It is a 16-element function with 16 Boolean variables

Complex functions can be constructed by composition of simple functions. The same idea applies to neural networks.

The neural network

Neural networks are functions. Like digital circuits, it is composed of simple functions. The basic unit of digital circuit is and, or, nand gate, while the basic unit of neural network is neuron.

neurons

So what is a neuron? A biological neuron is a cell with an input dendrite and an output axon. And the neuron in the neural network is an artificial neuron, and it’s also a function, or more accurately, it’s a class of functions.

The number of neuron inputs is variable, that is, it represents an NNN element function f(x1,… ,xn)f(x_1, … , x_n)f(x1,… Xn), and the NNN of different neurons can be different.

The neural network

Neurons combine with each other to form a neural network. As shown in the figure below:

The neural network consists of three neurons:


  • f 1 ( x 1 . . . . . x n ) f_1(x_1, … , x_n)

  • f 2 ( x 1 . . . . . x n ) f_2(x_1, … , x_n)

  • f 3 ( x 1 . x 2 ) f_3(x_1, x_2)

The function represented by neural network is: f(x1,… ,xn)=f3(f1(x1,… ,xn),f2(x1,… ,xn))f(x_1, … , x_n) = f_3(f_1(x_1, … , x_n), f_2(x_1, … , x_n))f(x1,… ,xn)=f3(f1(x1,… ,xn),f2(x1,… ,xn))

The problem

What is the function that the neuron represents?

It is not enough to know that it is an NNN element function. The basic units and, or, and nand gates in digital circuits can all list the truth table and draw the image. But what about neurons?

Reference software

Interactive chart version, please refer to: