Python Deep Learning – 5 minutes of fast tensors

“This is the 10th day of my participation in the First Challenge 2022. For details: First Challenge 2022.”

Hello, I’m Ding Xiaojie.

Let’s get back to the pointWhat is a tensorToday, we will look at several operation methods and characteristics of tensors.

All transformations in deep neural networks can be simplified to some tensor operations.

In the previous article we built the network by superimposing the Dense layer.

keras.layers.Dense(512, activation='relu')
Copy the code

This layer can be thought of as a function that inputs a 2D tensor and outputs another 2D tensor. The function is expressed as follows (where W: 2D tensor, b: vector).

There are three tensor operations:

• The dot product between the input tensor and the tensor W
• Addition between 2D tensors and vector b
• Relu operations. Relu (x) is the Max (x, 0)

Let’s look at each of these operations separately.

Element by element operation

The three operations above are element-by-element operations, that is, the operation applies independently to each element in the tensor. We first use Numpy to implement relu operations and tensor addition.

>>> import numpy as np
>>> x = np.array([[1, -2.3],
		  [4, -5.6]])
>>> y = np.array([[1, -2.3],
		  [4, -5.6]])
>>> z = x + y
>>> z
array([[  2,  -4.6],
       [  8, -10.12]])
>>> np.maximum(z, 0)
array([[ 2.0.6],
       [ 8.0.12]])
Copy the code

We can simply implement the above element-by-element relu ourselves.

>>> def simple_relu(x) :
        x = x.copy()
        for i in range(x.shape[0) :for j in range(x.shape[1]):
                x[i, j] = max(x[i, j], 0)
        return x

>>> simple_relu(z)
array([[ 2.0.6],
       [ 8.0.12]])
Copy the code

While the effects look the same, and the speeds are vastly different, the Numpy built-in functions have been optimized to hand over a lot of the operations to BLAS (basic linear algebra subroutines). BLAS is a low-level, highly parallel, efficient tensor operator, usually implemented in Fortran or C.

Tensor dot product & element-by-element product

The dot product operation, also known as the tensor product, is the most common and useful tensor operation. Unlike element-by-element operations, it merges the elements of the input tensor together. In Numpy, dot is used to implement the dot product.

Dot (vector, vector)

The dot product between two vectors is a scalar, and you can only do the dot product between vectors that have the same number of elements. Let’s look at an example.

>>> import numpy as np
>>> np.dot([1.2.3], [4.5.6])
32
Copy the code

Dot (matrix, vector)

The dot product of a vector x with a matrix y yields a vector in which each entry is the dot product between each column of x and y.

>>> np.dot([1.2], [[4.5.6],
                    [7.8.9]])
array([18.21.24])
Copy the code

Note: If nDIM of either tensor is greater than 1, then the dot operation is no longer symmetric, that is, dot(x, y) is not equal to dot(y, x).

In the example above, if you want to swap x and y, then the dimension of the vector has to be the same size as the 0th dimension of the matrix.

>>> np.dot([[4.5.6],
            [7.8.9]], [1.2.3])
array([32.50])
Copy the code

Dot (matrix, matrix)

The dot product between two matrices is the most commonly used. For two matrices X and Y, the dot product can be used only when X.shape [1] == Y.shape [0]. The result is a matrix (X.shape [0], y.shape[1]). The entry is the dot product between the rows of x and the columns of y

>>> np.dot([[1.2.3],
        [4.5.6]],
       [[1.2],
        [3.4],
        [5.6]])
array([[22.28],
       [49.64]])
Copy the code

The illustration below

Higher dimensional vector dot product

For tensor dot products of higher dimensions, the rules are similar to 2D tensors.

• (a, b, c, d) . (d,) -> (a, b, c)
• (a, b, c, d) . (d, e) -> (a, b, c, e)

Broadcast mechanism

If you add two tensors of different shapes, the smaller tensor is broadcast to match the shape of the larger one. Let’s look at an example.

>>> x = np.array([[1.2.3],
                  [3.4.5]])
>>> y = np.array([[1],
                  [3]])
>>> x + y
array([[2.3.4],
       [6.7.8]])
Copy the code

The final result returns a matrix of shape (2,3).

Tenseshape

Tensor deformation is changing the rows and columns of a tensor to get the desired shape. But the number of elements doesn’t change. Let’s look at an example.

>>> x = np.array([[1.2.3],
                  [4.5.6]])
>>> x
array([[1.2.3],
       [4.5.6]])
>>> x.shape
(2.3)
>>> y = x.reshape(3.2)
>>> y
array([[1.2],
       [3.4],
       [5.6]])
>>> y.shape
(3.2)
>>> z = x.reshape(6.1)
>>> z
array([[1],
       [2],
       [3],
       [4],
       [5],
       [6]])
>>> z.shape
(6.1)
Copy the code

Matrix transpose

Transpose is also a form of tensor deformation. To transpose a matrix is to swap rows and columns so that x[I, :] becomes x[:, I].

>>> x = np.array([[1.2.3],
                  [4.5.6]])
>>> x.shape
(2.3)
>>> y = np.transpose(x)
>>> y.shape
(3.2)
Copy the code

This article briefly describes the use of several tensor operations, there are many other uses are not involved, there is a need to read the relevant information to learn.

Deep Learning in Python, Fran? Ois Chaulet, translated by Zhang Liang

For those who are new to Python or want to get started with Python, you can follow the public account “Python New Horizons” to communicate and learn Python together. They are all beginners. Sometimes a simple question is stuck for a long time, but others may suddenly realize it with a little help. There are also nearly 1,000 resume templates and hundreds of e-books waiting for you to collect!