Matrix advance-norm

As a fast food tutorial, we’ll do as much code as possible, more tools, and less principles and formulas. But I also know that it’s impossible to say, because of the complexity of the problem. After exploring Tensorflow with you, I must write a further tutorial with a solid mathematical foundation.

Norm for the first time

General undergraduate linear algebra textbooks do not talk about norms and generalized inverse knowledge, need to learn the matrix theory course. But unfortunately, deep learning is used a lot. So we have to have a basic concept.

Whether it’s a vector or a matrix, we often need a measure of its size in machine learning. The first thing we think about the measure of a vector is the length of the vector. A more cultural term is the Euclid distance. So this big distance is just the square root of the sum of the squares of all the values. We can call tF.norm with ord=’ Euclidean ‘to find the Euclidean norm. Ex. :

>>> a02 = tf.constant([1,2,3,4],dtype=tf.float32) >>> sess.run(tf.norm(a02, ord=' Euclidean ')) 5.477226Copy the code

There’s no mystery here, we can do it with SQRT:

> > > np. SQRT (1 * 1 + 2 * 2 + 3 + 4 * 4 * 3) 5.477225575051661Copy the code

Now we generalize the norm of vectors to matrices. I’m going to take the square root of the sum of squares.

> > > a03 = tf. Constant ([[1, 2], [3, 4]], dtype = tf. Float32) > > > a03 (tf) Tensor Const_34: '0' shape = (2, 2) dtype=float32> >>> sess.run(a03) array([[1., 2.], [3., 4.]], dtype=float32)Copy the code

We had a row of vectors, and now we have a 2 by 2 matrix, and we’re going to keep taking the norm. There is now a fancy name for the Frobenius norm.

> > > sess. Run (tf) norm (a03, word = 2) 5.477226Copy the code

Well, it’s still the same as the norm of the vector 1,2,3,4.

Definition of norm

The Euclidean norm and the Frobenius norm are only special cases of the norm. More generally, the norm is defined as follows:



Among them,

A norm is essentially a function that maps vectors to non-negative values. When p = 2,The norm is called the Euclidean norm. Because it’s used so much in machine learning, it’s generallyAbbreviated as.

More strictly, a norm is any function that satisfies the following property:

  2. (This is called triangle inequality.)

Generalization of norm

In addition toIn addition to norm, it is also commonly used in machine learningThe norm is just the sum of the absolute values of all the elements.

Sometimes we just want to figure out how many elements there are in a vector or matrix, which is also called the number of elementsNorm. However, this term is unscientific because it does not meet the third of the three definitions above. It is generally recommended to useNorm.

Let’s seeExamples of norms:

> > > sess. Run (tf) norm (a03, word = 1)) 10.0Copy the code

And then there’s another normNorm, also known as Max norm. The maximum norm represents the absolute value of the element with the largest value in a vector.

We can use ord= Np.INF to find the maximum norm.

> > > sess. Run (tf) norm (a03, word = np. J inf) 4.0Copy the code

Norm and normed space

Finally, let’s look at the strict mathematical definition of the norm. Given the concepts and code implementation outlined above, this definition is now easy to understand.

Define vector norm 1: Let V be a linear space over the number field F, and for any vector x of V, it corresponds to a non-negative real number, the following conditions are met:

  1. Is qualitative:.If and only if x=0
  2. Homogeneity:
  3. Triangle inequality: for any, there areIs said,Is the norm of vector x,Is a normed space.

Definition 2 matrix norm: supposeFor each A, if it corresponds to A real function N(A), denoted by, it satisfies the following conditions:

  1. The negative:, positive characterization:
  2. Homogeneity:
  3. Triangle inequality:, said N = (A)Is the generalized matrix norm of A. Further, if rightGeneralized matrix norm of the same kindThere are the following conclusions:
  4. Compatibility of matrix multiplication:Is said,Is the matrix norm of A.