Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

twitter

The question is, today I’m reading the fish book (Advanced Deep Learning: Natural Language Processing). See SVD decomposition: Dimension reduction of the matrix using Singular Value Decomposition (SVD).

Then I hand-wrote the code in the book, and the result was different from the output in the book. I was going to skip over this problem and leave it at that, because the book also drew a scatter plot with Matplotlib, so I thought I’d just draw it and see if it’s different. Turns out I got a bug in my drawing, too. Without graphs, I had to look it up to see if there was something wrong with my SVD breakdown.

Then I went to Baidu, Baidu to what SVD results and the book is not the same ah and PPT is not the same ah how, but in the end did not say why, so I had to write a while.

What is the SVD

SVD is to decompose a matrix into U, S and V matrices. Where U and V are matrices whose column vectors are orthogonal to each other. S is the diagonal matrix that is 0 except for the diagonal elements.

The schematic might look something like this:

Then the decomposition formula written in the previous example of Zhihu looks something like this:

The original answer is to use an example of the image, the more the number of items decomposed, the finer the image (can refer to the Lagrangian remainder ah, the finer the number of items).

What is the physical meaning of singular value? – zhihu (zhihu.com)

Back to the code

I’m a math junkie, so I don’t know exactly what caused the difference, but after looking at the formula above, my guess is that it might be the difference in the number of retained terms. So I’m not the same as the answer in the book.

So how do you verify that your results are correct? Use a simple [email protected](s)@v.

  • @Is the tensor multiplication in Pytorch*It’s bitwise.)
  • The s-matrix in the result is a diagonal matrix, but the vector given is a vector that needs to be usedtorch.diag()Convert it to a diagonal matrix

Try a piece of code for yourself:

import torch

# Generate a random 5*5 tensor
tensor = torch.rand(5.5)
# Use pyTorch's built-in SVD for decomposition
u,s,v = torch.linalg.svd(tensor)

Output source tensors, decomposed three tensors, and verify the result
print(f'tensor:\n{tensor}')
print(f'U:\n{u}\nS:\n{s}\nV\n{v}')
print(f'res:\n{[email protected](s)@v}')
Copy the code

My result looks like this, and you can see that tensor is the same as res.

Is your result likely to be different from your matrix?

Like the one below. The one below looks different at first glance. You have the tensor[0][0], you have a 0, you have 5.0037E-07, 0.00000050037, is that a big difference from 0… So the difference can be very subtle. You can still view the resulting matrix as the same as the original matrix.