This is the 17th day of my participation in the First Challenge 2022

This article is my share of notes, mainly from ng’s deep learning course. 1

An interesting feature of Word embedding is that it helps to realize analogical reasoning.

Take a chestnut

Here we still use the example table we used in the previous section:


 Man  ( 5391 )  Woman  ( 9853 )  King  ( 4914 )  Queen  ( 7157 )  Apple  ( 456 )  Orange  ( 6257 )  Gender  – 1 1 – 0.95 0.97 0.00 0.01  Royal  0.01 0.02 0.93 0.95 – 0.01 0.00  Age  0.03 0.02 0.7 0.69 0.03 – 0.02  Food  0.04 0.01 0.02 0.01 0.95 0.97 . . . . . . . . . . . . . . . . . . . \begin{array}{c|cccccc} & \begin{array}{c} \text { Man } \\ (5391) \end{array} & \begin{array}{c} \text { Woman } \\ (9853) \end{array} & \begin{array}{c} \text { King } \\ (4914) \end{array} & \begin{array}{c} \text { Queen } \\ (7157) \end{array} & \begin{array}{c} \text { Apple } \\ (456) \end{array} & \begin{array}{c} \text { Orange } \\ (6257) \end{array} \\ hline \text {Gender} &-1 & 1 & -0.95&0.97&0.00 & 0.01 \\ text {Royal} &0.01&0.02&0.93& 0.95 & 0.01 & 0.00 \ \ \ text {Age} & 0.03 & 0.02 & 0.7 & 0.69 & 0.03 & 0.02 \ \ \ text {Food} & 0.04 & 0.01 & 0.02 & 0.01&0.95&0.97 \\ \text {… } &… &… &… &… &… &… \end{array}

  • Now let’s ask a question:

    Man corresponds to woman. So what does king correspond to?

  • We should reasonably know the answer:

    Queen

Automatic inference can be realized by using Word embedding.

In the last video we extracted 300 features, but we’re still using the first four.

  • Man:

e 5391 = [ – 1 0.01 0.03 0.09 ] E_ {5391} = \ left [\ begin {array} {c} – 1 \ \ \ \ \ \ \ 0.09 0.03 0.01 end {array} \ right]
  • Woman:

e 9853 = [ 1 0.02 0.02 0.01 ] E_ {9853} = \ left [\ begin {array} {c} 1 \ \ \ \ \ \ \ 0.01 0.02 0.02 end {array} \ right]
  • King:

e 4914 = [ – 0.95 0.93 0.70 0.02 ] E_ {4914} = \ left [\ begin {array} {c} – 0.95 \ \ \ \ \ \ 0.70 0.93 to 0.02] {array} \ \ end right
  • Queen:

e 7157 = [ 0.97 0.95 0.69 0.01 ] E_ {7157} = \ left [\ begin {array} {c} \ \ \ \ \ \ 0.69 0.95 0.97 0.01] {array} \ \ end right

Now we can find an interesting thing by subtracting this:


e m a n – e w o m a n = e 5391 – e 9853 = [ – 1 0.01 0.03 0.09 ] – [ 1 0.02 0.02 0.01 ] material [ – 2 0 0 0 ] – e_ e_ {man} {woman} = e_ {5391} – e_ {9853} = \ left [\ begin {array} {c} – 1 \ \ \ \ \ \ \ 0.09 0.03 0.01 end {array} \ right] – [\ \ left the begin {array} {c} 1 \ \ \ \ \ \ 0.02 0.02 0.01] {array} \ \ end right \ approx \ left [\ begin {array} {c} – 2 \ \ \ \ \ \ 0 0 0 \end{array}\right]

Also:


e k i n g – e q u e e n = e 4914 – e 7157 = [ – 0.95 0.93 0.70 0.02 ] – [ 0.97 0.95 0.69 0.01 ] material [ – 2 0 0 0 ] E_ {king} – e_ {queen} = e_ {4914} – e_ {7157} = \ left [\ begin {array} {c} – 0.95 \ \ \ \ \ \ 0.70 0.93 to 0.02] {array} \ \ end right – \ left [\ begin {array} {c} \ \ \ \ \ \ 0.69 0.95 0.97 0.01] {array} \ \ end right \ approx \ left [\ begin {array} {c} – 2 \ \ \ \ \ \ 0 0 0 \end{array}\right]

In addition:


e a p p l e – e o r a n g e = e 456 – e 6257 = [ 0.00 – 0.01 0.03 0.95 ] – [ 0.01 0.00 – 0.02 0.97 ] material [ 0 0 0 0 ] E_ {} apple – e_ = e_ {orange} {456} – e_ {6257} = \ left [\ begin {array} {c} \ \ \ \ \ \ 0.01 0.03 0.00 0.95] {array} \ \ end right – \ left [\ begin {array} {c} \ \ \ \ \ \ 0.02 0.00 0.01 0.97] {array} \ \ end right \ approx \ left [\ begin {array} {c} \ \ \ \ \ \ 0 0 0 0 \end{array}\right]

From the above three expressions, we can conclude that the main difference between man and woman lies in gender. The difference between King and Queen is also mainly in gender. There was no difference in any of the four characteristics. Because they don’t care about gender, power, age, and they’re all food.

algorithm

“Man corresponds to woman. So what does king correspond to?”

So how do we apply this to the algorithm.


e m a n – e w o m a n material e k i n g – e ? e_{man}-e_{woman} \approx e_{king}-e_{? }

Back to our picture below:

Remember we talked about 300 features. These 300 features are then mapped to a two-dimensional planar graph by nonlinear mapping.

I did two calculations for the last part, man minus woman, king minus Queen. They all came up with a 2 in the first dimension and a 0 in all the other dimensions. This is what the visualization looks like.

The calculation we’re going to do is:


e m a n – e w o m a n + e k i n g material e ? e_{man}-e_{woman} + e_{king}\approx e_{? }

namely


a r g m a x s i m ( e m a n – e w o m a n + e k i n g . e ? ) argmax \quad sim(e_{man}-e_{woman} + e_{king}, e_{? })

Added!

We’re using the T-SNE algorithm and adding 300 dimensional vectors to a two-dimensional plane. This step performs a nonlinear mapping. So we need to know. Differences due to attribute differences show the same vector orientation on the image, such as man and woman, king and Queen. But if you look at the picture. The direction of the vector arrow is the same, it does not mean that the difference properties of the two are the same. One and three, for example, are not a man and a woman.

Similarity function is commonly used


a r g m a x s i m ( e m a n – e w o m a n + e k i n g . e ? ) argmax \quad sim(e_{man}-e_{woman} + e_{king}, e_{? })

I said that what we’re ultimately going to calculate is this formula, so which similarity function are we going to use.

We usually do this using cosine similarity or using Euclidean distance.

Cosine similarity:


sin ⁡ ( u . v ) = u ⊤ v ∥ u ∥ 2 ∥ v ∥ 2 \sin (u, v)=\frac{u^{\top} v}{\|u\|_{2}\|v\|_{2}}

Euclidean distance:


sin ⁡ ( u . v ) = ∥ u – v ∥ 2 \sin (u, v)=\|u-v\|^{2}

In a different space. Measurements. The method of calculation is different. We’re going to use these two more here.

  1. DeepLearning AI China – the world’s leading online AI education and practice platform (deeplearningai.net)