“This is the 21st day of my participation in the First Challenge 2022. For details: First Challenge 2022.”

@TOC

preface

Hello! Friend!!! ଘ(੭, ᵕ)੭ Nickname: Haihong Name: program monkey | C++ player | Student profile: Because of C language, I got acquainted with programming, and then transferred to the computer major, and had the honor to win some state awards, provincial awards… Has been confirmed. Currently learning C++/Linux/Python learning experience: solid foundation + more notes + more code + more thinking + learn English! Machine learning small white stage article only as their own learning notes for the establishment of knowledge system and review know why!

The articles

Matrix theory for Machine Learning (1) : Sets and Mappings

Matrix Theory for Machine Learning (2) : Definitions and Properties of linear Spaces

Matrix theory (3) : Bases and coordinates of linear Spaces

Matrix theory for Machine Learning (4) : Basis transformation and coordinate transformation

Matrix theory for Machine Learning (5) : Linear subspaces

Matrix theory (6) : Intersection and sum of subspaces

2.1 Euclidean space

2.1.1 Euclidean space definition

Definition 2.1

Let VVV be a linear space over the real number field RRR. If any two vectors α,β\alpha,\betaα,β in VVV correspond to a unique and definite real number according to a certain rule, they are written as (α,β)(\alpha,\beta)(α,β), And (α,β)(\alpha,\beta)(α,β) satisfies:

  • For any alpha, beta, ∈ V \ alpha, beta \ V in alpha, beta ∈ V, a (alpha, beta) = (beta, alpha) (, alpha, and beta) = (, beta, and alpha) (alpha, beta) = (beta, alpha)
  • For any α,β,γ∈V\alpha,\beta,\gamma\in Vα,β,γ∈V, Have (alpha + beta, gamma) = (alpha and gamma) + (beta, gamma) (, alpha + \ beta, and gamma) = (, alpha, and gamma) + (, beta, and gamma) (alpha + beta, gamma) = (alpha and gamma) + (beta, gamma)
  • For arbitrary k ∈ R, alpha, beta, ∈ vitamin k \ in R, \ alpha, beta \ in vitamin k ∈ R, alpha, beta ∈ V, (k alpha, beta) = k (alpha, beta) (k \ alpha, beta) = k (, alpha, and beta) (k alpha, beta) = k (alpha, beta)
  • For any alpha, alpha and alpha in V ∈ V ∈ V, a (alpha, alpha) or 0 (, alpha, and alpha) \ geq0 (alpha, alpha) or 0, if and only if the alpha = 0 \ alpha = 0 alpha = 0, (alpha, alpha) = 0 (, alpha, and alpha) = 0 (alpha, alpha) = 0

Say (alpha, beta) (, alpha, and beta) (alpha, beta) is a vector of alpha \ alpha alpha and beta \ beta beta inner product

The real linear space VVV that defines the inner product is called Euclidean space, or Euclidean space for short

Example 1

In NNN dimensional vector space RnR^nRn, for any vector α=(a1, A2… , an), beta = (b1, b2,… ,bn)\alpha=(a_1,a_2,… ,a_n),\beta=(b_1,b_2,… , b_n) alpha = (a1, a2,… , an), beta = (b1, b2,… , BN) Definition rules:


( Alpha. . Beta. ) = a 1 b 1 + a 2 b 2 + . . . + a n b n (\alpha,\beta)=a_1b_1+a_2b_2+… +a_nb_n

Try to show that RnR^nRn on the inner product (α,β)(\alpha,\beta)(α,β) into a Euclidean space

prove

Let α,β∈Rn\alpha,\beta\in R^nα,β∈Rn, where α=(a1, A2… , an), beta = (b1, b2,… ,bn)\alpha=(a_1,a_2,… ,a_n),\beta=(b_1,b_2,… , b_n) alpha = (a1, a2,… , an), beta = (b1, b2,… ,bn)

According to the definition of the law:


( Alpha. . Beta. ) = a 1 b 1 + a 2 b 2 + . . . + a n b n (\alpha,\beta)=a_1b_1+a_2b_2+… +a_nb_n


( Beta. . Alpha. ) = b 1 a 1 + b 2 a 2 + . . . + b n a n (\beta,\alpha)=b_1a_1+b_2a_2+… +b_na_n

get


( Alpha. . Beta. ) = ( Beta. . Alpha. ) (\alpha,\beta)=(\beta,\alpha)

To set a gamma ∈ Rn, gamma = (c1, c2,… ,cn)\gamma\in R^n,\gamma=(c_1,c_2,… And c_n) gamma ∈ Rn, gamma = (c1, c2,… , cn), there is


( Alpha. + Beta. . gamma ) = ( a 1 + b 1 ) c 1 + ( a 2 + b 2 ) c 2 + . . . . + ( a n + b n ) c n = a 1 c 1 + b 1 c 1 + a 2 c 2 + b 2 c 2 + . . . + a n c n + b n c n = ( a 1 c 1 + a 2 c 2 + . . . + a n c n ) + ( b 1 c 1 + b 2 c 2 + . . . b n c n ) = ( Alpha. . gamma ) + ( Beta. . gamma ) (\alpha+\beta,\gamma)=(a_1+b_1)c_1+(a_2+b_2)c_2+…. +(a_n+b_n)c_n\\ \quad\\ \quad\quad\quad\quad\quad =a_1c_1+b_1c_1+a_2c_2+b_2c_2+… +a_nc_n+b_nc_n\\ \quad\\ \quad\quad\quad\quad\quad =(a_1c_1+a_2c_2+… +a_nc_n)+(b_1c_1+b_2c_2+… b_nc_n)\\ \quad\\ \quad\quad\quad\quad\quad =(\alpha,\gamma)+(\beta,\gamma)

Set k ∈ fairly Rk \ in fairly Rk ∈ R, there is


( k Alpha. . Beta. ) = k a 1 b 1 + k a 2 b 2 + . . . + k a n b n = k ( a 1 b 1 + a 2 b 2 + . . . + a n b n ) = k ( Alpha. . Beta. ) (k\alpha,\beta)=ka_1b_1+ka_2b_2+… +ka_nb_n=k(a_1b_1+a_2b_2+… +a_nb_n)=k(\alpha,\beta)

For any α∈Rn\alpha\in R^nα∈Rn


( Alpha. . Alpha. ) = a 1 2 + a 2 2 + . . . + a n 2 (\alpha,\alpha)=a_1^2+a_2^2+… +a_n^2

easy


( Alpha. . Alpha. ) = a 1 2 + a 2 2 + . . . + a n 2 p 0 (\alpha,\alpha)=a_1^2+a_2^2+… +a_n^2\geq0

If and only if α=0\alpha=0α=0, i.e. a1=a2=…. =an=0a_1=a_2=…. =a_n=0a1=a2=…. = the an = 0, (alpha, alpha) = 0 (, alpha, and alpha) = 0 (alpha, alpha) = 0

To sum up, RnR^nRn is a Euclidean space with respect to the inner product (α,β)(\alpha,\beta)(α,β)

Example 2

Rn in NNN dimension x nR ^ {n * n} Rn * n, for any vector A = (aij) n * n, B = (bij) n * nA = (a_ {ij}) _ {n * n}, B = (b_ {ij}) _ {n * n} A = (aij) n * n, B = (bij) n * n, defined rules


( A . B ) = ∑ i = 1 n ∑ j = 1 n a i j b i j (A,B)=\sum^n_{i=1}\sum^n_{j=1}a_{ij}b_{ij}

There are


( A . B ) = ∑ i = 1 n ∑ j = 1 n a i j b i j = T r ( A B T ) (A,B)=\sum^n_{i=1}\sum^n_{j=1}a_{ij}b_{ij}=Tr(AB^T)

Supplement knowledge

Tr(A)Tr(A)Tr(A) : The trace of the matrix A

Let A=(aijA=(a_{ij}A=(aij) be an NNN square matrix (that is, A matrix of N × NN ×nn×n), and the sum of the diagonal elements of AAA is called the trace of AAA

Write it as trA or Tr(A)trA or Tr(A)trA or Tr(A), i.e


T r ( A ) = a 11 + a 22 + . . . + a n n Tr(A)=a_{11}+a_{22}+… +a_{nn}

Because,

a_{11} & a_{12} &… & a_{1n}\\ a_{21} & a_{22} & … &a_{2n}\\ . & . & & . \\ . & . & & . \\ a_{n1} & a_{n2} &… & a_{nn}\\ \end{bmatrix}$$ $$B=\begin{bmatrix} b_{11} & b_{12} &… & b_{1n}\\ b_{21} & b_{22} & … &b_{2n}\\ . & . & & . \\ . & . & & . \\ b_{n1} & b_{n2} &… & b_{nn}\\ \end{bmatrix},B^T=\begin{bmatrix} b_{11} & b_{21} &… & b_{n1}\\ b_{12} & b_{22} & … &b_{n2}\\ . & . & & . \\ . & . & & . \\ b_{1n} & b_{2n} &… & b_ {nn} {bmatrix} \ \ \ end so $$$$(AB) ^ T = \ begin a_ {bmatrix} {11} & a_ {12} &… & a_{1n}\\ a_{21} & a_{22} & … &a_{2n}\\ . & . & & . \\ . & . & & . \\ a_{n1} & a_{n2} &… & a_{nn}\\ \end{bmatrix}\begin{bmatrix} b_{11} & b_{21} &… & b_{n1}\\ b_{12} & b_{22} & … &b_{n2}\\ . & . & & . \\ . & . & & . \\ b_{1n} & b_{2n} &… & b_{nn}\\ \end{bmatrix}=… $$$(AB^T)$is a $n by n$matrix, $\sum^n_{I =1}\sum^n_{j=1}a_{ij}b_{ij} \sum^n_{j=1} \sum^n_{j=1}a_{ij}b_{ij}$ Here is no longer verified in $$\ sum ^ n_ (I = 1} \ sum ^ n_ a_ {j = 1} {the ij} b_ {ij} = Tr (AB) ^ T $$- (1) for any $A, B \ in R ^ {n * n} $, A $$(A, B) = Tr (AB) ^ T = Tr ((AB) ^ T ^ T) = Tr (BA) ^ T = (B, A) $$> note: $A,B,C\in R^{n×n}$, $A,B,C\in R^{n×n}$ Have the $$(A + B, C) = Tr ((A + B) C ^ T) = Tr (AC BC ^ ^ T + T) = Tr (AC) ^ T + Tr (BC) ^ T = (A, C) + (B, C) $$> note: $Tr (K + Q) = Tr (K) + Tr (Q) $(3) for any $K \ in R, \ quad A, B \ in R ^ {n * n} $, have $$(kA, B) = Tr ((kA) ^ T B) = Tr (kAB) ^ T = kTr (AB) ^ T = K (A, B) $$> note: $A\in R^{n×n}$$A\in R^{n×n} Have $$(A, A) = \ sum ^ n_ (I = 1} \ sum ^ n_ a_ {j = 1} {the ij} a_ {ij} \ geq0 $$$A = if and only if \ boldsymbol0 $and $a_ {ij} = 0 $, $(A, A) = 0 $in conclusion, $R^{n×n}$the inner product $(A,B)$becomes A Euclidean space ### for any $\alpha\in V(V is the Euclidean space)$ $$(0,\alpha)=(\alpha,0)=0$$> $(0, \ alpha) = (0 \ alpha, \ alpha) = 0 (, alpha, and alpha) = 0 $, in particular, if $\ alpha $for any vector $\ beta \ in V $, have $(, alpha, and beta) = 0 $, There will be a $\ alpha = 0 $> when $\ take $\ alpha beta $$, have $(, alpha, and alpha) = 0 $, also means that $\ alpha = 0 $` (2) ` for any $\ alpha, \ beta, \ gamma \ in V $, Constant has $$(\ gamma, \ \ beta alpha +) = (\ gamma, \ alpha) + (, gamma, and beta) $$> $(+ \ \ gamma, \ alpha beta) = (, alpha + \ beta, and gamma) = (, alpha, and gamma) + (, beta, and gamma) = (\ gamma, \ alpha) + (, gamma, and beta) $similar, For any $\alpha,\beta\in V, \ k\in R$, Have $$\ \ beta, k (alpha) = (k \ alpha, beta) = k (, alpha, and beta) = k (, beta, and alpha) $$` (3) ` for any vector $\ alpha_1, European space \ alpha_2,… ,\alpha_s,\beta_1,\beta_2,… ,\beta_t$and real $k_1,… ,k_s,l_1,… , l_t $, Have the $$(\ sum ^ s_ {I = 1} k_i \ alpha_i, \ sum ^ t_ {j = 1} l_j \ beta_j) = \ sum ^ s_ {I = 1} \ sum ^ t_ {j = 1} k_il_j (\ alpha_i, \ beta_j) $$# # 2.1.2 vector length Let $\alpha$be a vector in Euclidean space, and the arithmetic root $\ SQRT {(\alpha,\alpha)}$of the non-negative real number $(\alpha,\alpha) is called the length of $\alpha$, $\lvert \alpha \rvert$> $\lvert \alpha \rvert=\ SQRT {(\alpha,\alpha)}$### theorem 2.1.1 Let $V$be Euclidean, then for any $\alpha,\beta \in V$and $k\in R$, there is -homoicity: $\lvert k\alpha \rvert=\lvert k\ rvert\lvert alpha \rvert$- Non-negative: $\lvert k\alpha \rvert=\lvert k\ rvert\ alpha \rvert$ $\lvert \alpha \rvert\geq0$if and only if $\alpha=0$, $\lvert \alpha \rvert=0$-Schwartz inequality: $(\alpha,\beta)^2\leq(\alpha,\alpha)(\beta,\beta)$, if and only if $\alpha$is linearly dependent on $\beta$– If $\alpha,\ neq0$, $$\lvert \frac{\alpha}{\lvert \alpha \rvert}\ rvert=\frac{1}{\lvert \alpha \rvert}\lvert \alpha \rvert=1$$ $\frac{\alpha}{\lvert \alpha \rvert}$is the unit vector multiplying $\alpha$by the reciprocal of the length of the non-zero vector $\alpha$. Let $\alpha,\beta$be any two non-zero vectors in Euclidean space, $$Q=arccos\frac{(\alpha,\beta)}{\lvert \alpha \rvert \lvert \beta \rvert}$$ If the inner product $(\alpha,\beta) of two vectors $\alpha,\beta$=0, then $\alpha$is orthogonal to $\beta$. Write $\alpha \perp \beta$> The zero vector is orthogonal to any vector ### proposition 1 $$\lvert \alpha +\ beta \rvert \leq \lvert \alpha \rvert +\lvert \beta \rvert

prove


∣ Alpha. + Beta. ∣ 2 = ( Alpha. + Beta. . Alpha. + Beta. ) = ( Alpha. . Alpha. ) + 2 ( Alpha. . Beta. ) + ( Beta. . Beta. ) Or less ( Alpha. . Alpha. ) + 2 ( Alpha. . Alpha. ) ( Beta. . Beta. ) + ( Beta. . Beta. ) ( Schwartz inequality is used ) = ( Alpha. . Alpha. ) + 2 ( Alpha. . Alpha. ) ( Beta. . Beta. ) + ( Beta. . Beta. ) = ∣ Alpha. ∣ 2 + 2 ∣ Alpha. ∣ ∣ Beta. ∣ + ∣ Beta. ∣ 2 = ( ∣ Alpha. ∣ + ∣ Beta. ∣ ) 2 \qquad{\lvert \alpha + \beta \rvert}^2=(\alpha+\beta,\alpha+\beta)\\ \quad\\ =(\alpha,\alpha)+2(\alpha,\beta)+(\beta,\beta)\\ \quad \\ \ leq (, alpha, and alpha) + 2 \ SQRT {(), alpha, and alpha, beta, and beta)} + (, beta, and beta) (using the Schwartz inequality) \ \ \ quad \ \ =(\alpha,\alpha)+2\sqrt{(\alpha,\alpha)}\sqrt{(\beta,\beta)}+(\beta,\beta)\\ \quad\\ =\lvert \alpha \rvert^2+2\lvert \alpha \rvert\lvert \beta \rvert+\lvert \beta \rvert^2\\ \quad\\ =(\lvert \alpha \rvert + \lvert \beta \rvert)^2

You can get


+ ∣ Beta. ∣ +\lvert \beta \rvert

Note: (alpha, beta) 2 or less (alpha, alpha, beta, beta) ⇒ (alpha, beta) or less (alpha, alpha, beta, beta) (, alpha, and beta) ^ 2 \ leq (), alpha, and alpha, beta, and beta) , quad, Rightarrow, quad (, alpha, and beta) \ leq \ SQRT {(), alpha, and alpha, beta, and beta)} (alpha, beta) 2 (alpha, alpha) or less (beta, beta) ⇒ (alpha, beta) (alpha, alpha) or less (beta, beta)

Proposition 2


∣ Alpha. ∣ – ∣ Beta. ∣ Or less ∣ Alpha. – Beta. ∣ \lvert \alpha \rvert – \lvert \beta \rvert \leq \lvert \alpha – \beta \rvert

prove

The alpha = (alpha – beta) + beta \ alpha = (\ alpha – \ beta) + \ beta alpha = (alpha – beta) + beta, get it


∣ Alpha. ∣ = ∣ ( Alpha. – Beta. ) + Beta. ∣ Or less ∣ Alpha. – Beta. ∣ + ∣ Beta. ∣ \lvert \alpha \rvert=\lvert (\alpha – \beta) + \beta \rvert\leq\lvert \alpha – \beta \rvert + \lvert \beta \rvert

namely


∣ Alpha. ∣ Or less ∣ Alpha. – Beta. ∣ + ∣ Beta. ∣ \lvert \alpha \rvert\leq\lvert \alpha – \beta \rvert + \lvert \beta \rvert

Transposition to


∣ Alpha. ∣ – ∣ Beta. ∣ Or less ∣ Alpha. – Beta. ∣ \lvert \alpha \rvert – \lvert \beta \rvert \leq \lvert \alpha – \beta \rvert


∣ Alpha. – gamma ∣ Or less ∣ Alpha. – Beta. ∣ + ∣ Beta. – gamma ∣ \lvert \alpha – \gamma \rvert \leq \lvert \alpha – \beta \rvert + \lvert \beta – \gamma \rvert

prove


∣ Alpha. – gamma ∣ = ∣ ( Alpha. – Beta. ) + ( Beta. – gamma ) ∣ Or less ∣ Alpha. – Beta. ∣ + ∣ Beta. – gamma ∣ \lvert \alpha – \gamma \rvert = \lvert (\alpha – \beta) + (\beta – \gamma) \rvert \leq \lvert \alpha – \beta \rvert + \lvert \beta – \gamma \rvert

Proposition 3

If α\alphaα is orthogonal to β\betaβ, Is ∣ alpha + beta ∣ 2 + 2 = ∣ alpha ∣ ∣ beta ∣ \ lvert \ \ beta alpha + 2 \ rvert ^ 2 = \ lvert \ alpha \ rvert ^ 2 + \ lvert \ beta \ rvert ∣ alpha ^ 2 + 2 = ∣ alpha beta ∣ ∣ ∣ beta ∣ 2 + 2

prove


∣ Alpha. + Beta. ∣ 2 = ( Alpha. + Beta. . Alpha. + Beta. ) = ( Alpha. . Alpha. ) + 2 ( Alpha. . Beta. ) + ( Beta. . Beta. ) \qquad{\lvert \alpha + \beta \rvert}^2=(\alpha+\beta,\alpha+\beta)\\ \quad\\ =(\alpha,\alpha)+2(\alpha,\beta)+(\beta,\beta)

Because alpha \ alpha alpha and beta \ beta beta orthogonal, so (alpha, beta) = 0 (, alpha, and beta) = 0 (alpha, beta) = 0, too


∣ Alpha. + Beta. ∣ 2 = ( Alpha. . Alpha. ) + ( Beta. . Beta. ) = ∣ Alpha. ∣ 2 + ∣ Beta. ∣ 2 \qquad{\lvert \alpha + \beta \rvert}^2=(\alpha,\alpha)+(\beta,\beta)=\lvert \alpha \rvert^2+\lvert \beta \rvert^2

namely


∣ Alpha. + Beta. ∣ 2 = ∣ Alpha. ∣ 2 + ∣ Beta. ∣ 2 \lvert \alpha + \beta \rvert^2=\lvert \alpha \rvert^2+\lvert \beta \rvert^2

conclusion

Description:

  • Refer to matrix Theory in your textbook
  • With the book concept explanation combined with some of their own understanding and thinking

The essay is just a study note, recording a process from 0 to 1

Hope to have a little help to you, if there is a mistake welcome small partners correct