A list,

1 PCA 1.1 Data dimension reduction Methods include principal component analysis (PCA), factor analysis (FA), and independent component analysis (ICA). Factor analysis: Independent component analysis:

1.2 PCA: The purpose is to reduce dimension. The actual principle of dimension reduction is to maximize the objective function (the variance after data projection is maximum).

Strong Push Principle blog post:Blog.csdn.net/fendegao/ar…

(1) Suppose there are m n-dimensional samples: {Z1,Z2… ,Zm}

(2) Sample center U is the sum of observed values of all samples /(MXN)

(3) After decentralization, the matrix {X1,X2… , Xm} = {Z1 -u, Z2 – U,… ,Zm-U}

(4) Remember the vector W with n elements, then the projection of sample X1 in the W direction is the inner product of the two, X1.w

(5) The objective function of PCA is to maximize projection



The objective equation can be solved in matrix form by:

(1) Construct the Lagrangian operator, take the derivative to 0, and find that the vector with the largest projection is the eigenvector corresponding to the largest eigenvalue.

According to the cumulative contribution rate of the eigenvalues, you can specify how many W vectors to select as the K-L transformation matrix. If 4 principal components are selected, each n-dimensional sample becomes (1xn) x(nx4)=1×4 vector after matrix transformation, namely, the purpose of dimension reduction is achieved.

(2) SVD singular value decomposition: Dimension reduction requires only the right singular matrix, namely the eigenvector of AA(T), and does not require the covariance matrix of A. Memory friendly.

1.3 FACE recognition based on PCA (1) Based on the face sample database, such as real face photography (bank, station) and other data acquisition methods, to establish a face database. (2) The eigenvalues and eigenvectors of the covariance matrix of the training face database are obtained. (3) For the face that needs to be discriminated, judge its projection on the feature vector and which training sample projection is closest.

!!!!!!!!! Note: : It should be noted that the covariance matrix is the covariance between dimensions, so it is the NXN dimension. However, in practical application, for example, image dimension reduction (assuming that an image has 200*10 pixels and 100 images), a pixel is a dimension, then the original covariance matrix XX ‘is (2000×100) x (100×2000) dimension. At this time, the substitution matrix P=X ‘X(100×2000) X(2000×100) can be considered to replace:



The eigen value of P is the eigen value of the original covariance matrix, and the eigen vector of P left multiplied by the data matrix is the eigen vector of the original covariance matrix.

LDA LDA: Linear discriminant analysis, also known as Fisher’s linear discriminant, is a common dimension reduction technique. Basic idea: The high-dimensional pattern samples are projected into the optimal discriminant vector space to extract classification information and compress the dimension of feature space. After projection, the maximum interclass distance and minimum intra-class distance of the pattern samples in the new subspace are guaranteed, that is, the pattern has the best separability in the space. LDA dimension reduction of dimension is directly related to the number of categories, and the dimension of the data itself is it doesn’t matter, such as the original data is n, a total of C category, so the LDA dimension reduction, dimension values range for C (1, 1), for example, assume that image classification, two categories are example, each image with 10000 d character said, Then, after LDA, there are only 1-dimensional features, and this dimension has the best classification ability. For many cases of two-category classification, there is only one dimension left after LDA, and it seems to be enough to find the threshold that works best. Let’s say that x is an n-dimensional column vector, and to get x down to C by LDA, all you have to do is find a projection matrix w, which is an N by C matrix, and take the transpose of W and multiply it by x, and it becomes C. (The mathematical representation of a projection is multiplied by a matrix.) The key here is to find a projection matrix! Moreover, the projection matrix should ensure that the pattern samples have the largest interclass distance and the smallest intra-class distance in the new subspace.

2.2 Mathematical representation of LDA:













Ii. Source code

clear all
clc
close all
start=clock;
sample_class=1:40; Sample_classnum =size(sample_class,2); % Number of sample categoriesfprintf('The program starts.................... \n\n');

for train_samplesize=3:8;
    train=1:train_samplesize; % For each training sample test= train_SAMplesize +1:10; Train_num =size(train,2); Test_num =size(test,2); % Number of test samples per class Address =[PWD'\ORL\s']; Allsamples =readsample(address,sample_class,train); % using PCA for dimension reduction [newSample Base]= PCA (allsamples,0.9);
    %计算Sw,Sb
    [sw sb]=computswb(newsample,sample_classnum,train_num);
    
    %读取测试样本
    testsample=readsample(address,sample_class,test);
    best_acc=0; % Optimal recognition rate % Find the best projection dimensionfor temp_dimension=1:1:length(sw) vsort1=projectto(sw,sb,temp_dimension); % training sample and testsample are projected respectively tstsample=testsample*base*vsort1; trainsample=newsample*vsort1; Accuracy =computaccu(tstsample,test_num,trainsample,train_num);ifaccuracy>best_acc best_dimension=temp_dimension; % Save best projection dimension best_acc=accuracy; End end % -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- output shows -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --fprintf('The number of training samples for each type is: %d\n',train_samplesize);
    fprintf('The best projection dimension is: %d\n',best_dimension);
    fprintf('FisherFace recognition rate: %.2f%%\n',best_acc*100);
    fprintf('Program runtime: %3.2fs\n\n',etime(clock,start)); End function [newsample basevector]=pca(patterns,num) % end function [newsample basevector]= PCA (patterns,num) %1The required feature number is num when num is greater than0Less than or equal to1Denotes that the energy of the eigennumber obtained is num % output: Basevector denotes the eigenvector corresponding to the maximum eigenvalue obtained, and newsample denotes the representation of the sample obtained under the mapping of Basevector %. [u v]=size(patterns); totalsamplemean=mean(patterns);for i=1:u
    gensample(i,:)=patterns(i,:)-totalsamplemean;
end
sigma=gensample*gensample';
[U V]=eig(sigma);
d=diag(V);
[d1 index]=dsort(d);
if num>1
    for i=1:num
        vector(:,i)=U(:,index(i));
        base(:,i)=d(index(i))^(- 1/2)* gensample' * vector(:,i);
    end
else
    sumv=sum(d1);
    for i=1:u
        if sum(d1(1:i))/sumv>=num
            l=i;
            break;
        end
    end
    for i=1:l
        vector(:,i)=U(:,index(i));
        base(:,i)=d(index(i))^(- 1/2)* gensample' * vector(:,i); End End function sample=readsample(address,classnum,num) % % input: address is the address of the sample to be read,classnum is the class of the sample to be read,num is the sample of each class; Allsamples =[]; image=imread([pwd'\ORL\s1_1.bmp']); [rows cols]=size(image); % Gets the number of rows and columns of the imagefor i=classnum
    for j=num
        a=imread(strcat(address,num2str(i),'_',num2str(j),'.bmp'));
        b=a(1:rows*cols);
        b=double(b);
        allsamples=[allsamples;b];
    end
end
Copy the code

Third, the operation result

Fourth, note

Version: 2014 a