Introduction of particle swarm optimization and RBF
The swarm behavior of birds and fish in nature has always been the research interest of scientists. Biologist Craig Reynolds proposed a highly influential model of bird clustering in 1987. In his simulation, each individual follows the following rules: avoid collisions with neighboring individuals; match the speed of neighboring individuals; Fly to the center of the flock, and the whole flock flies to the target. Using only the above three simple rules, we can simulate the phenomenon of bird flock flight very closely. In 1990, biologist Frank Heppner also proposed a bird model with a difference: birds are attracted to habitats. In the simulation, each bird has no specific flight target at the beginning, but only uses simple rules to determine its flight direction and speed. When one bird flies to the habitat, the birds around it will also fly to the habitat, and eventually the whole flock will fall into the habitat. In 1995, American social psychologist James Kennedy and electrical engineer Russell Berhart jointly proposed ParticleS Warm Optimization (PSO), The proposed algorithm is inspired by the research results of modeling and simulating the group behavior of birds. Their model and simulation algorithm mainly modified Frank Heppner’s model to make particles fly to the solution space and land at the optimal solution. Particle swarm optimization (PSO) has attracted extensive attention and become a research hotspot in the field of evolutionary computing due to its simplicity and easy implementation. Swarm Intelligence, co-authored by J.Kennedy and R.Eberhart in 2001, further expanded the influence of swarm intelligence [], followed by a large number of research reports and results on particle swarm optimization algorithm, which set off a research boom at home and abroad [2-7]. Particle swarm optimization (PSO) algorithm is derived from the regularity of bird swarm activity, and a simplified model is established by using swarm intelligence. It simulates the foraging behavior of birds, compares the search space for solving the problem to the flight space of birds, abstracts each bird into a particle without mass and volume, and uses it to represent a possible solution of the problem. The process of finding the optimal solution of the problem is regarded as the process of birds looking for food, and then solves the complex optimization problem. Particle swarm optimization (PSO), like other evolutionary algorithms, is also based on the concept of “population” and “evolution”, and achieves the search of optimal solution in complex space through cooperation and competition among individuals. At the same time, it is not like other evolutionary algorithms for individual evolution operator operation such as crossover, mutation, selection, but the individual in the group as there is no quality and volume in l d search space of particles, each particle moves in a certain speed in the solution space, and to its own historical best position P best and neighborhood historical best position g best gathered, Implement the evolution of candidate solutions. Particle swarm optimization (PSO) has a good biological social background and is easy to understand. Due to its few parameters and easy implementation, pSO has strong global search ability for nonlinear and multi-peak problems, and has been widely concerned in scientific research and engineering practice. At present, the algorithm has been widely used in function optimization, neural network training, pattern classification, fuzzy control and other fields.
1.2 Theory of Particle swarm Algorithm 1.2.1 Particle swarm algorithm describes birds in the process of predation, members of a flock can obtain the discovery and flight experience of other members through information exchange and sharing between individuals. In a world where food sources are sporadic and unpredictable, the advantages of such collaboration are decisive, far outweighing the disadvantages of competition for food. Particle swarm optimization (pso) and inspired by the behavior of birds feed on to mimic this behavior, will the searching space of optimization problems in birds flying space, will every bird abstracted as a particle, particle no quality, no volume, used to characterize the a feasible solution of the problem, optimization problem to search for the optimal solution is equal to birds for food source. Particle swarm algorithm formulates simple behavior rules for each particle similar to bird movement, so that the movement of the whole particle swarm shows characteristics similar to bird predation, so as to solve complex optimization problems. The information sharing mechanism of particle swarm optimization algorithm can be interpreted as a symbiotic cooperative behavior, that is, each particle is constantly searching, and its searching behavior is affected by other individuals in the group to varying degrees [8]. Meanwhile, these particles also have the ability to remember the best position they have experienced. That is, their search behavior is not only influenced by other individuals but also guided by their own experience. Based on the unique search mechanism, pSO firstly generates the initial population, that is, randomly initializes the velocity and position of particles in the feasible solution space and velocity space, in which the position of particles is used to represent the feasible solution of the problem, and then solves the optimization problem through the cooperation and competition of individual particles among the population. Particle swarm optimization algorithm is derived from the research on the predation behavior of birds: a group of birds randomly search for food in the area, and all birds know how far they are from the food, so the simplest and effective search strategy is to search the area around the nearest bird to the food. Particle swarm optimization (PSO) is applied to solve optimization problems. In particle swarm optimization, the potential solution to each optimization problem is a bird in the search space, called a particle. All particles have a fitness value determined by the optimized function, and each particle has a velocity that determines the direction and distance it flies. The particles then follow the current optimal particle to search the solution space [9].
Particle swarm optimization (PSO) firstly initializes the particle swarm randomly in a given solution space, and the dimension of the knowledge space is determined by the number of variables of the problem to be optimized. Each particle has its initial position and velocity, and then iteratively optimizes it. In each iteration, each particle by tracking two “extreme” to update their spatial location in the solution space and speed: a extreme value is a single particles themselves to find the optimal solution in the process of iterative particle, the particle called the individual extremum: another extreme value is population all particles in the process of iteration to find the optimal solution of the particle, the particle is the global extremum. The method described above is called global particle swarm optimization. If we do not use all the particles of the population but only use some of them as the neighbor particles of the particle, then the extreme value in all the neighbor particles is the local extreme value, which is called the local particle swarm optimization algorithm.
1.2.3 Characteristics of PARTICLE Swarm Optimization Algorithm Particle swarm optimization algorithm is essentially a random search algorithm, it is a new intelligent optimization technology. The algorithm can converge to the global optimal solution with high probability. It is proved by practice that it is suitable for dynamic and multi-objective optimization environment, and has faster calculation speed and better global search ability compared with the traditional optimization algorithm. (1) Particle swarm optimization algorithm is an optimization algorithm based on swarm intelligence theory. It guides optimization search through swarm intelligence generated by cooperation and competition among particles in the population. Compared with other algorithms, particle swarm optimization is an efficient parallel search algorithm. (2) Both particle swarm optimization and genetic algorithm randomly initialize the population, using the adaptive value to evaluate the merits of the individual and carry out a certain random search. However, particle swarm optimization (PSO) decides the search according to its own speed without the crossover and variation of genetic algorithm. Compared with the evolutionary algorithm, particle swarm optimization retains the global search strategy based on population, but its speed-displacement model is simple to operate and avoids complex genetic operation. (3) Since each particle still maintains its individual extreme value at the end of the algorithm, that is, particle swarm optimization algorithm can not only find the optimal solution of the problem, but also obtain several better sub-optimal solutions, so the application of particle swarm optimization algorithm to scheduling and decision problems can give a variety of meaningful solutions. (4) The unique memory of particle swarm optimization enables it to dynamically track the current search situation and adjust its search strategy. In addition, particle swarm optimization is not sensitive to the size of the population.
1.3 Types of particle swarm optimization 1.3.1 Elementary particle swarm optimization 
When between 1.2, particle swarm optimization algorithm has faster convergence speed; When w>1.2, the algorithm tends to fall into local extremum. In addition, w can be dynamically adjusted in the search process: at the beginning of the algorithm, w can be given a large positive value, and with the progress of the search, W can be linearly reduced, which can ensure that at the beginning of the algorithm, each particle can explore a better region in the global range with a large speed step. In the later stage of the search, a smaller value of W ensures that particles can conduct a fine search around the extreme point, so that the algorithm has a greater probability of convergence to the global optimal solution position. Adjusting W allows you to balance global and local search capabilities. At present, the linear decreasing weight strategy proposed by Shi adopts more dynamic inertia weight values, and its expression is as follows:50. For most problems 10 particles is a good result: but for more difficult problems or for certain types of problems, the number of particles can go up to 100 or 200. In addition, the larger the number of particles is, the larger the search space of the algorithm is, and it is easier to find the global optimal solution. Of course, the longer the algorithm runs.Inertial weight WThe inertial weight W is a very important control parameter in the standard particle swarm optimization algorithm, which can be used to control the development and exploration ability of the algorithm. The magnitude of the inertial weight indicates how much is inherited from the particle’s current velocity. When the inertia weight value is large, the global optimization ability is strong, and the local optimization ability is weak: When the inertia weight value is small, the global optimization ability is weak, and the local optimization ability is strong. The choice of inertia weight usually includes fixed weight and time varying weight. Fixed weight means that constant is selected as the inertia weight value, and its value remains unchanged in the evolution process, with a general value of [0.8, 1.2]. Time-varying weight is to set a certain change interval and gradually reduce the inertia weight in a certain way in the evolution process. The selection of time – varying weights includes range of change and decline rate. A fixed inertial weight allows the particle to maintain the same exploration and exploitation capacity, while a time-varying weight allows the particle to have different exploration and exploitation capacity at different stages of evolution.The acceleration constants c1 and C2The acceleration constants C and C2 adjust the maximum stride length to P best and G best, respectively. They determine the influence of individual particle experience and group experience on particle trajectory, respectively, and reflect the information exchange between particle swarm. If Cr =c2=0, the particle will fly to the boundary at its current flight speed. At this point, the particle can only search a limited area, so it is difficult to find the optimal solution. If q=0, it is a “social” model, in which the particle lacks cognitive ability and only has group experience. Its convergence speed is fast, but it is easy to fall into local optimum. If OY =0, it is a “cognitive” mode. There is no shared information of the society and no information interaction between individuals, so the probability of finding the optimal solution is small. A group with a size of D is equivalent to running N independent particles. So c1=C2, usually c1= CG =1.5. In this way, individual experience and group experience have the same important influence, making the final optimal solution more accurate.The maximum velocity of the particle is vmaxThe particle speed in each dimension of space has a maximum speed limit value vd Max, used to control the particle speed, so that the speed is controlled within the range [-vimax, + VA Max], which determines the strength of the problem space search, the value is generally set by the user. Vmax is a very important parameter. If it is too large, particles may pass through the good region; if it is too small, particles may not be able to detect the region beyond the local optimal region adequately. They may get stuck in local optimality and not be able to move far enough to jump out of local optimality to a better position in space. The researchers point out that setting Vmax and adjusting inertia weights are equivalent, so! Max is generally used to set the initialization of the population, that is, vmax is set as the variation range of each dimension variable, instead of careful selection and adjustment of the maximum speed.Stopping criteriaThe maximum number of iterations, computational accuracy, or the maximum number of stagnant steps for the optimal solution ▲ T (or acceptable satisfactory solution) is usually considered to be the stop criterion, the termination condition of the algorithm. According to the specific optimization problem, the setting of the stop criterion should take into account the performance of the algorithm, such as the solution time, optimization quality and search efficiency.Setting up the neighborhood structureThe global pSO takes the whole population as the neighborhood of particles and has the advantage of fast convergence, but sometimes the algorithm will fall into local optimization. The local version of particle swarm optimization algorithm takes the individuals with similar positions as the neighborhood of particles, and the convergence speed is slow and it is not easy to fall into the local optimal value. In practical applications, the global particle swarm optimization algorithm can be used to find the direction of the optimal solution, that is, to get a rough result, and then the local particle swarm optimization algorithm can be used to conduct fine search near the optimal point.Boundary condition processingWhen the position or velocity of one or several dimensions exceeds the set value, the boundary condition processing strategy can limit the position of particles in the feasible search space, so as to avoid the expansion and divergence of the population, and avoid the blind search of particles in a large range, thus improving the search efficiency. There are many ways to do this, such as setting the maximum position limit Xmax and the maximum speed limit Vmax. When the maximum position or speed is exceeded, a numerical value is randomly generated in the range to replace it, or it is set to the maximum value, that is, boundary absorption.
2. Introduction of BP neural network
BP (Back Propagation) neural network was proposed by Rumelhart and McCelland in 1986. See their paper Learning Representations by Back-propagating Errors published in Nature. BP neural network is one of the most widely used neural network models. BP network can learn and store a large number of input-output mode mappings without revealing the mathematical equations describing the mappings in advance. Its learning rule is to use the fastest descent method, through back propagation to constantly adjust the weight and threshold of the network, so as to minimize the sum of squares of error of the network.
Last time we said that the multilayer perceptron has a bottleneck in how to obtain the weights of hidden layers. Since we can’t get the weight of the hidden layer directly, can we adjust the weight of the hidden layer indirectly by getting the error of the output result and the expected output of the output layer first? BP algorithm is designed with this idea. Its basic idea is that the learning process is composed of two processes: signal forward propagation and error back propagation. In the forward propagation, the input sample is introduced from the input layer, and then transmitted to the output layer after being processed layer by layer by layer. If the actual output of the output layer is not consistent with the desired output (teacher signal), then the back propagation stage of the error is entered. In reverse transmission, the output is transmitted back to the input layer layer by layer through the hidden layer in some form, and the error is apportioned to all the units of each layer, so as to obtain the error signal of each unit, which is used as the basis for correcting the weight of each unit. The detailed flow of these two processes will be described later.
The signal flow diagram of BP algorithm is shown in the figure below 
3.1 Topology structure of BP networkAs we said last time, BP networks are actually multilayer perceptrons, so their topology is the same as that of multilayer perceptrons. Because the single hidden layer (three layer) perceptron has been able to solve simple nonlinear problems, it is the most common application. The topology of the three-layer perceptron is shown in the figure below. A simple three-layer BP:
Training a BP neural network is actually to adjust the weight and bias of the network. The training process of BP neural network is divided into two parts:
Forward transmission, layer by layer wave transfer output value; Reverse feedback, reverse layer by layer adjustment of weight and bias; Let’s look at forward transmission first. Before training the network, we need to randomly initialize weights and biases, take a random real number of [−1,1] [-1,1][−1,1] for each weight, take a random real number of [0,1][0,1] [0,1] for each bias, Then forward transmission begins.
The training of neural network is completed by multiple iterations, each iteration uses all the records of the training set, while each training network uses only one record, which is described abstractly as follows:
whileTermination conditions not met:for record:dataset:
trainModel(record)
Copy the code
5.1 Network structure There are N nn neurons in the input layer, P PP neurons in the hidden layer, and Q QQ neurons in the output layer.
5.2 Definition of Variables 
In the design of BP network, the number of layers of the network, the number of neurons in each layer, the activation function, the initial value and the learning rate should generally be considered. The following are some selection principles.
6.1 Layer theory of Network has proved that a network with deviation and at least one S-type hidden layer plus one linear output layer can approach any rational function. Increasing the number of layers can further reduce the error and improve the accuracy, but it also complicates the network. Also not to be used only with nonlinear activation function of single-layer network to solve the problem, because you can do it with a single network problems, using adaptive linear network will be solved, and the adaptive linear network speed, for the nonlinear function can only be used to solve the problem, single precision is high enough, also only add layer can achieve the desired results.
6.2 Number of Hidden Layer Neurons Network training accuracy can be improved by adopting a hidden layer and increasing the number of its neurons, which is much simpler than increasing the number of network layers in structural implementation. Generally speaking, we use precision and training network time to measure the design quality of a neural network: (1) When the number of neurons is too small, the network cannot learn well, and the number of training iterations is relatively large, and the training accuracy is not high. (2) When there are too many neurons, the network has more powerful functions and higher accuracy, and the number of training iterations is also larger, which may lead to over fitting. Thus, the principle of selecting the number of hidden layer neurons of neural network is as follows: on the premise of solving the problem, add one or two neurons to accelerate the error reduction speed.
6.3 Selection of Initial Weights Generally, the initial weights are random numbers ranging from −1 to 1. In addition, After analyzing how the two-layer network trains a function, Widelow et al. proposed a strategy of selecting an initial weight of S √r, where R is the number of inputs and S is the number of neurons at the first layer.
6.4 Learning Rate The learning rate ranges from 0.01 to 0.8. A large learning rate may cause system instability, but a small learning rate causes slow convergence and requires a long training time. For a more complex network, different learning rates may be required at different positions of the error surface. In order to reduce the training times and time for finding the learning rate, a more appropriate method is to adopt a variable adaptive learning rate, so that the network can set different learning rates at different stages.
6.5 Selection of Expected Error In the process of network design, an appropriate value of expected error should also be determined after comparative training, which is relative to the number of hidden layer nodes required. In general, two networks with different expected error values can be trained at the same time, and finally one of them can be determined by integrating factors.
7 Limitations of BP network BP network has the following problems: (1) Long training time is required: this is mainly caused by the small learning rate, which can be improved by changing or adaptive learning rate. (2) Completely unable to train: this is mainly reflected in the paralysis of the network. Usually, in order to avoid this situation, one is to choose a smaller initial weight and adopt a smaller learning rate. (3) Local minimum: The gradient descent method adopted here may converge to the local minimum, and better results may be obtained by using multi-layer networks or more neurons.
The main objective of P algorithm is to speed up the training speed and avoid falling into local minimum, etc. Common improvement methods include driving factor algorithm, adaptive learning rate, changing learning rate and action function shrinkage method. The basic idea of momentum factor method is to add a value proportional to the previous weight change to each weight change on the basis of back propagation, and generate a new weight change according to the back propagation method. The adaptive learning rate method is for some specific problems. The principle of the method to change the learning rate is that, in successive iterations, if the reciprocal sign of the objective function is the same for a certain weight, the learning rate of the weight increases; otherwise, if the sign is opposite, the learning rate of the weight decreases. And the shrink-back rule is to shift the function, which is to add a constant.
Three, some source code
%% Clear the environment variable CLC clear %% Number of nodes inputnum=8;
hiddennum=8;
outputnum=1; %% Training data forecasting data extraction and normalization % load input and output data load humidity tday =350; % Training data time sday =30; % Sdata = (tday+sday)*24; % Total number of collected data oddata =24; % Indicates the number of data collected in a day10Day data will be10Minutes of data extracted, total2880A temperature = Temp (2:1:sdata,:);
humidity = Humidity(2:1:sdata,:); % information=[temp humidity]; % Initializing data [m,n]=size(temperature); n =8;
% m = (m-sday*oddata/10)-oddata/10-n+1;
m = m-sday*oddata-oddata-n+1; % Input = zeros(m,n); output = zeros(m,1); Find training data and prediction datafor i = 1:m
input_train(i,1:n/2) = temperature(i:i+3, :); input_train(i,n/2+1:n) = humidity(i:i+3, :); output_train(i,:) = temperature(i+oddata,:); end [m,n] = size(temperature); n =8;
m = (m-tday*oddata)-oddata-n+1;
for i = 1:m
input_test(i,1:n/2) = temperature(i+tday*oddata:i+3+tday*oddata,:);
input_test(i,n/2+1:n) = humidity(i+tday*oddata:i+3+tday*oddata,:);
output_test(i,:) = temperature(i+oddata+tday*oddata,:);
end
input_train=input_train';
output_train=output_train';
input_test = input_test';
output_test=output_test'; % inputn,inputps =mapminmax(input_train,0.1);
[outputn,outputps]=mapminmax(output_train,0.1); Net = newFF (inputn,outputn,hiddennum); % parameter initializes % two parameters in the particle swarm optimization algorithm c1 =1.49445;
c2 = 1.49445;
maxgen=100; % number of evolutions sizepop=30; % population size % % position and speed interval Vmax=1;
Vmin=- 1;
popmax=5;
popmin=- 5; %% initializes population position, speed, fitnessfor i=1:sizepop
pop(i,:)=5*rands(1.81);
V(i,:)=rands(1.81); % rands to generate1*81Are distributed in (- 1~ +1Pseudo random number between fitness (I) = fun (pop (I, :), inputnum hiddennum, outputnum,.net, inputn, outputn); [bestfitness bestindex]=min(fitness); zbest=pop(bestindex,:); % gbest=pop; % Individual fitnessgBest =fitness; % Individual optimum fitness value fitnesszBest =bestfitness; % global optimum fitness value %% iterative optimizationfor i=1:maxgen
i;
for j=1Update: sizepop % velocity V (j) = V (j:) + c1 * rand * (gbest (j:) - pop rand (j, :)) + c2 * * (zbest - pop (j:)); V(j,find(V(j,:)>Vmax))=Vmax; V(j,find(V(j,:)<Vmin))=Vmin; % population update pop(j,:)=pop(j,:)+0.2*V(j,:); pop(j,find(pop(j,:)>popmax))=popmax; pop(j,find(pop(j,:)<popmin))=popmin; % % adaptive variation % pos=unidrnd(81); %unidrnd (N) generates a slave1Up to N discrete random number %if rand>0.95
% pop(j,pos)=5*rands(1.1); % % end fitness value of fitness (j) = fun (pop (j:), inputnum hiddennum, outputnum,.net, inputn, outputn); endfor j=1:sizepop % Individually optimal updateiffitness(j) < fitnessgbest(j) gbest(j,:) = pop(j,:); fitnessgbest(j) = fitness(j); End % group optimal updateif fitness(j) < fitnesszbest
zbest = pop(j,:);
fitnesszbest = fitness(j);
end
end
yy(i)=fitnesszbest; End % the function function to return forecast and the absolute value of error between actual value and the function of error = fun (x, inputnum, hiddennum outputnum,.net, inputn, outputn) % % this function used to calculate the fitness value of x Input Individual %inputnum INPUT Number of input layer nodes %outputnum input Number of hidden layer nodes % NET Input Network % INputn input training input data %outputn input training output data % Error Output Individual fitness value % extraction w1=x(1:inputnum*hiddennum);
B1=x(inputnum*hiddennum+1:inputnum*hiddennum+hiddennum);
w2=x(inputnum*hiddennum+hiddennum+1:inputnum*hiddennum+hiddennum+hiddennum*outputnum);
B2=x(inputnum*hiddennum+hiddennum+hiddennum*outputnum+1:inputnum*hiddennum+hiddennum+hiddennum*outputnum+outputnum); % Net. Trainparam. epochs=50;
net.trainParam.lr=0.1;
net.trainParam.goal=0.00001;
net.trainParam.show=100;
net.trainParam.showWindow=0;
subplot(2.3.5)
plot(tr.epoch,tr.perf'.'r',tr.epoch,tr.vperf'.':g',tr.epoch,tr.tperf'.'-.b');
% legend(['Terminating algebra =' num2str(a)])
legend('training'.'confirm'.'test'.- 1)
title( 'Error curve')
ylabel('MSE error squared '.'fontsize'.12)
xlabel('Evolutionary algebra'.'fontsize'.12%% determining coefficient R^2
fenmu = 0; fenzi =0;
for i= 1:m
fenmu = fenmu + (output_test(:,i)-mean(output_test,2)). ^2;
fenzi = fenzi + (output_test(:,i)-BPoutput(:,i)).^2;
end
R = 1- fenzi/fenmu; %% Root mean square error RMSE MSE =0;
for i = 1:m
MSE = MSE + ((BPoutput(:,i)-output_test(:,i)).^2);
end
RMSE = sqrt(MSE/m); %% Plot each evaluation function value subplot(2.3.6)
plot(BPoutput,output_test,'. ');
% legend(['RMSE = ',num2str(RMSE)]);
title(['R^2 = ',num2str(R),'; RMSE = ',num2str(RMSE)])
ylabel('Sequence true value'.'fontsize'.12)
xlabel('Predicted value'.'fontsize'.12)
toc
Copy the code4. Operation results
Matlab version and references
1 matlab version 2014A
[1] Yang Baoyang, YU Jizhou, Yang Shan. Intelligent Optimization Algorithm and Its MATLAB Example (2nd Edition) [M]. Publishing House of Electronics Industry, 2016. [2] ZHANG Yan, WU Shuigen. MATLAB Optimization Algorithm source code [M]. Tsinghua University Press, 2017. [3] Zhou Pin. Design and Application of MATLAB Neural Network [M]. Tsinghua University Press, 2013. [4] Chen Ming. MATLAB Neural Network Principle and Examples of Fine Solution [M]. Tsinghua University Press, 2013. [5] FANG Qingcheng. MATLAB R2016a Neural Network Design and Application of 28 Cases Analysis [M]. Tsinghua University Press, 2018.