Boosting model

Boosting Model – a model that integrates multiple decision tree models

Boosting model random forest Model and Boosting Model-model integration Bagging and Boosting method random forest model The generation of each decision tree model is independent of each other -- sample resampling can get different training sets to generate different decision tree models, Boosting model: New decision tree model generation - based on the results of the generated decision tree model - the generation of decision tree model is not independent of each other - the new decision tree generation depends on the previous decision tree model 1. What are the common Boosting methods based on decision tree model and their respective principles? AdaBoost and GBDT AdaBoost: Increase the weight of the misclassified data in the previous decision model - the next generated decision tree model will try to classify these training sets correctly GBDT: Calculate the decline direction of loss function gradient - insufficient positioning model - establish a new decision tree model - more widely applied 1. Abstract: The advantages and disadvantages of random forest model and GBDT model are summarized. Random forest model and GBDT model are widely used. Based on decision tree model, they can deal with scenarios where discrete variables and continuous variables coexist - Performance greatly improved for larger training sets - slower training - finding faster methodsCopy the code

XGBoost model

Boosting model with better performance

Based on the traditional GBDT model, XGBoost has made the following adjustments to the ** algorithm ** : 1. Traditional GBDT-CART tree ** base learner ** XGBoost - also ** support linear classifiers ** - base learner ** L1 and L2 regularized logistic regression model/linear regression model - improves the model's application range 1. Traditional GBDT - ** optimization ** - loss function first derivative information XGBoost - optimization - loss function second order Taylor expansion - get the first derivative and second derivative ** speed up the optimization ** 1. XGBoost - ** Loss function ** adds the regular term ** - Control the complexity of the model ** - Tradeoff the Angle of variance and bias - reduces the variance of the model - the learned model is simpler - prevents overfitting - improves the model ** generalization ability ** 1. Reference random forest model - sampling ** feature ** - Supporting column sampling in the process of generating decision tree - preventing overfitting + reducing computation 1. Automatic processing ** missing value ** - separate it as a branch algorithm improvement + performance greatly improved [on large data processing - further improve computing efficiency, improve speed (support parallelism)] 1. XGBoost's parallel operation model: After one iteration - proceed to the next iteration One of the most time-consuming steps in parallel decision tree model learning - sorting ** eigenvalues ** [Determining the best split point] XGBoost - before training - sorting the data in advance - saving the structure as a block - reuse this structure in subsequent iterations - greatly reducing the amount of computation block structure - parallelization is possible Node selection - calculates the gain of each feature - selects the feature with the largest gain as the node - calculates the gain of each feature - implements parallel operation based on block structureCopy the code