Based on an overview

1. Cross validation: Usually used when the data is not very sufficient, it is divided into simple cross validation, K-fold cross validation and retention cross validation (sample size ≤50). K-fold cross verification is to divide the training set into K mutually exclusive subsets of similar size. Each time, the union of K-1 subsets is used as the training set, and the remaining subsets are used as the verification set for k times of training and verification. Finally, the mean value of K test results is returned, and the model and parameters with the minimum loss function are selected at last.The purpose of cross – validation is to select suitable models and parameters. When the data set is small, the cross-validation method is used for model screening when there are multiple candidate models. Cross validation is used to evaluate a model when there is only one.2. Model error and fitting effect:

  • Model error = Bias + Variance + data error

Error is a measure of model performance. Among the model errors, the deviation is caused by the model’s inability to represent the complexity of basic data, and the variance is caused by the model’s excessive sensitivity to the data used in training.

Deviation: It measures the deviation between the predicted value and the real value of the learning algorithm, and describes the fitting ability of the algorithm itself. When the model is not complex enough to capture the basic relationship, there will be a large deviation, and the accuracy will be reduced, that is, underfitting.

The variance: Measures the changes in learning performance caused by changes in training sets of the same size. It describes the influence of data disturbance, namely the stability of the model. Excessively high variance indicates that the model cannot generalize the predicted results to more data, but is highly sensitive to training sets, namely over-fitting. regular: The lower the complexity of the model, the larger the deviation, the smaller the variance (underfitting); The higher the complexity of the model, the smaller the deviation and the larger the variance (overfitting).

  • Overfitting and underfitting:

Solution:

situation performance methods
Owe fitting Models are bad at training and predicting evaluation indicators 1. Look for better features; 2. More features (added complexity)
A fitting The model performed well on evaluation metrics on the training set, but poorly on test sets and new data 1. Increase the data set (noise point ratio is reduced); 2. Reduce data features (reduce complexity); 3. Regularization method; 4. Cross validation method; 5. Reduce model complexity (such as decision tree pruning); 6. Integrated learning methods.
3. Evaluation criteria
  • Classification problems:
– Predicting 1 Predict 0 A combined
The actual 1 TP FN TP+FN
Actual 0 FP TN FP+TN
A combined TP+FP TN+FN TP+FP+TN+FN
The evaluation index The formula understand
accuracy(Accuracy) (TP+TN)/(TP+FP+TN+FN) Predict the correct sample proportion, — >1 is better
Accurate rate(Precision) TP/(TP+FP) The percentage of the sample that was predicted to be 1 that was actually 1
The recall rateRecall (sensitive rate, true rate) TP/(TP+FN) The proportion of the sample that is predicted to be 1 in the sample that is actually 1, such as earthquake prediction, will cause great harm if it happens once, and the indicator — >1 is better
Specific degrees(Specificity) SP=TN/(FP+TN) In contrast to the recall rate, the percentage of the sample that was actually zero that was predicted to be zero
F ratio F=2/(1/ accuracy +1/ recall rate) Balance accuracy and recall
  • Dichotomous problems:

  • Regression problems: mean square error (MSE), root mean square error (RMSE), evaluation absolute error MAE