This is the 10th day of my participation in the August More Text Challenge

In this paper, we introduce Precision, Recall, Sensitivity, Specificity, Accuracy, FNR, FPR, TNR, TPR, F1 Score. The basic meaning of Balanced F Score is presented, and the formulas and examples are given.

Basic definition

Evaluation indicators Predicted results
Is the sample Negative samples
The actual

situation		 Is the sample TP FN
Negative samples FP TN

For details, see Machine Learning – Basics – TP, FN, FP, TN.

Example use cases

Sample information

Sample number 1 2 3 4 5 6 7 8 9 10
Real category P P P P P P P N N N
Sample number 1 2 3 4 5 6 7 8 9 10
Predicted class P P P N N N N N N N
Evaluation indicators TP 3 TN 3 FP 0 FN 4

Predict – 2

Sample number 1 2 3 4 5 6 7 8 9 10
Predicted class P P N N P P P P N N
Evaluation indicators TP 5 TN 2 FP 1 FN 2

Predict – 3

Sample number 1 2 3 4 5 6 7 8 9 10
Predicted class P P P P P P P P P P
Evaluation indicators TP 7 TN 0 FP 3 FN 0

Precision

Accuracy, accuracy, accuracy

Meaning: Predict the correct proportion of all positive samples:


P r e c i s i o n = T P T P + F P Precision=\frac{TP}{TP+FP}

Prediction use case Precision:


P r e c i s i o n To predict 1 = T P T P + F P = 3 3 + 0 = 1 Precision_ 1} {prediction = \ frac {TP} {TP + FP} = \ frac {3} {3 + 0} = 1

P r e c i s i o n To predict 2 = T P T P + F P = 5 5 + 1 material 0.83 Precision_ 2} {prediction = \ frac {TP} {TP + FP} = \ frac {5} {5 + 1} \ approx 0.83

P r e c i s i o n To predict 3 = T P T P + F P = 7 7 + 3 = 0.7 Precision_ 3} {prediction = \ frac {TP} {TP + FP} = \ frac {7} {7 + 3} = 0.7

It can be seen that all positive samples judged by Prediction 1 are correct, so this prediction has the highest accuracy rate.

Precision can be understood as the confidence probability of positive samples judged by the model. The higher the probability, the more reliable the positive samples judged by the model.

FDR (False Discorvery Rate)

Overkill rate (industrial defect). Meaning: Reflects the proportion of negative samples in the positive samples judged by the detector:


F D R = F P T P + F P = 1 – P r e c i s i o n FDR=\frac{FP}{TP+FP}=1-Precision

Predict the use case FA:


F D R To predict 1 = F P T P + F P = 0 3 + 0 = 0 FDR_ 1} {prediction = \ frac {FP} {TP + FP} = \ frac {0} {3 + 0} = 0

F D R To predict 2 = F P T P + F P = 1 5 + 1 material 0.17 FDR_ 2} {prediction = \ frac {FP} {TP + FP} = \ frac {1} {5 + 1} \ approx0.17

F D R To predict 3 = F P T P + F P = 3 7 + 3 = 0.3 FDR_ 3} {prediction = \ frac {FP} {TP + FP} = \ frac {3} {7 + 3} = 0.3

Recall/Sensitivity/TPR (True Positive Rate)

Recall rate, recall rate, sensitivity, true rate

Meaning: Predicted correct proportion of all positive samples to actual positive samples:


R e c a l l = S e n s i t i v i t y = T P R = T P T P + F N Recall=Sensitivity=TPR=\frac{TP}{TP+FN}

Recall/Sensitivity/TPR:


R e c a l l To predict 1 = T P T P + F N = 3 3 + 4 material 0.43 Recall_ 1} {prediction = \ frac {TP} {FN} TP + = \ frac {3} {3 + 4} \ approx0.43

R e c a l l To predict 2 = T P T P + F N = 5 5 + 2 material 0.71 Recall_ 2} {prediction = \ frac {TP} {FN} TP + = \ frac {5} {5 + 2} \ approx0.71

R e c a l l To predict 3 = T P T P + F N = 7 7 + 0 = 1 Recall_ 3} {prediction = \ frac {TP} {FN} TP + = \ frac {7} {7 + 0} = 1

Recall rate and precision rate are considered from different angles. We do not pay attention to whether the model is accurate enough to judge positive samples, but pay attention to the proportion of positive samples selected by the model to all positive samples.

Therefore, the simplest strategy to determine that all samples are positive can get a 100% recall rate, because the positive samples detected by this model are “complete”.

Specificity/TNR (True Negative Rate)

Specificity, true negative ratio. Meaning: The proportion of all negative samples correctly predicted to the actual negative samples:


S p e c i f i c i t y = T N R = T N T N + F P Specificity=TNR=\frac{TN}{TN+FP}

Predict use case Specificity/TNR:


S p e c i f i c i t y To predict 1 = T N T N + F P = 3 3 + 0 = 1 Specificity_ 1} {prediction = \ frac {TN} {TN + FP} = \ frac {3} {3 + 0} = 1

S p e c i f i c i t y To predict 2 = T N T N + F P = 2 2 + 1 material 0.67 Specificity_ 2} {prediction = \ frac {TN} {TN + FP} = \ frac {2} {2 + 1} \ approx0.67

S p e c i f i c i t y To predict 3 = T N T N + F P = 0 0 + 3 = 0 Specificity_ 3} {prediction = \ frac {TN} {TN + FP} = \ frac {0} {0 + 3} = 0

Similar to recall, it describes the situation on the other side.

FPR (False Positive Rate)

False positive rate, false detection rate, false alarm rate. Meaning: Predict the proportion of the number of negative samples misjudged as positive samples to the actual total negative samples:


F P R = F P F P + T N FPR=\frac{FP}{FP+TN}

Prediction use case FPR:


F P R To predict 1 = F P F P + T N = 0 0 + 3 = 0 FPR_ 1} {prediction = \ frac {FP} {FP + TN} = \ frac {0} {0 + 3} = 0

F P R To predict 2 = F P F P + T N = 1 1 + 2 material 0.33 FPR_ 2} {prediction = \ frac {FP} {FP + TN} = \ frac {1} {1 + 2} \ approx0.33

F P R To predict 3 = F P F P + T N = 3 3 + 0 = 1 FPR_ 3} {prediction = \ frac {FP} {FP + TN} = \ frac {3} {3 + 0} = 1

FNR (False Negative Rate)

False negative rate, false alarm rate, false detection rate. Meaning: Predict the proportion of the number of positive samples misjudged as negative to the actual total positive samples:


F N R = F N F N + T P = 1 – R e c a l l FNR=\frac{FN}{FN+TP}=1-Recall

Prediction use case FNR:


F N R To predict 1 = F N F N + T P = 4 4 + 3 material 0.57 FNR_ 1} {prediction = \ frac {FN} {FN + TP} = \ frac {4} {4 + 3} \ approx0.57

F N R To predict 2 = F N F N + T P = 2 2 + 5 material 0.29 FNR_ 2} {prediction = \ frac {FN} {FN + TP} = \ frac {2} {2 + 5} \ approx0.29

F N R To predict 3 = F N F N + T P = 0 0 + 7 = 0 FNR_ 3} {prediction = \ frac {FN} {FN + TP} = \ frac {0} {0 + 7} = 0

Accuracy

The proportion of the number of samples predicted to be correct in all experiments.


A c c u r a c y = T P + T N T P + F P + T N + F N Accuracy=\frac{TP+TN}{TP+FP+TN+FN}

Accuracy of predicted use cases:


A c c u r a c y To predict 1 = T P + T N T P + F P + T N + F N = 3 + 3 3 + 0 + 3 + 4 = 0.6 Accuracy_ 1} {prediction = \ frac {TP + TN} {TP + FP + TN + FN} = \ frac {3 + 3} {3 + 0 + 3 + 4} = 0.6

A c c u r a c y To predict 2 = T P + T N T P + F P + T N + F N = 5 + 2 5 + 1 + 2 + 2 = 0.7 Accuracy_ 2} {prediction = \ frac {TP + TN} {TP + FP + TN + FN} = \ frac {5 + 2} {5 + 1 + 2 + 2} = 0.7

A c c u r a c y To predict 3 = T P + T N T P + F P + T N + F N = 7 + 0 7 + 3 + 0 + 0 = 0.7 Accuracy_ 3} {prediction = \ frac {TP + TN} {TP + FP + TN + FN} = \ frac {7 + 0} {7 + 3 + 0 + 0} = 0.7

In fact, the prediction model 1,2 has some correct classification ability for positive and negative samples, while the prediction model 3 only adopts the strategy of “judge all samples as positive”, which achieves the highest accuracy. It is not the problem of this index, but the unbalanced data distribution itself.

The positive samples were in the majority, and the model of Prediction 3 successfully predicted that the positive samples accounted for the majority of the data, so the strategy achieved a high correct rate of return.

Error Rate

Error rate Meaning: The percentage of the total number of samples in all trials in which the prediction was wrong.


E r r o r _ R a t e = F P + F N T P + F P + T N + F N = 1 – A c c u r a c y Error\_Rate=\frac{FP+FN}{TP+FP+TN+FN}=1-Accuracy

Accuracy of predicted use cases:


E r r o r _ R a t e To predict 1 = F P + F N T P + F P + T N + F N = 0 + 4 3 + 0 + 3 + 4 = 0.4 The Error \ _Rate_ 1} {prediction = \ frac {FP + FN} {TP + FP + TN + FN} = \ frac {4} 0 + {3 + 0 + 3 + 4} = 0.4

E r r o r _ R a t e To predict 2 = F P + F N T P + F P + T N + F N = 1 + 2 5 + 1 + 2 + 2 = 0.3 The Error \ _Rate_ 2} {prediction = \ frac {FP + FN} {TP + FP + TN + FN} = \ frac {1 + 2} {5 + 1 + 2 + 2} = 0.3

E r r o r _ R a t e To predict 3 = F P + F N T P + F P + T N + F N = 3 + 0 7 + 3 + 0 + 0 = 0.3 The Error \ _Rate_ 3} {prediction = \ frac {FP + FN} {TP + FP + TN + FN} = \ frac {3 + 0} {7 + 3 + 0 + 0} = 0.3

F1 Score / Balanced F Score

F1 score is defined as the harmonic mean of model accuracy and recall, which takes into account both accuracy and recall of classification models.


F 1 S c o r e = 2 x P r e c i s i o n x R e c a l l P r e c i s i o n + R e c a l l F_1 Score=2\times\frac{Precision\times Recall}{Precision+ Recall}

Prediction use case F1 Score:


F 1 S c o r e To predict 1 = 2 x P r e c i s i o n To predict 1 x R e c a l l To predict 1 P r e c i s i o n To predict 1 + R e c a l l To predict 1 = 2 x 1 x 3 7 1 + 3 7 = 0.6 F_1 Score_{forecast 1}=2\times\frac{Precision_{forecast 1}\times Recall_{forecast 1}}{Precision_{forecast 1}+ Recall_{forecast 1}}=2\times\frac{1\times \ frac {3} {7}} {1 + \ frac {3} {7}} = 0.6

F 1 S c o r e To predict 2 = 2 x P r e c i s i o n To predict 2 x R e c a l l To predict 2 P r e c i s i o n To predict 2 + R e c a l l To predict 2 = 2 x 5 6 x 5 7 5 6 + 5 7 material 0.77 F_1 Score_{prediction 2}=2\times\frac{Precision_{prediction 2}\times Recall_{prediction 2}}{Precision_{prediction 2}+ Recall_{prediction 2}}=2\times\frac{\frac{5}{6} \ times \ frac {5} {7}} {\ frac {5} {6} + \ frac {5} {7}} \ approx 0.77

F 1 S c o r e To predict 3 = 2 x P r e c i s i o n To predict 3 x R e c a l l To predict 3 P r e c i s i o n To predict 3 + R e c a l l To predict 3 = 2 x 0.7 x 1 0.7 + 1 material 0.82 F_1 Score_{forecast 3}=2\times\frac{Precision_{forecast 3}\times Recall_{forecast 3}}{Precision_{forecast 3}+ Recall_{forecast 3}}=2\times\frac{0.7 \times 0.7 + 1} {1} \ approx 0.82