Evaluation indexes of classification model in SKLearn (IV) : Jaccard similarity coefficient, hinge loss and logarithmic loss

This is the 27th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

Jaccard similarity coefficient

The jaccard_score function calculates the average value of the Jaccard similarity coefficient between pairs of tags, also known as the Jaccard index.

The Jaccard similarity coefficient of the ith sample has the real label set yiy_iyi and the predicted label set Y ^ I \hat{y}_iy^ I, and its formula is defined as:

Jaccard_score works in a similar way to Precision_recall_FSCOre_support, which is an aggregated measure for dichotomous problems and can be extended for multi-label and multi-category scenarios by using the Average parameter.

Example code for a binary scenario is as follows:

import numpy as np
from sklearn.metrics import jaccard_score
y_true = np.array([[0.1.1],
                   [1.1.0]])
y_pred = np.array([[1.1.1],
                   [1.0.0]])
print(jaccard_score(y_true[0], y_pred[0]))
print(jaccard_score(y_true[1], y_pred[1]))
Copy the code

Running results:

0.6666666666666666
0.5
Copy the code

In a multi-label scenario with a two-label indicator:

# (0.5 + 0.66666) / 2 = 0.58333
print(jaccard_score(y_true, y_pred, average='samples'))

# (1/2 + 1/2 + 2/2) / 3
print(jaccard_score(y_true, y_pred, average='macro'))

print(jaccard_score(y_true, y_pred, average=None))
Copy the code

Running results:

0.5833333333333333 0.666666666666 [0.5 0.5 1.]Copy the code

The multi-classification problem is binarized and treated like the multi-label problem:

y_pred = [0.2.1.2]
y_true = [0.1.2.2]

print(jaccard_score(y_true, y_pred, average=None)) A third # 1/1.two survivors
print(jaccard_score(y_true, y_pred, average='macro')) # (1 + 0.333) / 3 = 0.444
print(jaccard_score(y_true, y_pred, average='micro')) # (1 + 1)/(1 + 2 + 3)
Copy the code

Running results:

[1.         0.         0.33333333]
0.4444444444444444
0.3333333333333333
Copy the code

Hinge loss

The Hinge_Loss function calculates the average distance between the model and the data using hinge loss, a unilateral measure that considers only the prediction error. (Hinge losses are used for maximum edge classifiers, such as support vector machines.)

If the tag is encoded with +1 and -1, y is the true value, and w is the predictive decision output by decision_function, then the hinge loss is defined as:

Hinge_loss uses multiple classification variants if there are more than two tags.

If yWY_WYw is the prediction decision of real labels and yTY_TYt is the maximum prediction decision of all other labels, where the prediction decision is output by the decision function, then the multi-classification hinge loss is defined as:

Here is a small example demonstrating the use of hinge loss functions under a SVM classifier in a binary classification problem:

from sklearn import svm
from sklearn.metrics import hinge_loss
X = [[0], 
     [1]]
y = [-1.1]

est = svm.LinearSVC(random_state=0)
est.fit(X, y)

pred_decision = est.decision_function([[-2], 
                                       [3], 
                                       [0.5]])
print(pred_decision)
print(hinge_loss([-1.1.1], pred_decision))
Copy the code

Running results:

0.3030367603854425 [2.18177944 2.36355888 0.09088972]Copy the code

Here is an example demonstrating the use of hinge loss functions in a multi-classification problem with a SVM classifier:

X = np.array([[0], 
              [1], 
              [2], 
              [3]])
Y = np.array([0.1.2.3])
labels = np.array([0.1.2.3])
est = svm.LinearSVC()
est.fit(X, Y)

pred_decision = est.decision_function([[-1], 
                                       [2], 
                                       [3]])
print(pred_decision)
y_true = [0.2.3]
print(hinge_loss(y_true, pred_decision, labels=labels))
Copy the code

Running results:

[[1.27270694 0.03417093-0.68376942-1.4016749] [-1.45454139-0.58116657-0.37610526-0.17099403] [-2.36362417 0.5641092543547738-0.78627908-0.27355054, 0.23923292]]Copy the code

Log Loss

Logarithmic loss, also known as logistic regression loss or cross entropy loss, is defined on probability estimates. It is commonly used for (polynomial) logistic regression and neural networks, as well as some variants of expectation-maximization (EM algorithm), and can be used to evaluate the probabilistic output of a classifier (predict_proba) rather than its discrete predictions.

With real labels y ∈ {0, 1} y \ \ {0, 1 \} in y ∈ {0, 1} and probability estimates p = Pr ⁡ (y = 1) p = \ operatorname {Pr} (y = 1) = p (y = 1) Pr of binary classification, The logarithmic loss per sample is the negative logarithmic likelihood of the classifier given the real label:

The extension from dichotomies to multi-classifications is as follows, let the real labels of a set of samples be encoded as a binary indicator matrix Y from 1 to K, that is, if the label K of sample I comes from a set of K labels, then Yi, K =1y_{I, K} = 1yi, K =1. P for probability estimation matrix, among them, PI, k = Pr ⁡ p_ (yi, k = 1) {} I, k = \ operatorname {Pr} (y_ {} I, k = 1) PI, k = Pr (yi, k = 1). Then the logarithmic loss of the whole set is:

To see how this generalizes the binary logarithmic loss given above, note that in the binary case, PI, 0 = 1 – PI, 1 p_ {I, 0} = 1 – p_ {I, 1} PI, 0 = 1 – PI, 1 and yi, 0 = 1 – yi, 1 y_ {I, 0} = 1 – y_ {I, 1} yi, 0 = 1 – yi, 1, therefore, In yi, k ∈ {0, 1} y_ {} I, k \ \ {0, 1 \} in yi, k ∈ {0, 1} expanding internal combined dual logarithm losses is presented.

The log_loss function calculates the logarithmic loss based on the real tag list and probability matrix returned by the estimator’s predict_proba method.

Sample code:

from sklearn.metrics import log_loss
y_true = [0.0.1.1]
y_pred = [[9..1.], [8..2.], [3..7.], [.01.99.]]
print(log_loss(y_true, y_pred))
Copy the code

Running results:

0.1738073366910675
Copy the code

Where, the first [.9,.1] in y_pred means that the probability of the first sample label being 0 is 90%. The logarithmic loss is nonnegative.

conclusion

Reference documentation

• Jaccard similarity coefficient score
• Hinge loss
• Log loss