TensorFlow HOWTO 2.1 Support Vector Classification (Soft Interval)

In the traditional machine learning method, support vector machine is a more powerful method, but the calculation process is very complicated. Soft interval support vector machines (SVMS) reduce their constraints and simplify the computation.

steps

Import the required packages.

import tensorflow as tf
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import sklearn.datasets as ds
import sklearn.model_selection as msCopy the code

Import data and preprocess it. We used all the samples in the iris data set to predict whether the samples were iris mountain (the first kind) based on sepal length and petal length. Note that support vector machines only accept labels of 1 and -1.

iris = ds.load_iris() x_ = iris.data[:, [0, 2]] y_ = (iris.target == 0).astype(int) y_[y_ == 0] = -1 y_ = np.expand_dims(y_ , 1) x_train, x_test, y_train, Y_test = \ Ms. Train_test_split (x_, y_, train_size=0.7, test_size=0.3)Copy the code

Define hyperparameters.

N_input = 2 N_epoch = 2000 LR = 0.05 LAM = 0.05Copy the code

Build a model.

x = tf.placeholder(tf.float64, [None, n_input])
y = tf.placeholder(tf.float64, [None, 1])
w = tf.Variable(np.random.rand(n_input, 1))
b = tf.Variable(np.random.rand(1, 1))
z = x @ w + bCopy the code

Define loss, optimization, and accuracy metrics. There are many indicators for classification problems, but I’ll show you just one.

We use a combination of Hinge losses and L2 losses. Hinge loss is:

$mean(\max(1 – Z \otimes Y, 0))$

In the original model, the constraint was that the sample had to fall outside the support boundary, i.e. $yz >= 1$. We add this constraint to the loss, and we get Hinge loss. It means that its loss is zero for points that satisfy the constraint and $1 – yz$for points that do not satisfy the constraint. This keeps the sample as far out of the support boundary as possible.

L2 loss to minimize support boundary geometrical distance, which is $\ frac {2} {| | w} $.

hinge_loss = tf.reduce_mean(tf.maximum(1 - y * z, 0))
l2_loss = lam * tf.reduce_sum(w ** 2)
loss = hinge_loss + l2_loss
op = tf.train.AdamOptimizer(lr).minimize(loss)

y_hat = tf.to_double(z > 0) - tf.to_double(z <= 0)
acc = tf.reduce_mean(tf.to_double(tf.equal(y_hat, y)))Copy the code

Train the model with the training set.

losses = []
accs = []

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver = tf.train.Saver(max_to_keep=1)
    
    for e in range(n_epoch):
        _, loss_ = sess.run([op, loss], feed_dict={x: x_train, y: y_train})
        losses.append(loss_)Copy the code

Use test sets to calculate accuracy.

        acc_ = sess.run(acc, feed_dict={x: x_test, y: y_test})
        accs.append(acc_)Copy the code

Print the loss and measure value every 100 steps.

        if e % 100 == 0:
            print(f'epoch: {e}, loss: {loss_}, acc: {acc_}')
            saver.save(sess,'logit/logit', global_step=e)Copy the code

Get the decision boundary:

x_plt = x_[:, 0] y_plt = x_[:, 1] c_plt = y_.ravel() x_min = x_plt.min() - 1 x_max = x_plt.max() + 1 y_min = y_plt.min() - 1 y_max = y_plt.max() + 1 X_rng = np. Arange (x_min, x_max, 0.05) y_rng = np. Arange (x_min, y_max, 0.05) x_rng = np. y_rng) model_input = np.asarray([x_rng.ravel(), y_rng.ravel()]).T model_output = sess.run(y_hat, feed_dict={x: model_input}).astype(int) c_rng = model_output.reshape(x_rng.shape)Copy the code

Output:

Epoch: 0, Loss: 4.511212919815273, ACC: 0.22222222222222 Epoch: 100, Loss: 0.0814942611949705, ACC: 1.0 epoch: 200, Loss: 0.07629443566925993, ACC: 1.0 epoch: 300, Loss: 0.07146107394130172, ACC: 1.0 epoch: 400, Loss: 0.06791927215796319, ACC: 1.0 epoch: 500, Loss: 0.06529065400047798, ACC: 1.0 epoch: 600, Loss: 0.06335060635876646, ACC: 1.0 epoch: 700, Loss: 0.061836271593737835, ACC: 1.0 epoch: 800, Loss: 0.06079800773555345, ACC: 1.0 epoch: 900, Loss: 0.06042716484730995, ACC: 1.0 epoch: 1000, Loss: 0.06091475237291386, ACC: 1.0 epoch: 1100, Loss: 0.06021069445352348, ACC: 1.0 epoch: 1200, Loss: 0.06019457351257251, ACC: 1.0 epoch: 1300, Loss: 0.06000348375369489, ACC: 1.0 epoch: 1400, Loss: 0.060206981088196394, ACC: 1.0 epoch: 1500, Loss: 0.060210741691625935, ACC: 1.0 epoch: 1600, Loss: 0.060570783158962985, ACC: 1.0 epoch: 1700, Loss: 0.06003457018203537, ACC: 1.0 epoch: 1800, Loss: 0.060203912161627175, ACC: 1.0 epoch: 1900, Loss: 0.06019910894894441, ACC: 1.0Copy the code

Draw the entire data set and decision boundaries.

plt.figure() cmap = mpl.colors.ListedColormap(['r', 'b']) plt.scatter(x_plt, y_plt, c=c_plt, Contourf (x_rng, y_rng, c_rng, alpha=0.2, lineWidth =5, cmap=cmap) plt.title('Data and Model') plt.xlabel('Petal Length (cm)') plt.ylabel('Sepal Length (cm)') plt.show()Copy the code

plt.figure()
plt.plot(losses)
plt.title('Loss on Training Set')
plt.xlabel('#epoch')
plt.ylabel('Cross Entropy')
plt.show()Copy the code

plt.figure()
plt.plot(accs)
plt.title('Accurary on Testing Set')
plt.xlabel('#epoch')
plt.ylabel('Accurary')
plt.show()Copy the code

Further reading

• Stanford CS229 Notes: 12. Support vector machines
• Wikipedia: Support vector machine