The principle of

Let’s say that the input space (the eigenspace) isAnd the output space is

model


It’s called a perceptron.

  • 和 Is the perceptron model parameter
  • It’s called a weight or a weight vector.
  • It’s called bias
  • said 和 The inner product
  • It’s a sign function

strategy

The training data set is assumed to be linearly separableThe goal of perceptron learning is to obtain a separate hyperplane which can completely separate positive and negative instance points of the training set. In order to find such a hyperplane, the parameters of perceptron model are determined 和 , need to determine a learning strategy, that is, define (experience) loss function and minimize the loss function.

A natural selection of the loss function is the total number of misclassification points. However, such loss function is not the continuous differentiable function of parameter W and b, so it is not easy to optimize. Another alternative to the loss function is the total distance from the misclassification point to the hyperplane S, which is used by the perceptron.

The total distance between all misclassification points and the hyperplane S is


Don’t consider, the loss function of perceptron learning is obtained.

algorithm

Original form

Input: Training data set, including; vector;

Output:; Perceptron model.

  1. Selection of initial value
  2. Select data in the training set
  3. if

  1. Go to 2 until there are no misclassification points in the training set.
    # original
    def _fit(self):
        n_samples, n_features = self.X.shape
        # select the initial value w0,b0
        self.w = np.zeros(n_features, dtype=np.float64)
        self.b = 0.0

        is_finished = False
        # Keep looping
        while not is_finished:
            count = 0  # Record the number of misclassification points
            for i in range(n_samples):
                # if yi (w. xi + b) 0 or less
                if self.y[i] * self.sign(self.w, self.X[i], self.b) <= 0:
                    self.w += self.l_rate * np.dot(self.y[i], self.X[i])
                    self.b += self.l_rate * self.y[i]
                    self._wbs.append((i, self.w, self.b))
                    count += 1

                # Until there are no misclassification points in the training set
                if count == 0:
                    is_finished = True
Copy the code

This learning algorithm is intuitively explained as follows:

When an instance point is misclassified, that is, on the wrong side of the separation hyperplane, the values of W and B are adjusted to make the separation hyperplane move to the side of the misclassified point, so as to reduce the distance between the misclassified point and the hyperplane, until the hyperplane passes over the misclassified point and it is correctly classified.

The algorithm is the basic algorithm of perceptron learning, corresponding to the following dual form, called the original form.

Perceptron learning algorithm is simple and easy to implement.

Dual form of perceptron learning algorithm

  2. Select data in the training set
  3. if

  1. Go to 2 until there is no misclassified data.
    # Dual form
    def _fit_dual(self):
        In the dual form, the amount of computation can be greatly reduced by solving Gram matrix only when the training instance appears in the form of inner product.
        n_samples, n_features = self.X.shape
        self._alpha = np.zeros(n_samples, dtype=np.float64)
        self.w = np.zeros(n_features, dtype=np.float64)
        self.b = 0.0

        self._cal_gram_matrix()

        i = 0
        while i < n_samples:
            if self._dual_judge(i) <= 0:
                self._alpha[i] += self.l_rate
                self.b += self.l_rate * self.y[i]
                i = 0
            else:
                i += 1

        for i in range(n_samples):
            self.w += self._alpha[i] * self.X[i] * self.y[i]
Copy the code

In the dual form, the training examples only appear in the form of inner product. For convenience, the inner product between instances in the training set can be calculated in advance and stored in the form of a matrix, which is the so-called Gram matrix.

test

The PyTest.Mark. Parametrize decorator in the PyTest fixture enables use case parameterization. Here you pass in five sets of parameters directly.

import pytest
from slmethod.perceptron import Perceptron
import numpy as np


# 
@pytest.mark.parametrize("dual, l_rate", [(True, None), (False, None),
                                          (False.1), (False.0.1),
                                          (False.0.01)])
def test_perceptron(dual, l_rate):train_X = np.array([ ... ] ) train_y = np.array([ ... ] ) clf = Perceptron(dual=dual, l_rate=l_rate) clf.fit(train_X, train_y) test_X = np.array([[10.3], [- 29.5]])
    test_y = np.array([1.- 1])
    predict_y = clf.predict(test_X)
    assert np.array_equal(test_y, predict_y)
Copy the code

See GitHub: github.com/iOSDevLog/s…

animation

Matplotlib.org/api/animati…

Show only 2D data

    def show2d(self, name=None):
        if (self.X.shape[1] != 2) :raise ValueError("X must have 2d array.")
Copy the code

Take the minimum and maximum of X to draw a line

        minX = np.min(self.X[:, 0])
        maxX = np.max(self.X[:, 0])
        x_points = np.array([minX, maxX])
Copy the code

Import related libraries

from matplotlib import pyplot as plt
from matplotlib import animation
import numpy as np
Copy the code

Static figure

        fig, ax = plt.subplots()
        ax.scatter(self.X[:, 0], self.X[:, 1], c=self.y, s=1, marker="o")
        line, = ax.plot(x_points,
                        np.zeros(len(x_points)),
                        "r-",
                        linewidth=2,
                        label="slmethod perceptron")
Copy the code

Update the animation

Next, construct the custom animation function update, which is used to update the y coordinate of each x on each frame. The parameter represents frame I:

        def update(iter):
            (index, w, b) = self._wbs[iter]
            # title
            title = "iter: {}, index: {}".format(iter, index)
            plt.title(title)
            # show w and b
            wb = "w0: {}, w1: {}, b: {}".format(w[0], w[1], b)
            ax.set_xlabel(wb)
            # update y
            y_points = -(w[0] * x_points + b) / w[1]
            line.set_ydata(y_points)

            return line, ax
Copy the code

Initialize the

Then, construct the start frame function init:

        def init():
            line.set_ydata(np.zeros(len(x_points)))
            return line,
Copy the code

Generate animation

Next, we call the FuncAnimation function to generate the animation.

Parameter Description:

  • FIG is the figure drawn by animation
  • Func custom animation function, that is, pass in the function update just defined
  • Frames The length of an animation, the number of frames contained in a cycle
  • Init_func custom start frame, that is, pass in the newly defined function init
  • Interval Update frequency ms
        anim = FuncAnimation(fig,
                             update,
                             init_func=init,
                             frames=len(self._wbs),
                             interval=200)
Copy the code

According to the animation

            plt.show()
Copy the code

savegif

            anim.save(name, writer="imagemagick")
Copy the code

Save and display cannot be at the same time, wonder why?

Please check the specific code: github.com/iOSDevLog/s…

use

PIP install

pip install slmethod
Copy the code

The use method is very similar to sklearn, but the following steps can be omitted.

  1. To get the data
  2. Data preprocessing
  3. Divide test set and training set
  4. Estimator fit
  5. visualization
  6. Predictive test set
import numpy as np
from sklearn.datasets import make_blobs
from slmethod.perceptron import Perceptron

X, y = make_blobs(n_samples=500,
                  n_features=2,
                  centers=2,
                  cluster_std=0.2,
                  random_state=59)

y = np.where(y == 1.1.- 1)

# Primitive perceptron
origin_cls = Perceptron(dual=False)
origin_cls.fit(X, y)
origin_cls.show_anim()

# Dual form
dual_cls = Perceptron(dual=True)
dual_cls.fit(X, y)
dual_cls.show2d()
Copy the code

Code: github.com/iOSDevLog/s…