Python least square method solves the linear regression model

Machine learning linear regression model

Linear regression is a linear model that assumes a linear relationship between input variable X and a single output variable Y

Specifically, using a linear regression model, it is possible to calculate the output variable y: y=ax+b from a linear combination of a set of input variables X

Given an example with d attributes (features) described x = (x1; x2; … ; Xd), where Xi is the value of x on the ith attribute (feature). The Linear Model tries to learn a function that can predict by linear combination of attributes (feature), namely:

Generally written in vector form:

W = {w1,w2,w3,,,,,,}.

 

Many more powerful nonlinear models can be derived from linear models by introducing hierarchical structures or higher dimensional mappings.

Least square solution

The method of model solving based on the minimization of mean square error is called “least square method”.

Its main idea is to select unknown parameters to minimize the sum of squares of the difference between the theoretical value and the observed value.

 

We assume that there is only one number of input attributes (features) :

In linear regression, the least square method is an attempt to find a line that minimizes the sum of the Euclidean distances from all samples to the line.

Solving linear regression

Solve for w and b, such thatThe minimization process is called the “least squares parameter estimation” of the linear regression model.

Take the derivative of E(w,b) with respect to w and b, and you get

If you set the partial derivatives to 0, you get

–

Code implementation


### 0. Introduce dependencies
import numpy as np
import matplotlib.pyplot as plt
1. Import data
points = np.genfromtxt('data.csv',delimiter=', ')
Extract two columns of data in points as x and y respectively
x = points[:, 0]
y = points[:, 1]
# Use PLT to draw scatter plots
plt.scatter(x, y)
plt.show()


### 2. Define the loss function
The loss function is a function of the coefficients, plus the x, y of the data passed in
def compute_cost(w, b, points) :
    total_cost = 0
    M = len(points)
    
    # Calculate the square loss point by point, then calculate the average
    for i in range(M):
        x = points[i, 0]
        y = points[i, 1]
        total_cost += (y - w*x -b) ** 2
    return total_cost/M
    
    
### 3. Define the algorithm fitting function
Let's define a function to find the mean
def average(data) :
    sum = 0
    num = len(data)
    for i in range(num):
        sum += data[i]
        
    return sum/num
 
# Define the core fit function
def fit(points) :
    M = len(points)
    x_bar = average(points[:,0])
    
    sum_yx = 0
    sum_x2 = 0
    sum_delta = 0
    
    for i in range(M):
        x = points[i, 0]
        y = points[i, 1]
        sum_yx += y * (x - x_bar)
        sum_x2 += x ** 2
    # Calculate w according to the formula
    w = sum_yx / (sum_x2 - M * (x_bar ** 2))
    
    for i in range(M):
        x = points[i, 0]
        y = points[i, 1]
        sum_delta += (y - w * x)
    b = sum_delta / M
    
    return w, b


# # # 4. Test
w ,b = fit(points)
print("w is: ", w)
print("b is: ", b)
cost = compute_cost(w, b, points)
print("cost is : ",cost)
### 5. Draw the fit curve
plt.scatter(x,y)
# For each x, calculate the predicted y value
pred_y = w * x + b
plt.plot(x, pred_y, c='r')
plt.show()
Copy the code

If you’re interested in the Matlab version you can try it out for yourself,