Implement a simple proxy server

Proxy Server Overview

The function of a Proxy Server is to obtain network information on behalf of network users. Figuratively speaking, it is the network information transfer station, is the intermediary agency between individual network and Internet service provider, responsible for forwarding legitimate network information, control and registration of forwarding.

As a bridge between Internet and Intranet, proxy server plays an extremely important role in practical applications. It can be used for multiple purposes. The most basic function is connection, and it also includes security, cache, content filtering, access control management and other functions. More importantly, proxy server is an important security function provided by Internet link-level gateway, and its work is mainly in the open System Interconnection (OSI) dialogue layer

This use of Go language to achieve a simple HTTP proxy server, mainly divided into the following parts to complete:

Implement a simple Web server
Implement a simple proxy server
- Manual implementation
  - Configure the proxy WEB object through the INI file
  - Implement a basic proxy server based on access paths
  - Implement Basic authentication of proxy server
  - Implemented using the GO built-in proxy function
- Implement proxy server load balancing
  - Simple random load
  - IP_HASH load
  - Load weighted randomness
  - Polling load
    - Polling weighted
    - Smooth polling weighting
- Load balancing HTTPSERVER health check
  - Simple health check
  - Implementing a simple FailOver

Implement a simple Web server

Complete two Web servers using Go’s Http and listen on ports 9001 and 9002, respectively

type web1Handler struct{}

func (h web1Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	_, _ = writer.Write([]byte("WEB1"))}type web2Handler struct{}

func (h web2Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	_, _ = writer.Write([]byte("WEB2"))}func main(a) {

	c := make(chan os.Signal)

	go func(a) {
		_ = http.ListenAndServe(": 9001", web1Handler{})
	}()

	go func(a) {
		_ = http.ListenAndServe(": 9002", web2Handler{})
	}()

	signal.Notify(c, os.Interrupt)

	s := <-c

	log.Println(s)
}

Copy the code

Manual implementation

Configure the proxy WEB object through the INI file

Create env.ini file to store the list of WEB servers that need to be propped

[proxy]

[proxy.a]
path=/a
pass=http://localhost:9001

[proxy.b]
path=/b
pass=http://localhost:9002
Copy the code

Read configuration files. Read ini files using third-party dependencies

go get github.com/go-ini/ini
Copy the code

var ProxyConfigs map[string]string

type EnvConfig *os.File

func init(a) {
	ProxyConfigs = make(map[string]string)
	EnvConfig, err := ini.Load("env.ini")
	iferr ! =nil {
		fmt.Println(err)
	}
	section, _ := EnvConfig.GetSection("proxy")
	ifsection ! =nil {
		sections := section.ChildSections()
		for _, s := range sections {
			path, _ := s.GetKey("path")
			pass, _ := s.GetKey("pass")
			ifpath ! =nil&& pass ! =nil {
				ProxyConfigs[path.Value()] = pass.Value()
			}
		}
	}
}
Copy the code

Implement basic proxy functions based on access paths

Obtain the PrxoyConfigs configuration item list, and obtain the corresponding path and Web server access path

for k, v := range ProxyConfigs {
    fmt.Println(k,v)
    if matched, _ := regexp.MatchString(k, request.URL.Path); matched == true {
        // Proxy processing
        RequestUrl(request, writer, v)
        return
    }
}
_, _ = writer.Write([]byte("defaut"))
Copy the code

The proxy server implements Basic authentication by returning the original HTTP Request header and HTTP Response Header to the browser

Basic is a simple HTTP authentication method. The client transmits the user name and password to the server in plain text (Base64 encoding format) for authentication. Usually, HTTPS is required to ensure the security of information transmission

Basic authentication adds the www-Authenticate Header to the Response Header. After the browser identifies Basic, the dialog box Realm is displayed indicating the security domain of the protected document on the Web server

Enable Basic authentication for WEB1 servers

func (h web1Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	auth := request.Header.Get("Authorization")
	if auth == "" {
		writer.Header().Set("WWW-Authenticate".'Basic realm=" You must enter a username and password "')
		writer.WriteHeader(http.StatusUnauthorized)
		return
	}

	authList := strings.Split(auth, "")
	if len(authList) == 2 && authList[0] = ="Basic" {
		res, err := base64.StdEncoding.DecodeString(authList[1])
		if err == nil && string(res) == "tom:123" {
			_, _ = writer.Write([]byte(fmt.Sprintf("web1,form ip:%s", GetIp(request))))
			return
		}
	}
	_, _ = writer.Write([]byte("Wrong username or password"))}Copy the code

The effect is shown below:

What the proxy server needs to do is copy the input headers and output headers.

func CloneHead(src http.Header, dest *http.Header) {
	for k, v := range src {
		dest.Set(k, v[0])}}Copy the code

Proxy server Proxy logic

func RequestUrl(request *http.Request, writer http.ResponseWriter, url string) {
	fmt.Println(request.RemoteAddr)
	newReq, _ := http.NewRequest(request.Method, url, request.Body)

	CloneHead(request.Header, &newReq.Header)

	if ip := request.Header.Get(XForwardedFor); ip == "" {
		newReq.Header.Add(XForwardedFor, request.RemoteAddr)
	}

	response, _ := http.DefaultClient.Do(newReq)

	getHeader := writer.Header()
	CloneHead(response.Header, &getHeader)
	writer.WriteHeader(response.StatusCode)

	defer response.Body.Close()
	c, _ := ioutil.ReadAll(response.Body)
	_, _ = writer.Write(c)
}
Copy the code

, by using the method of manual implementation above agent already know about agent in logic, how about go whether existing proxy function, the answer is yes, direct use httpUtil. NewSingleHostReverseProxy direct implementation

for k, v := range ProxyConfigs {
    fmt.Println(k,v)
    if matched, _ := regexp.MatchString(k, request.URL.Path); matched == true {

        target, _ := url.Parse(v)
        proxy := httputil.NewSingleHostReverseProxy(target)
        proxy.ServeHTTP(writer, request)
        // RequestUrl(request, writer, v)
        return}}Copy the code

Implement proxy server load balancing

Load Balance refers to balancing loads (work tasks) and allocating them to multiple operation units, such as the Web server, FTP server, enterprise core application server, and other main task servers, so as to cooperatively complete work tasks.

Random load selects a random server from the server list for access through a random algorithm. According to probability theory, as the number of times the client calls the server increases, the actual effect tends to be equal distribution of each server requested to the server, that is, to achieve the effect of polling.

To facilitate the viewing, adjust the web server code and write the Web server access address to the proxy.

The web server

type web1Handler struct{}

func (h web1Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	_, _ = writer.Write([]byte("web1"))}type web2Handler struct{}

func (h web2Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	_, _ = writer.Write([]byte("web2"))}Copy the code

Add LoadBalance

package util

import (
	"math/rand"
	"time"
)

type HttpServer struct {
	Host string
}

type LoadBalance struct {
	Servers []*HttpServer
}

func NewHttpServer(host string) *HttpServer {
	return &HttpServer{
		Host: host,
	}
}

func NewLoadBalance(a) *LoadBalance {
	return &LoadBalance{
		Servers: make([]*HttpServer, 0),}}func (b *LoadBalance) AddServer(server *HttpServer) {
	b.Servers = append(b.Servers, server)
}

func (b *LoadBalance) SelectByRand(a) *HttpServer {
	rand.Seed(time.Now().UnixNano())
	index:=rand.Intn(len(b.Servers))
	return b.Servers[index]
}
Copy the code

Use the GO RAND function to randomly select the HTTPSERVER

Adjust the PROXY

func (*ProxyHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	defer func(a) {
		if err := recover(a); err ! =nil {
			writer.WriteHeader(500)
			_, _ = writer.Write([]byte("server error"))
			log.Println(err)
		}
	}()

	bl := NewLoadBalance()
	bl.AddServer(NewHttpServer("http://localhost:9001"))
	bl.AddServer(NewHttpServer("http://localhost:9002"))

	hostUrl, _ := url.Parse(bl.SelectByRand().Host)

	proxy := httputil.NewSingleHostReverseProxy(hostUrl)
	proxy.ServeHTTP(writer, request)
}
Copy the code

The end result: visit local http://localhost:8080 and the page will display either web1 or Web2 content randomly

The IP_HASH load is calculated based on the IP address of the client to which the request belongs and then sent to the corresponding backend.

Therefore, requests from the same client are sent to the same backend, unless the backend is unavailable, so IP_HASH can maintain the session effect.

This can be implemented in GO using the CRC algorithm (cyclic redundancy check) and the terminology algorithm.

ip:="127.0.0.1"
fmt.Println(crc32.ChecksumIEEE([]byte(ip)))
// Compute output by IP :3619153832
Copy the code

Added the method of getting HTTPSERVER by IP

func (b *LoadBalance) SelectByIpHash(ip string) *HttpServer {
	index := int(crc32.ChecksumIEEE([]byte(ip))) % len(b.Servers)
	return b.Servers[index]
}
Copy the code

Set the PROXY to the IP_HASH PROXY

ip := request.RemoteAddr
//hostUrl, _ := url.Parse(bl.SelectByRand().Host)
hostUrl, _ := url.Parse(bl.SelectByIpHash(ip).Host)

proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

In the end, the same server is accessed when accessing http://localhost:8080.

Load weighted randomization is a random algorithm that adds weights to httpServers. The httpServers are randomly selected based on the weights.

HTTPSERVER WERIGHT calculate the weight of the number of arrays, the weight will account for the number of arrays, random selection, the probability is greater.

Adjust the LoadBalance

/ / add WEIGHT
type HttpServer struct {
	Host   string
	Weight int
}


// Initialize LOADBALANCE and SERVERINDICES
var BL *LoadBalance
var ServerIndices []int

func init(a) {
	BL = NewLoadBalance()
	BL.AddServer(NewHttpServer("http://localhost:9001".5))
	BL.AddServer(NewHttpServer("http://localhost:9002".15))

	for index, server := range BL.Servers {
		if server.Weight > 0 {
			for i := 0; i < server.Weight; i++ {
				ServerIndices = append(ServerIndices, index)
			}
		}
	}

	fmt.Println(ServerIndices)
}

Copy the code

Adjust the PROXY to use random weighting

hostUrl, _ := url.Parse(BL.SelectByWeightRand().Host)

proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

The end result, when you visit http://localhost:8080, is 1 to 3, because the weights are now set to 5 and 15

Disadvantages: You need to generate an array slice for the HTTPSERVER list. If the weight value is set too high, it can cause memory problems.

The improved algorithm calculates the value range according to the weight

If the permission is set to 5:2:1,

Through 5, 7 (5 + 2), 8 (5 + 2 + 1)

[0,5] [5,7] [7,8]

Then according to [0,8) within a random number, random number in which interval, that is, which HTTPSERVER

Adjust the WEIGHT RAND method

func (b *LoadBalance) SelectByWeightRand2(a) *HttpServer {
	rand.Seed(time.Now().UnixNano())
	sumList := make([]int.len(b.Servers))
	sum := 0
	for i := 0; i < len(b.Servers); i++ {
		sum += b.Servers[i].Weight
		sumList[i] = sum
	}
	rad := rand.Intn(sum) / / /)
	for index, value := range sumList {
		if rad < value {
			return b.Servers[index]
		}
	}
	return b.Servers[0]}Copy the code

Adjust the PROXY to use modified methods

hostUrl, _ := url.Parse(BL.SelectByWeightRand2().Host)
proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

The polling load is the rotation of requests from users to internal servers: starting at server 1, going all the way to server N, and then starting the cycle again

Adjust the LoadBalance Server list and add the curIndex value to calculate the current HTTPSERVER

type LoadBalance struct {
	Servers  []*HttpServer
	CurIndex int // Points to the current server, 0 by default
}
Copy the code

Add a polling algorithm

func (b *LoadBalance) RoundRobin(a) *HttpServer {
	server := b.Servers[b.CurIndex]
	b.CurIndex = (b.CurIndex + 1) % len(b.Servers)
	return server
}
Copy the code

Use a polling algorithm

hostUrl, _ := url.Parse(BL.RoundRobin().Host)
proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

The end result is sequential access to HTTPSERVER

If the result is not as expected during implementation, check for a browser default request such as “/favicon.icon”.

Polling weighting adds weights on the basis of polling, which is basically consistent with the idea of load weighting random

Add polling weighted algorithm (using weighted array slices to calculate HTTPSERVER)

func (b *LoadBalance) RoundRobinByWeight(a) *HttpServer {
	server := b.Servers[ServerIndices[b.CurIndex]]
	b.CurIndex = (b.CurIndex + 1) % len(ServerIndices)
	return server
}
Copy the code

Use polling weights

hostUrl, _ := url.Parse(BL.RoundRobinByWeight().Host)
proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

Interval algorithm is used for polling weighting

func (b *LoadBalance) RoundRobinByWeight2(a) *HttpServer {
	server := b.Servers[0]
	sum := 0
	for i := 0; i < len(b.Servers); i++ {
		sum += b.Servers[i].Weight
		if b.CurIndex < sum {
			server = b.Servers[i]
			if b.CurIndex == sum- 1&& i ! =len(b.Servers)- 1 {
				b.CurIndex++
			} else {
				b.CurIndex = (b.CurIndex + 1) % sum
			}
			break}}return server
}
Copy the code

Interval weighting algorithm is used

hostUrl, _ := url.Parse(BL.RoundRobinByWeight2().Host)
proxy := httputil.NewSingleHostReverseProxy(hostUrl)
proxy.ServeHTTP(writer, request)
Copy the code

Smooth polling weighting is used to solve the original polling weighting existence must use the high weight of HTTPSERVER pressure is too big shortcomings, smooth polling weighting as long as ensure in the total weight number, HTTPSERVER as long as can appear its weight can, without the order of the execution of the high weight of HTTPSERVER, Then execute the low-weight HTTPSERVER.

Add the original WEIGHT to the HTTPSERVER by adding the CURWERIGHT value to the HTTPSERVER, and subtract the total WEIGHT from the HTTPSERVER. Up to HTTPSERVER WEIGHT is 0.

Examples are as follows:

The weight	hit	Weight after hit
{s1:3, S2 :1,s3:1} (initialization weight)	S1 (maximum)	{s1:-2,s2:1:s3:1} s1 minus 5
{s1:-2,s2:2,s3:2} add 3 to s1 and 1 to others	s2	{s1:1,:s2:-3,s3:2} s2 minus 5
{s1:4, s2: – 2, s3:3}	s1	{s1:-1,:s2:-2,s3:3} s1 minus 5
{s1:2, s2:1, s3:4}	s3	{s1:2,:s2:-1,s3:-1} s3 minus 5
{s1:5,s2:0,s3:0}	s1	{s1:0,s2:0,s3:0} s1 minus 5

Adjust HTTPSERVER and add CURWEIGHT

type HttpServers []*HttpServer
type HttpServer struct {
	Host      string
	Weight    int
	CurWeight int // Default is 0
}
Copy the code

Add a smooth polling method

func (b *LoadBalance) RoundRobinByWeight3(a) *HttpServer {
	for _, s := range b.Servers {
		s.CurWeight = s.CurWeight + s.Weight
	}
	sort.Sort(b.Servers)
	fmt.Println(b.Servers)
	max := b.Servers[0]
	max.CurWeight = max.CurWeight - SumWeight
    
	test := ""
	for _, s := range b.Servers {
		test += fmt.Sprint(s.Host,s.CurWeight, ",")
	}
	fmt.Println(test)
    
	return max
}
Copy the code

Load balancing HTTPSERVER health check

Simple health check

The STATUS of the HTTP service is changed periodically

The HEAD request in HTTP returns only the HTTP header, not the HTTP BODY, avoiding too much BODY content and small transmission.

Add the STATUS attribute to HTTPSERVER

type HttpServer struct {
	Host      string
	Weight    int
	CurWeight int    // Default is 0
	Status    string // Status, default UP, DOWN DOWN
}
Copy the code

Add a periodic check object

package util

import (
	"net/http"
	"time"
)

type HttpChecker struct {
	Servers HttpServers
}

func NewHttpChecker(servers HttpServers) *HttpChecker {
	return &HttpChecker{
		Servers: servers,
	}
}

func (h *HttpChecker) Check(timeout time.Duration) {
	client := http.Client{
		Timeout: timeout,
	}
	for _, s := range h.Servers {
		res, err := client.Head(s.Host)
		ifres ! =nil {
			res.Body.Close()
		}
		iferr ! =nil {
			s.Status = "DOWN"
			continue
		}
		if res.StatusCode >= 200 && res.StatusCode < 400 {
			s.Status = "UP"
		} else {
			s.Status = "DOWN"}}}Copy the code

The check object is called when the server is initialized

func checkServers(servers HttpServers) {
	t := time.NewTicker(time.Second * 3)
	check := NewHttpChecker(servers)
	for {
		select {
		case <-t.C:
			check.Check(time.Second * 2)
			for _, s := range servers {
				fmt.Println(s.Host, s.Status)
			}
			fmt.Println("-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -")}}}go func(a) {
    checkServers(BL.Servers)
}()
Copy the code

The HTTPSERVER STATUS is marked DOWN when the server is shut DOWN and UP when the server is started

---------------------
http://localhost:9001 UP
http://localhost:9002 UP
http://localhost:9003 UP
---------------------
http://localhost:9001 UP
http://localhost:9002 DOWN
http://localhost:9003 DOWN
---------------------
http://localhost:9001 UP
http://localhost:9002 DOWN
http://localhost:9003 DOWN
---------------------
http://localhost:9001 DOWN
http://localhost:9002 DOWN
http://localhost:9003 DOWN
---------------------
http://localhost:9001 UP
http://localhost:9002 UP
http://localhost:9003 UP
---------------------

Copy the code

Implement a simple FailOver combined with health check to handle faulty HttpServers

Counter algorithm

Add FAILCOUNT and SUCCESSCOUNT properties to HTTPSERVER,

Add FAILMAX and RECOVERCOUNT properties to HTTPCHECKER

type HttpServer struct {
	Host      string
	Weight    int
	CurWeight int    // Default is 0
	Status    string // Status, default UP, DOWN DOWN
	FailCount int    // Number of errors 0
	SuccessCount int
}

type HttpChecker struct {
	Servers      HttpServers
	FailMax      int
	RecoverCount int
}

func NewHttpChecker(servers HttpServers) *HttpChecker {
	return &HttpChecker{
		Servers:      servers,
		FailMax:      6,
		RecoverCount: 3.// Continuously successful, this value is marked UP}}Copy the code

HTTPCHECKER adds failed and successful method handling methods

func (h *HttpChecker) Fail(server *HttpServer) {
	if server.FailCount >= h.FailMax {
		server.Status = "DOWN"
	} else {
		server.FailCount++
	}
	server.SuccessCount = 0
}

func (h *HttpChecker) Success(server *HttpServer) {
	if server.FailCount > 0 {
		server.FailCount--
		server.SuccessCount++
		if server.SuccessCount == h.RecoverCount {
			server.FailCount = 0
			server.Status = "UP"
			server.SuccessCount = 0}}else {
		server.Status = "UP"}}Copy the code

Add the FAILOVER mechanism for common polling

Notice that all servers are DOWN

// Check whether all servers are DOWN
func (b *LoadBalance) IsAllDown(a) bool {
	downCount := 0
	for _, s := range b.Servers {
		if s.Status == "DOWN" {
			downCount++
		}
	}
	if downCount == len(b.Servers) {
		return true
	}
	return false
}
// Common polling
func (b *LoadBalance) RoundRobin(a) *HttpServer {
	server := b.Servers[b.CurIndex]
	b.CurIndex = (b.CurIndex + 1) % len(b.Servers)
	// recursive query
	if server.Status == "DOWN" && !b.IsAllDown() {
		return b.RoundRobin()
	}
	return server
}
Copy the code

Add FAILWEIGHT to the HTTPSERVER. The FAILWEIGHT is specified by the current FAILWEEIGHT+=WEIGHT*(1/FailFactor). If the value is 0, the server is DOWN. If the HTTPSERVER health check succeeds, the FAILWEIGHT is set to 0

Added weighted polling FAILOVER

type HttpServer struct {
	Host         string
	Weight       int
	CurWeight    int    // Default is 0
	FailWeight   int    // Reduce the weight
	Status       string // Status, default UP, DOWN DOWN
	FailCount    int    // Number of errors 0
	SuccessCount int
}

type HttpChecker struct {
	Servers      HttpServers
	FailMax      int
	RecoverCount int
	FailFactor   float64 // The weight reduction factor is 5.0 by default
}


func (h *HttpChecker) Fail(server *HttpServer) {
	if server.FailCount >= h.FailMax {
		server.Status = "DOWN"
	} else {
		server.FailCount++
	}
	server.SuccessCount = 0

	fw := int(math.Floor(float64(server.Weight)) * (1 / h.FailFactor))
	if fw == 0 {
		fw = 1
	}
	server.FailWeight += fw
	if server.FailWeight > server.Weight {
		server.FailWeight = server.Weight
	}
}

func (h *HttpChecker) Success(server *HttpServer) {
	if server.FailCount > 0 {
		server.FailCount--
		server.SuccessCount++
		if server.SuccessCount == h.RecoverCount {
			server.FailCount = 0
			server.Status = "UP"
			server.SuccessCount = 0}}else {
		server.Status = "UP"
	}

	server.FailWeight = 0
}

// Interval algorithm
func (b *LoadBalance) RoundRobinByWeight2(a) *HttpServer {
	server := b.Servers[0]
	sum := 0
	for i := 0; i < len(b.Servers); i++ {

		// Check whether the weight is 0, which indicates that the server is unavailable
		realWeight := b.Servers[i].Weight - b.Servers[i].FailWeight
		if realWeight == 0 {
			continue
		}
		//sum += b.Servers[i].Weight
		sum += realWeight
		if b.CurIndex < sum {
			server = b.Servers[i]
			if b.CurIndex == sum- 1&& i ! =len(b.Servers)- 1 {
				b.CurIndex++
			} else {
				b.CurIndex = (b.CurIndex + 1) % sum
			}
			break
		} else {
			b.CurIndex = 0}}return server
}
Copy the code

Smooth weighting FAILOVER is basically the same as normal polling weighting, as long as the true weight is obtained in the smooth weighting method

func (b *LoadBalance) getSumWeight(a) int {
	sum := 0
	for _, s := range b.Servers {
		realWeight := s.Weight - s.FailWeight
		if realWeight > 0 {
			sum += realWeight
		}
	}
	return sum
}


func (b *LoadBalance) RoundRobinByWeight3(a) *HttpServer {

	for _, s := range b.Servers {
		s.CurWeight = s.CurWeight + s.Weight - s.FailWeight // Get the real weight
	}

	sort.Sort(b.Servers)

	fmt.Println(b.Servers)

	max := b.Servers[0]
	// max.CurWeight = max.CurWeight - SumWeight
	max.CurWeight = max.CurWeight - b.getSumWeight()

	test := ""
	for _, s := range b.Servers {
		test += fmt.Sprint(s.Host, s.CurWeight, ",")
	}
	fmt.Println(test)

	return max
}

Copy the code

Implement a simple proxy server

Proxy Server Overview

Implement a simple Web server