1. Define requirements

When we use the Go concurrent invocation interface to make HTTP requests, it’s easy to just add the go keyword in front of func(). Because concurrency is so easy, we sometimes need to control the number of concurrent requests.

Now there is a requirement: there are 10 million mobile phone numbers in the local area. The interface of the home of the aggregated data mobile phone number needs to be called and the query result data such as province, city, area code, zip code and operator needs to be recorded.

Tips: The test interface is aggregation data free: mobile phone number home interface. If you need to test the following test cases, you are advised to register the interface first. Aggregated data also provides a number of free interfaces for developers to debug.

Let’s first simulate the outage problem caused by uncontrolled coroutines and create a new onewarning.goAs follows:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "net/url"
)

func main(a) {
    // The interface requests the URL
    apiUrl := "http://localhost" // Do not use interface address tests
    max := 1<<63 - 1             // The maximum integer, simulating a large amount of data

    // Initialize parameters
    param := url.Values{}
    // Set the request parameters. The urlencode problem has been handled inside the method
    param.Set("a"."test") // The mobile phone number to be queried or the first seven digits of the mobile phone number

    for i := 0; i < max; i++ {
        go func(i int) {
            // Some logical code...
            fmt.Printf("start func: %dn", i)
            // Send the request
            data, err := Get(apiUrl, param)
            iferr ! =nil {
                fmt.Println(err)
                return
            }
            fmt.Println(string(data))
        }(i)
    }
}

// Initiate a network request in Get mode
func Get(apiURL string, params url.Values) (rs []byte, err error) {
    var Url *url.URL
    Url, err = url.Parse(apiURL)
    iferr ! =nil {
        fmt.Printf("Error parsing URL :rn%v", err)
        return nil, err
    }
    // If there are Chinese arguments in the argument, the method will URLEncode
    Url.RawQuery = params.Encode()
    resp, err := http.Get(Url.String())
    iferr ! =nil {
        fmt.Println("err:", err)
        return nil, err
    }
    defer resp.Body.Close()
    return ioutil.ReadAll(resp.Body)
}
Copy the code

High energy command:!! Do not execute!!

go run waning.go
Copy the code

The screenshot of the instant top command is shown below, and you can see that the CPU is suddenly up:

Since there is no limit to the number of Goroutines, it would be more efficient if all tasks were executed in concurrent Goroutines. But when the goroutine is created without control, the server system resource usage spikes and goes down until the process is automatically killed or no other service can be provided.

Second, with buffer channel control concurrency

In the above case, we did not control the number of Goroutines, which caused the outage, so as long as we control the number of Goroutines, we can avoid this problem!

Here is the code to use sync.waitGroup {} to control the amount of concurrency. Create a new main.go and say:

package main

import(
    "fmt"
    "io/ioutil"
    "math/rand"
    "net/http"
    "net/url"
    "sync"
    "time"
)

type Limit struct {
    number  int
    channel chan struct{}}// Limit struct initialization
func New(number int) *Limit {
    return &Limit{
        number:  number,
        channel: make(chan struct{}, number),
    }
}

// The Run method creates a limited goroutine of the go f function
func (limit *Limit) Run(f func(a)) {
    limit.channel <- struct{} {}go func(a) {
        f()
        <-limit.channel
    }()
}

// The WaitGroup object has a counter inside, starting at 0
// There are three methods: Add(), Done(), Wait() to control the number of counters
var wg = sync.WaitGroup{}

const (
    concurrency = 5 // Control concurrency
)

func main(a) {
    start := time.Now()
    limit := New(concurrency) // New Limit controls the number of concurrent requests
    // The interface requests the URL
    apiUrl := "http://apis.juhe.cn/mobile/get" // Do not use interface address tests
    // Max := int(math.pow10 (8)
    max := 5                                    // Test 5 times first

    // Initialize parameters
    param := url.Values{}
    param.Set("key"."The KEY of your application") // The interface requests the Key

    for i := 0; i < max; i++ {
        wg.Add(1)
        value := i
        goFunc := func(a) {
            fmt.Printf("start func: %dn", value)
            // Set the request parameters. The urlencode problem has been handled inside the method
            phone := RandMobile()
            param.Set("phone", phone) // The mobile phone number to be queried or the first seven digits of the mobile phone number
            // Send the request
            data, err := Get(apiUrl, param)
            iferr ! =nil {
                fmt.Println(err)
                return
            }
            // Other logic code...
            fmt.Println("phone: ", phone, string(data))
            wg.Done()
        }
        limit.Run(goFunc)
    }

    // Block code to prevent exit
    wg.Wait()

    fmt.Printf("Time: %fs", time.Now().Sub(start).Seconds())
}

// Initiate a network request in Get mode
func Get(apiURL string, params url.Values) (rs []byte, err error) {
    var Url *url.URL
    Url, err = url.Parse(apiURL)
    iferr ! =nil {
        fmt.Printf("Error parsing URL :rn%v", err)
        return nil, err
    }
    // If there are Chinese arguments in the argument, the method will URLEncode
    Url.RawQuery = params.Encode()
    resp, err := http.Get(Url.String())
    iferr ! =nil {
        fmt.Println("err:", err)
        return nil, err
    }
    defer resp.Body.Close()
    return ioutil.ReadAll(resp.Body)
}

var MobilePrefix = [...]string{"130"."131"."132"."133"."134"."135"."136"."137"."138"."139"."145"."147"."150"."151"."152"."153"."155"."156"."157"."158"."159"."170"."176"."177"."178"."180"."181"."182"."183"."184"."185"."186"."187"."188"."189"}

// GeneratorPhone generates a mobile phone number
func RandMobile(a) string {
    return MobilePrefix[RandInt(0.len(MobilePrefix))] + fmt.Sprintf("%0*d".8, RandInt(0.100000000))}// Specify range random int
func RandInt(min, max int) int {
    rand.Seed(time.Now().UnixNano())
    return min + rand.Intn(max-min)
}
Copy the code

Go run main.go and the result is as follows:

It’s easier to use go buffered channels to control the number of concurrent goroutines, and we can also wrap the New() and Run() methods into external projects.

Expand your thinking

Here are two questions you can try to think about.

Thought 1: Why do we use Sync.waitGroup?

If we do not block the main process with sync.waitgroup, the subcoroutine will be aborted when the main program finishes. So the program ends before the remaining Goroutine comes and executes.

Thought 2: Why does the code define a channel data structure as struct instead of bool?

Because empty struct variables have a memory footprint of 0 and bool variables have a memory footprint of 1, this maximizes the server’s memory footprint.

func main(a){
  a :=struct{}{}
  b := true
  fmt.Println(unsafe.Sizeof(a))  / / 0
  fmt.Println(unsafe.Sizeof(b))  / / 1
}
Copy the code

Original link: www.sdk.cn/details/350…

SDK is a neutral communities, there are a variety of front knowledge, has the rich API, developers have a love of learning artificial intelligence, have humor developers take you learn python, there will be the next hot HongMeng, when various elements together, let us together to build professional, fun and imaginative valuable developer community, Help developers realize their self-worth!