What could go wrong if Goroutine numbers were not controlled?

First, we all know that Goroutine has the following two characteristics:

  • Light size (small memory, about 2KB)
  • Excellent GMP scheduling (see Golang’s GMP principles and scheduling process)

Is it true that Goroutine can be opened indefinitely? If you’re working on a server or some high-business scenario, can you turn on Goroutine at will and not control its life cycle? Let these Goroutines die on their own, most people would think: after all, with a strong GC and a good scheduling algorithm, it should be unlimited.

Let’s start with an example:

demo1.go

package main

import (
    "fmt"
    "math"
    "runtime"
)

func main(a) {
    // Simulate the number of services required by the user
    task_cnt := math.MaxInt64

    for i := 0; i < task_cnt; i++ {
        go func(i int) {
            / /... do some busi...

            fmt.Println("go func ", i, " goroutine count = ", runtime.NumGoroutine())
        }(i)
    }
}
Copy the code

In the end, the operating system sends a kill signal to forcibly terminate the process.

signal: killed
Copy the code

As you can see, if we quickly turn on goroutine (without controlling the number of concurrent Goroutines), it will occupy the operating system resources (CPU, memory, file descriptors, etc.) for a short time.

  • The CPU usage fluctuates
  • Memory usage continues to rise
  • Main process crashed (killed)

These resources are actually shared by all user-mode programs, so a large number of Goroutine openings can eventually cause problems not only for themselves, but also for other running programs.

To sum up, so when we write code in peacetime development, we need to pay attention to the number of goroutine opened in the code, and whether the goroutine opened can be controlled, we must pay attention to the problem.

Several ways to control goroutine

Method 1: Limit channels with buffers

package main

import (
	"fmt"
	"math"
	"runtime"
)

// Simulate a goroutine executing a business
func doBusiness(ch chan bool, i int) {
	fmt.Println("go func:", i, "goroutine count:", runtime.NumGoroutine())
	<-ch
}

func main(a) {
	max_cnt := math.MaxInt64
	// max_cnt := 10
	fmt.Println(max_cnt)

	ch := make(chan bool.3)

	for i := 0; i < max_cnt; i++ {
		ch <- true
		go doBusiness(ch, i)
	}
}
Copy the code

Results:

...
go func  352283  goroutine count =  4
go func  352284  goroutine count =  4
go func  352285  goroutine count =  4
go func  352286  goroutine count =  4
go func  352287  goroutine count =  4
go func  352288  goroutine count =  4
go func  352289  goroutine count =  4
go func  352290  goroutine count =  4
go func  352291  goroutine count =  4
go func  352292  goroutine count =  4
go func  352293  goroutine count =  4
go func  352294  goroutine count =  4
go func  352295  goroutine count =  4
go func  352296  goroutine count =  4
go func  352297  goroutine count =  4
go func  352298  goroutine count =  4
go func  352299  goroutine count =  4
go func  352300  goroutine count =  4
go func  352301  goroutine count =  4
go func  352302  goroutine count =  4
...
Copy the code

As a result, the program does not crash, but executes sequentially, and the number of Go’s is limited to 3 (4 because there is also a main Goroutine).

Here we use a channel buffer of 3, which actually limits the speed of writing code to the channel:

 for i := 0; i < go_cnt; i++ {
        ch <- true
        go busi(ch, i)
}
Copy the code

The speed of ch< -true in the for loop, because this speed determines how fast go can be created, and how fast go can end depends on how fast doBusiness() is executed. In effect, we can guarantee that the number of Goroutines running at the same time is the same as the number of buffers, thus achieving a limited effect.

However, there is a small problem with this code, that is, if we make the number of go_cnt smaller, the result will be typed incorrectly.

package main

import (
	"fmt"
	// "math"
	"runtime"
)

func doBusiness(ch chan bool, i int) {
	fmt.Println("go func:", i, "goroutine count:", runtime.NumGoroutine())
	<-ch
}

func main(a) {
	// max_cnt := math.MaxInt64
	max_cnt := 10

	ch := make(chan bool.3)

	for i := 0; i < max_cnt; i++ {
		ch <- true
		go doBusiness(ch, i)
	}
}
Copy the code

Results:

go func  2  goroutine count =  4
go func  3  goroutine count =  4
go func  4  goroutine count =  4
go func  5  goroutine count =  4
go func  6  goroutine count =  4
go func  1  goroutine count =  4
go func  8  goroutine count =  4
Copy the code

You can see that some of the goroutines don’t print out, because after Main turns all of the goroutines on, main just exits, and we know that when main exits, all of the goroutines end, As a result, some Goroutines quit before they could be implemented. So to execute all of the Goes, you need to block at the end of main.

Method 2: Use sync

import ( "fmt" "math" "sync" "runtime" ) var wg = sync.WaitGroup{} func doBusiness(i int) { fmt.Println("go func ", i, " goroutine count = ", Runtime.numgoroutine ()) wg.done ()} func main() {task_cnt := math.maxint64 for I := 0; i < task_cnt; i++ { wg.Add(1) go doBusiness(i) } wg.Wait() }Copy the code

Obviously, using Sync alone would not control the number of Goroutines, and the end result would still be a crash.

Method 3: Channel and sync synchronization combination to achieve control of Goroutine

package main import ( "fmt" "math" "sync" "runtime" ) var wg = sync.WaitGroup{} func doBusiness(ch chan bool, i int) { fmt.Println("go func ", i, " goroutine count = ", Task_cnt := math.MaxInt64 ch := make(chan bool) 3) for i := 0; i < task_cnt; i++ { wg.Add(1) ch <- true go doBusiness(ch, i) } wg.Wait() }Copy the code

Results:

/ /... go func 228856 goroutine count = 4 go func 228857 goroutine count = 4 go func 228858 goroutine count = 4 go func 228859 goroutine count = 4 go func 228860 goroutine count = 4 go func 228861 goroutine count = 4 go func 228862 goroutine count  = 4 go func 228863 goroutine count = 4 go func 228864 goroutine count = 4 go func 228865 goroutine count = 4 go func 228866 goroutine count = 4 go func 228867 goroutine count = 4 //...Copy the code

So our program doesn’t crash with a resource explosion. And the number of Go runs is controlled within the range of buffer 3.

Method 4: Use unbuffered channel and task send/execution separation

package main import ( "fmt" "math" "sync" "runtime" ) var wg = sync.WaitGroup{} func doBusiness(ch chan int) { for t := range ch { fmt.Println("go task = ", t, ", goroutine count = ", runtime.NumGoroutine()) wg.Done() } } func sendTask(task int, Ch chan int) {wg.add (1) ch < -task} func main() {ch := make(chan int) no buffer channel goCnt := 3 for i := 0; i < goCnt; I++ {// start go go doBusiness(ch)} taskCnt := math.maxint64; t < taskCnt; T ++ {sendTask(t, ch)} wg.wait ()}Copy the code

The execution process is roughly as follows. In fact, the sending and execution of tasks are separated from each other. To make the message go out, the frequency of typing SendTask can be set, as can the number of Goroutine executions. That is to control both inputs (production) and outputs (consumption). Make control more flexible. This is also the initial design concept of many Go frameworks’ Worker work pools. The diagram below:

The above is the content to share with you today, more high-quality technical articles welcome to pay attention to the public number [Go keyboard man].