preface

Recently, I am learning Go language, following the concept of “the best way to learn a language is to use it”, thinking about what to achieve with Go. I happen to have a problem that annoys me in my work, so I use Go to solve it. Even if it is not used in the production environment, it can also be a way to learn Go language.

First, let’s introduce the problem:

There are more than ten machines in the group, which use Cron to execute some scripts and shell commands regularly. At the beginning, when there are few tasks, everyone remembers which machine is executing what. As time goes by, after several changes of personnel, there are more and more tasks, and no one can remember which tasks are executing on which machine. Troubleshooting and solving problems with background scripts is becoming more and more troublesome.

There are ways to solve this problem:

  • Maintain a wiki and update the wiki whenever a task changes, but forget to update the wiki and the task becomes an orphan, making it harder to check when something goes wrong.
  • Set up a machine, periodically pull cron configuration files from each machine, compare statistics, and display the results together, but the command is written in a variety of ways, comparing command is also a mindless thing.
  • The use of open source distributed task scheduling tasks, more heavy, and generally to decorate the database, background, more trouble.

In addition, task modification is also very inconvenient. If you want to modify a task in crontab, you need to find o&M operations. Although there is a solution to this problem, using crontab cronfile. TXT to directly make crontab load files, a new problem is introduced: the real-time loading of task files is not easy to control.

To solve these problems, I combined Cron and task management, and spent a little time each day after work implementing a small feature that ended up in a working version of GoTorch. Look at GitHub’s commit statistics

 

GitHub- Zhenbianshu-Gotorch welcome star/fork/issue

Introduce the features:

  • Cron +, second level timing, making task execution more flexible;
  • The task list file path can be customized. You are advised to use the version control system.
  • Built-in log and monitoring system, convenient for students to expand arbitrarily;
  • The configuration file is smoothly reloaded. Once the configuration file is changed, the configuration file is smoothly loaded without affecting the ongoing tasks.
  • IP address, maximum number of tasks to execute, and task type, supporting more flexible task configuration;

Here are the technical points of function realization:

Welcome to reprint, but please take this use address: http://www.cnblogs.com/zhenbianshu/p/7905678.html, thank you.

cron+

Before implementing cron-like features, I briefly looked at the cron source code, which can be downloaded at busybox.net/downloads/ and unpacked at miscutils > crond.c.

Cron’s implementation is cleverly designed as follows:

Data structure:

  1. Cron has a global structure, Global, that holds the task list for each user;
  2. Each task list is a CronFile structure that holds the user name, task list, etc.
  3. Each task CronLine has shell command, execution PID, execution time array cl_Time and other attributes.
  4. The maximum length of the execution time array is determined by the maximum value of “time sharing day month week”, and the value of the executable time point is set totrue, for example, at 3 o ‘clock every daycl_Hrs[3]=true;

Mode of execution:

  1. Cron is awhile(true)Type of long loop, each timesleepTo the beginning of the next minute.
  2. Cron iterates through the user cron profile at the beginning of each minute, parsing the updated profile into a task to store in the global structure, and periodically checking for changes to the profile.
  3. Cron then resolves the current time toThe n minute/hour/day/month/weekAnd check if cal_Time[n] is true, the task is executed.
  4. Write pid when executing a task to prevent repeated execution;
  5. Cron also performs some subsequent exception detection and error handling.

After understanding the execution mode of CRON, I felt that traversing tasks for judgment in each time unit would cause performance loss. Moreover, I realized the second-level execution, and the performance loss of traversing judgment would be greater, so I considered optimization as follows:

Set a next_time timestamp for each task and update the timestamp after a single execution. For each unit of time, just check task.next_time == current_time.

Later,, due to the irregular carry of the date format of “second time, day, month and week”, the code is too complex, and the implementation efficiency is not better than the original, finally gave up this idea. It uses the same execution idea as Cron.

In addition, I added three ways to limit the execution of tasks:

  • IP: obtain the local Intranet IP address when the service is started and check whether the IP address is in the IP address list before the task is performed.
  • Task type: If the task is daemon, the system interrupts and starts the task directly when the task is not being executed.
  • Maximum number of executions: Set a slice composed of the PID of an executing task on each task, and verify the current number of executions before each execution.

In the task startup mode, goroutine is directly used with the exec package. Each execution of the task starts a new Goroutine, saves the PID, and handles errors at the same time. Since the service may scan the task multiple times in one second, I added the attribute of the timestamp of the last execution of a process to each task, which will be compared in the next execution to prevent the task from being scanned multiple times in one second.

daemon

This service is made of a similar nginx service, I will process PID saved in a temporary file, on the process operation through the command line to send signals to the process, only need to pay attention to the abnormal case timely clean pid file.

Here’s how the Go daemon is created:

Because the Runtime may create multiple threads (for memory management, garbage collection, Goroutine management, etc.) when the Go program is started, fork and multi-threaded environment do not coexist, so there is no fork method in Go in Unix systems. To start the daemon, I use exec immediately after execution, which is fork and exec, which is supported by the Go exec package.

If the parent process is not 1, fork and exec a process that is the same as your own. If the parent process is not 1, fork and exec a process that is the same as your own. Close the connection between the new process and the terminal, and exit the original process.

CMD := exec.Command(filePath, os.args [1:]...) Cmd.stdin = nil cmd.stdout = nil cmd.stderr = nil cmd.stderr = nil cmd.stdout = nil cmd.stderr = nil cmd.stderr = nil cmd.stderr = nil cmd.stdout = nil cmd.start ( Return // The parent process exitsCopy the code

Semaphore processing

Once a process is made a daemon, communication between the process and the outside world has to rely on semaphores. The Go signal package and Goroutine can easily listen to and process semaphores. We also use the Kill method in the Syscall package to send the semaphore to the process.

We listen to SIGTERM, a semaphore sent by Kill by default, to handle cleanup before the service exits, and I use SIGUSR2, a user-defined semaphore sent by the terminal to notify the service of restart.

The complete process of a semaphore from monitoring to capturing to processing is as follows:

  1. First we use create a type ofos.SygnalUnbuffered channel to store the semaphore.
  2. usesignal.Notify()The function registers the semaphore to listen for, passes in the channel you just created, and receives the semaphore when it is captured.
  3. Create a Goroutine when there is no signal in a channelsignal := <-channelWill be blocked.
  4. Once the Go program catches the semaphore it is listening for, it passes the semaphore through a channel, and the Goroutine stops blocking.
  5. The corresponding semaphore is processed through the following code.

The corresponding code is as follows:

c := make(chan os.Signal) signal.Notify(c, syscall.SIGTERM, Syscall.sigusr2) // Start a goroutine async signal go func() {s := <-c if s == syscall.sigterm {task.end () logger.Debug("bootstrap", "action: end", "pid "+strconv.Itoa(os.Getpid()), "signal "+fmt.Sprintf("%d", s)) os.Exit(0) } else if s == syscall.SIGUSR2 { task.End() bootStrap(true) } }()Copy the code

summary

Gotorch’s development took about three months, with 1 to 3 commits per day, half an hour, and three major rebuilds, particularly frequent changes in code format. But use Go development is really quite comfortable, Go code is very simple, GofMT is very convenient to use. Go also has a smooth learning curve, and simple development is possible once you’re familiar with the standard packages. Easy to learn and efficient, it’s no wonder Go has become so popular.

If you have any questions about this article, please leave a comment below. If you think this article is helpful to you, you can click the recommendation below to support me. The blog is always updated, welcome to follow.

Reference:

On fork() and multithreaded programming in Linux

Linux semaphores SIGNAL details