Recently do a small program development task, mainly responsible for the background part of development; According to project requirements, three scheduled tasks need to be implemented:

  • Update wechat token regularly, every 2 hours;

  • Product timing online;

  • Periodically check whether the background service is alive.

    Using Python to do these three tasks, you need to use timing knowledge;

There are many ways to implement fixed and scheduled tasks in Python. Find the following four ways, each of which has its own application scenario. Here’s a quick look at some common implementations of scheduled tasks in Python:

1. Cycle +sleep;

2. Timer class in thread module;

3. The schedule module;

4. Timing framework: APScheduler

Set a task before you start (so you don’t have to rely on external circumstances) :

1: Monitors CPU and memory usage periodically or at a fixed point.

2: Saves the time, CPU, and memory usage to log files.

First to realize the system monitoring function:

Preparations: Install psutil: PIP install psutil

Function implementation

Import psutil import time import datetime #logfile: Def MonitorSystem(logfile = None): Cpuper = psutil.cpu_percent() System memory size, used memory, available memory, Mem = psutil.virtual_memory() # memper = mem.percent # now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:%M:%S') line = f'{ts} cpu:{cpuper}%, mem:{memper}%' print(line) if logfile: logfile.write(line) 12345678910111213141516171819Copy the code

Code running results:

CPU :0.6%, MEM :77.2% 1Copy the code

Next, we need to implement periodic monitoring, such as 3S to monitor the use of system resources.

Easiest way to use: Sleep

The simplest way to do this is to use while+sleep:

Def loopMonitor(): while True: MonitorSystem() #2s time. Sleep (3) loopMonitor() 123456Copy the code

Output result:

2019-03-21 14:28:42 CPU :1.5, mem:77.6% 2019-03-21 14:28:45 CPU :1.6, mem: 1.0% Mem: 1.0%, meM: 1.0%, meM: 1.0%, meM: 1.0%Copy the code

This approach has a problem: Only a single scheduled task can be processed.

New task: need to monitor the network sent and received bytes per second, the code implementation is as follows:

def MonitorNetWork(logfile = None): Now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:% m :%S')  line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}' print(line) if logfile: logfile.write(line) MonitorNetWork() 1234567891011Copy the code

Code execution result:

2019-03-21 14:47:21 bytessent=169752183, bytesrecv=1107900973
1
Copy the code

If we monitor two tasks simultaneously in a while loop, we have a wait problem and cannot monitor the network every second.

Implementation of Timer

The most basic understanding of timer is timer. We can start multiple scheduled tasks. These timer tasks are executed asynchronously, so there is no problem of waiting for sequential execution.

Let’s start with the basic use of Timer:

Import: from threading import Timer

Main methods:

Timer(interval, function, args=None, kwargs=None) Create a Timer Cancel () Cancel the Timer start() Execute join(self, Timeout =None) wait for the thread to finish

The timer can be executed only once. If the timer needs to be executed repeatedly, you need to add another task.

Let’s start with the basics:

Print (datetime.datetime.now()) print(datetime.datetime.now()) MonitorSystem) #1S Execute once nTimer = Timer(1, Stimer.start () ntimer.start () # Wait for end stimer.join () ntimer.join () # Record end time print(datetime.datetime.now()) 123456789101112131415Copy the code

Output result:

Bytessent =171337324, bytesRecv =1109002349 2019-03-21 15:13:39 CPU :1.4%, Mem: 93.2% of the 2019-03-21 15:13:39. 745187, 1234Copy the code

As you can see, it takes 3S, but what we want to do is monitor the network every second; How to deal with it.

The Timer can only be executed once, so the task needs to be added again after the execution is completed. We modify the code:

from threading import Timer import psutil import time import datetime def MonitorSystem(logfile = None): cpuper = psutil.cpu_percent() mem = psutil.virtual_memory() memper = mem.percent now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:%M:%S') line = f'{ts} cpu:{cpuper}%, mem:{memper}%' print(line) if logfile: Start () def MonitorNetWork(logfile = None): netinfo = psutil.net_io_counters() now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:%M:%S') line = f'{ts} bytessent={netinfo.bytes_sent}, bytesrecv={netinfo.bytes_recv}' print(line) if logfile: Logfile.write (line) # start timer task, Per second to perform a Timer (1, MonitorNetWork). The start () MonitorSystem MonitorNetWork () () 12345678910111213141516171819202122232425262728Copy the code

Execution Result:

2019-03-21 15:18:21 CPU :1.5%, mem:93.2% 2019-03-21 15:18:21 bytessent=171376522, bytesrecv=1109124678 2019-03-21 15:18:22 bytessent=171382215, Bytesrecv = 10912864, bytesrecv= 10912864, bytesrecv= 10912864, Mem :93.2% 2019-03-21 15:18:24 bytessent=171386341, bytesrecv=1109131110 2019-03-21 15:18:25 bytessent=171388527, bytesrecv=1109132600 2019-03-21 15:18:26 bytessent=171390590, bytesrecv=1109134008 12345678Copy the code

As can be seen from the time, the two tasks can be carried out simultaneously without waiting problems.

The essence of a Timer is to execute tasks in a threaded manner and destroy them after each execution, so there is no need to worry about resources.

Scheduling module: schedule

Schedule is a third-party lightweight task scheduling module that can be customized by second, minute, hour, date or event execution time.

Installation method:

pip install schedule

Let’s look at an example:

import datetime import schedule import time def func(): now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:%M:%S') print('do func time :',ts) def func2(): Now = datetime.datetime.now() ts = now. Strftime ('%Y-%m-%d %H:% m :%S') print('do func2 time: ',ts) def taskList (): Schedule.every (1).seconds.do(func) # Create a task that is executed every 2 seconds Schedule.every (2).seconds. Do (func2) # execute 10S for I in range(10): schedule.run_pending() time.sleep(1) tasklist() 1234567891011121314151617181920212223Copy the code

Execution Result:

Do func time: 2019-03-22 08:51:38 do func2 time: 2019-03-22 08:51:39 do func time: 2019-03-22 08:51:39 do func time: Do func2 time: 2019-03-22 08:51:41 do func time: 2019-03-22 08:51:41 do func time: 2019-03-22 08:51:41 do func time: 2019-03-22 08:51:41 do func time: 2019-03-22 08:51:41 do func time: Do func2 time: 2019-03-22 08:51:43 do func time: 2019-03-22 08:51:43 do func time: 2019-03-22 08:51:43 do func time: 2019-03-22 08:51:43 do func time: 2019-03-22 08:51:43 do func time: Do func2 time: 2019-03-22 08:51:45 do func time: 2019-03-22 08:51:45 do func time: The 12345678910111213 08:51:46 2019-03-22Copy the code

Analysis of execution process:

1> Because it is executed under JUPyter, the schedule task will be cleared first; 2> Add tasks in schedule by time interval; 3> Here add func at second intervals and func2 at two-second intervals; 4>schedule After a task is added, the task needs to be queried and executed. 5> In order to avoid occupying resources, query to point tasks every second, and then execute them in sequence;

Sleep (2); sleep(2);

Then just do the func work and print the result:

do func time : 2019-03-22 09:00:59
do func time : 2019-03-22 09:01:02
do func time : 2019-03-22 09:01:05
123
Copy the code

You can see the time interval is 3S, why not 1S?

This is a problem because this is executed sequentially, with the func sleeping for 2S and the circular task query sleeping for 1S.

We need to be aware of this blocking when we use this way to perform tasks.

Let’s take a look at the common usage of schedule module:

Create Job # schedule. Every (1), Seconds.do (func) query and execute schedule.every(1).seconds.do(func) # Add tasks to execute by minute schedule.every(1).minutes.do(func) # Add tasks to execute by day Schedule.every (1).days.do(func) # Add task every week 1, Schedule. Every ().monday. Do (func) # 1, Start at 1:15 schedule.every().monday.at("12:00").do(job) 123456789101112Copy the code

This approach has limitations: if the work task is very time-consuming, it will affect the execution of other tasks. We can consider configuring this module to use a concurrency mechanism.

The task framework APScheduler

APScheduler is Python’s scheduled task framework for executing periodic or scheduled tasks,

Scheduled tasks can be based on date, time interval, and the crontab type of scheduled tasks on Linux.

The framework can not only add and delete scheduled tasks, but also store tasks in the database to realize the persistence of tasks, which is very convenient to use.

PIP install apscheduler

Apscheduler components and a brief description:

1> Triggers (triggers) : 2> Job Stores: Used to store the scheduled jobs. By default, job Stores can store the jobs in memory and store them in MongoDB or Redis. 3> Executors: 4> Schedulers: Schedulers connect other parts together and provide interfaces for users to add, set and delete tasks. Qun: 784,758,214 In Python Exchange experience! Here’s a simple example:

import time from apscheduler.schedulers.blocking import BlockingScheduler def func(): now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:%M:%S') print('do func time :',ts) def func2(): Now = datetime.datetime.now() ts = now.strftime('%Y-%m-%d %H:% m :%S') print('do func2 time: ',ts) time.sleep(2) def dojob(): Create scheduler: Add_job (func, 'interval', seconds=2, Scheduler. Add_job (func2, 'interval', seconds=3, id='test_job2') scheduler.start() dojob() 123456789101112131415161718192021Copy the code

Output result:

Do func time: 2019-03-22 10:32:20 do func2 time: 2019-03-22 10:32:21 do func time: 2019-03-22 10:32:22 do func time: 2019-03-22 10:32:22 do func2 time: 2019-03-22 10:32:22 do func time: 2019-03-22 10:32:22 Do func2 time: 2019-03-22 10:32:24 do func time: 2019-03-22 10:32:26 123456Copy the code

In the output, you can see that the delay of the task does not affect the execution of other tasks.

The APScheduler framework provides rich interfaces to implement scheduled tasks. You can refer to the official documents to see how to use them.

Final choice:

Briefly summarize the above four kinds of fixed time tasks:

1: Cycle +sleep is suitable for short answer test,

2: Timer can realize scheduled tasks, but for fixed-point tasks, the current time point needs to be checked.

3: Schedule can be executed at a fixed point, but tasks need to be detected in the loop and are blocked;

4. APScheduler framework is more powerful, which can add fixed point and scheduled tasks directly;

After comprehensive consideration, we decided to use APScheduler framework, which is simple to implement. We only need to create tasks directly and add them to the scheduler.

Click 666 white piao to get or add QQ: 3271330538 exchange