This is the 26th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

Guidelines on pit

This is a troubleshooting guide, which is derived from the fact that bloggers use APScheduler frequently recently. Some common problems are summarized herein, hoping to be helpful to everyone.

Error 1: Can’t Connect to MySQL

If you are using SQLAlchemy as the job storage medium, this can cause problems if the parameters are not configured properly.

  • Problem description

    When it comes to job data reads/updates, we call the Session of SQLAlchemy to do so. Can’t Connect to MySQL or Lose Connection.

  • Problem analysis

    MySQL has a validity period for each database connection, the default should be 8 hours, but this can be configured. SQLAlchemy’s connections are pooled and have a default recycle time.

    If SQLAlchemy created a connection with 2 hours of reclaim time, but MySQL configured a connection with 1 hour of failure time, then there is a problem:

    SQLAlchemy thinks the connection is still valid, but MySQL thinks it’s invalid, so it doesn’t recognize it when it connects to MySQL. The above errors will occur.

  • The solution

    Configure the pool_recycle parameter in jobStore. Ensure that this parameter is smaller than the connection timeout of the database.

    For example, if I set it to 25 minutes, then even if the database timeout is only 30 minutes, I can work fine.

Connection reclaim time is 1500 seconds
job_store = {
        'default': SQLAlchemyJobStore(url="Database JDBC connection address", engine_options={"pool_recycle": 1500})}Copy the code

Problem 2 The job was deleted inexplicably

I have to say, it’s a sick setup, and the problem is that your service runs around and your scheduled tasks disappear. This is very, very painful.

  • Problem description

    In my case, APScheduler automatically removes jobs that cannot be deserialized from the database to memory.

  • Problem analysis

    Pay attention to the specific log. After my investigation, I found that the serialization method used by APScheduler is pickle. Compared with JSON, it is a serialization class library that we are not familiar with but have heard of.

    But there is a serious problem with this library. For example, my python3.7 has a pickle.DEFAULT_PROTOCOL parameter in it, which was 3 at 3.7 and changed to 4 after Python3.9.

    The pickling.DEFAULT_PROTOCOL parameter is used by default in APScheduler for this parameter.

    Imagine that if Python3.9 is used locally but the server is 3.7, you might add the job with DEFAULT_PROTOCOL=4, but when the server deserializes the job, it gets 3, so it cannot deserialize the job, and finally drops the job.

  • The solution

    Specify the serialization type, preferably the Python version of the machine you are deploying, which can currently be 2 or 3.

job_store = {
        'default': SQLAlchemyJobStore(url="Database connection address", pickle_protocol=3, engine_options={"pool_recycle": 1500})
Copy the code

Question 3. Repeat execution using Gunicorn or Uvicorn

Take a look at my last article, which uses socket/ distributed locks.

Test Platform Series (82) addresses repetitive execution of the APScheduler

Problem 4 Distributed deployment

As mentioned above, both distributed deployment and multi-worker mode. APScheduler support is not very friendly.

Because it is designed to give a single Python project a timed task scenario, it does not consider distributed issues such as clustering. It works fine within a single Python service. Let’s look at an example:

  • The Web service is deployed on server A

  • Web services are deployed on server B

    The two services are identical. The reason why two machines are deployed is for load balancing. So if I call the new job method and the request goes to server A, what will happen?

  1. Server A receives data, associates with the local FUNC, and loads the job to the memory
  2. Describes memory persistence to the database after loading

But would you say that server B’s Jobstore has this job? Of course not, because it doesn’t have any concept of clustering. There is no mechanism for server A to receive A command and send it to server B.

The most intuitive result is that you start N workers in Gunicorn/Uvicorn and call the Add interface to add jobs, and you won’t notice the problem of job duplication.

Once you restart the service and all 8 workers load jobs from the database into memory, you will soon encounter repetitive tasks.

This article is limited to my superficial understanding of APScheduler, and I hope to correct any mistakes.