Problem description

  • Gunicorn + Flask deplores the Python service. When worker > 2, print logs to stdout are out of order

Problem analysis

  • In a multi-process deployment environment, print is insecure and a global process lock needs to be added

Problem processing

  • Multiprocessing. Lock and threading.Lock are common in Python. Multiprocess locks Lock child resources, while multithreading locks child resources.
  • Gunicorn + Flask architecture, Gunicorn will start multiple worker sub-processes, each of which can be regarded as an independent Flask process. Now the Flask application needs to apply lock resources in all worker processes, and lock resources need to be mutually exclusive in other workers
  • General solution: DB record lock, REDis record lock, file lock, file lock is selected here, the specific code is as follows:
# -*- coding: utf-8 -*- import os import time import fcntl class Lock(object): @staticmethod def get_file_lock(): return FileLock() class FileLock(object): def __init__(self): lock_file = 'HYDRA_FILE_LOCK' lock_dir = '/tmp' self.file = '%s%s%s' % (lock_dir, os.sep, lock_file) self._fn = None self.release() def acquire(self): try: self._fn = open(self.file, 'w') fcntl.flock(self._fn, fcntl.LOCK_EX) except Exception: pass def release(self): try: if self._fn: fcntl.flock(self._fn, fcntl.LOCK_UN) self._fn.close() except Exception: pass if __name__ == "__main__": import time N = 50 lock = Lock.get_file_lock() start_time = time.time() for i in range(N): a = 2**9999 t1 = (time.time() - start_time) / float(N) start_time = time.time() for i in range(N): print(a) t2 = (time.time() - start_time) / float(N) start_time = time.time() for i in range(N): Lock.acquire () print(a) Lock.release () t3 = (time.time() -start_time)/float(N) print(" % s "% STR (t1)) print (" print directly time-consuming: % s" % STR (t2) print (" locked print time-consuming: % s "% STR (t3))Copy the code

summary

  • The file locking module is used in Linux to ensure atomic operation of the locking process
  • Locking is bound to bring performance loss, please evaluate after use

That’s all!