Small knowledge, big challenge! This article is participating in the creation activity of “Essentials for Programmers”. Before, I was working on a Python project, which was executed on Windows. During the execution process, the memory kept increasing until it ran out and the program stopped. At first I thought it was business code, but later I used Python’s tools to check for memory leaks and found that it was a service program written in Python

Python memory mechanism

To understand why python memory leaks occur, it is important to understand python’s memory mechanism. Python is an interpreted, compiled, interactive, object-oriented scripting language where variables don’t have to be declared, they don’t have to be typed, and we don’t have to use Python like C, where we have to new an object and then release it manually. The Python interpreter gives you automatic reclamation, and it has a memory manager to handle the complex memory management tools, which is one of the things that makes Python easy to learn. There are two important parts of Python’s memory management mechanism: memory pooling and garbage collection

Memory pool

When creating a large number of objects that consume little memory, frequent calls to New /malloc result in a large amount of memory fragmentation, resulting in inefficient memory usage. The function of a memory pool is to apply for a certain number of memory blocks with the same size in advance to be used as spare memory. When new memory requirements are met, the memory pool is used to allocate new memory. In this way, memory fragmentation is reduced and efficiency is improved.

Python’s memory architecture has six layers -2 through 3, and python’s object manager is located at layers 1 through 3.

At layer L3, python built-in objects (ints,dict,list, STR, etc.) have separate private memory pools. The memory pools are not shared between objects, meaning that the memory freed by ints is not used by dict. At layer l2, when the requested memory size is less than 256kb, memory allocation is mainly implemented by python’s object allocator at layer l1. When the requested memory size is greater than 256kb, memory allocation is implemented by python’s native memory allocator. Essentially calling c library functions such as malloc/realloc to free memory, Python calls its destructor when an object’s reference count becomes zero. Calling the destructor does not necessarily mean that free will eventually be called to free up memory, so frequent requests to free memory will make Python less efficient. Therefore, the destructor also uses the memory mechanism, and the memory requested from the memory pool is returned to the memory pool to avoid frequent requisition and release actions.

Garbage collection mechanism

Reference count – Python has a trace variable called a reference counter that increments each reference count by one; When the object’s reference count is zero, the object is put into the garbage collection queue. There is a python function that knows how many times the current object has been referenced sys.getretcount(obj) returns +1, because calling this function also produces a temporary reference.