I’d like to share with you some of the problems that my team and I experienced on a project. In this project, we had to store and process a fairly large dynamic list. Testers complain of insufficient memory during testing. Here’s a simple way to solve this problem by adding a line of code.

The result of the picture



So let me explain how it works.

First, let’s consider a simple “learning” example, creating a Dataltem class that is a person’s personal information, such as name, age, address, etc.

Copy the code

Beginner’s question: How do I know how much memory more than one of these objects takes up?

First, let’s try to solve this:

Copy the code

We got 56bytes, which seems to take up very little memory, which is pretty satisfying. So, let’s try another example of an object with more data:

Copy the code

The answer is still 56bytes, and at this point, it seems we realize something is wrong? Not everything is as it first seems.

● Intuition doesn’t let us down. It’s not that simple. Python is a very flexible language with dynamic typing, and it stores a lot of additional data for its work. They take up a lot on their own.

For example, sys.getsizeof(“”) returns 33bytes, which is an empty line of up to 33bytes! And sys.getsizeof(1) returns 24bytes, a whole number that takes up 24bytes (I would like to consult C programmers, away from the screen, do not want to read further, lest you lose faith in aesthetics). For more complex elements such as dictionaries, sys.getsizeof(.()) returns 272 bytes, which is for empty dictionaries, I won’t continue, I hope the principle is clear and RAM manufacturers need to sell their chips.

But let’s go back to our DataItem class and the initial beginner’s doubts.

How much memory does this class take up?

First, we print the entire contents of the class in lowercase:

Copy the code

This function will show you the hidden “behind the scenes” that makes all Python functions (types, inheritance, and everything else) work.

The results were impressive:



How much memory does all this take up?

Here’s a function that computes the actual sizeof an object by recursively calling getSizeof.

Copy the code

Let’s try it:

Copy the code

We got 460bytes and 484bytes, which seems to be true.

Using this function, you can perform a series of experiments. For example, I want to know how much space the data will take up if the DataItem structure is placed in a list. The get_size ([d1]) function returns 532bytes, which obviously has the same overhead as the 460+ mentioned above. But get_size ([d1, d2]) returns 863bytes, less than the above 460 + 484. Get_size ([d1, d2, d1]) results are even more interesting — we get 871 bytes, just a little more, which means Python is smart enough not to allocate memory for the same object again.

Now, let’s look at the second part of the problem.

Is it possible to reduce memory overhead?

Yes, you can. Python is an interpreter, and we can extend our class at any time, for example, by adding a new field:

Copy the code

Great, but what if we don’t need this feature? We can force the interpreter to specify the list object of the class using the __slots__ command:

Copy the code

More information can be found in the documentation (RTFM), which says “__ dict__ and __weakref__”. Using __dict__ saves a lot of space “.

We confirm: yes, indeed, get_size (d1) returns… 64 bytes instead of 460 bytes, which is seven times less. Additionally, objects are created 20% faster (see the first screen capture of this article).

Alas, the real use of such large memory gains is not due to other overhead. Create an array of 100,000 by simply adding elements, and look at the memory consumption:

Copy the code

Instead of using __slots__ to take up 16.8MB of memory, we use it to take up 6.9MB. This action is certainly not the best, but it does make the least amount of code change. (Not 7 times of course, but it’s Not bad at all, considering that the code change was minimal.)

Present shortcomings. Enabling __slots__ disallows creation of all elements, including __dict__, which means, for example, that the following code to convert a structure to JSON will not run:

Copy the code

This problem is easy to fix, and it is enough to generate dict programming in a way that loops through all elements:

Copy the code

It is also not possible to dynamically add new class variables to the class, but in this case, this is not required.

The last test of the day. What’s interesting is how much memory the entire program requires. Add an infinite loop to the program so that it doesn’t end and look at memory consumption in Windows Task Manager.

No __slots__ :



6.9MB becomes 27Mb… Boy, after all, we saved memory, and 27Mb instead of 70 is not a bad example for adding a line of code

Note: The TraceMelc debug library uses a lot of additional memory. Obviously, she adds extra elements to each object she creates. If you turn it off, the total memory consumption will be much lower, and the screenshot shows two options:



What if you want to save more memory?

This can be done using the Numpy library, which allows you to create structures in C style, but in my case it requires a deeper refinement of the code, and the first approach is sufficient.

Oddly enough, Habre has never analyzed the use of __slots__ in detail, and I hope this article will fill that gap.

conclusion

This article appears to be an anti-Python AD, but it is not. Python is very reliable (you have to work very hard to “degrade” Python programs), and it’s an easy language to read and code. These advantages outweigh the disadvantages in many cases, but if you need maximum performance and efficiency, you can use a library like numpy, written in C++, that works with data quickly and efficiently.


The original article was published on November 29, 2018

Alex Maison

This article is from the cloud community partner “Machine Learning Algorithms and Python Learning”. For more information, check out “Machine Learning Algorithms and Python Learning”.