The SYS module in Python is very basic and important. It mainly provides variables that the interpreter uses (or maintains) and functions that strongly interact with the interpreter.

This article will make frequent use of the module’s getSizeof () method, so I’ll give you a quick overview:

  • This method is used to get the size of an object in bytes.
  • It only counts the memory directly occupied, not the memory of the objects referenced within the object

Here’s an intuitive example:

import sys

a = [1.2]
b = [a, a]  # i.e. [[1, 2], [1, 2]]

# a and B both have only two elements, so the size of the direct occupation is equal
sys.getsizeof(a) # Result: 80
sys.getsizeof(b) # Result: 80
Copy the code

The above example illustrates one thing: a statically created list containing only two elements takes up 80 bytes of memory on its own, regardless of the object to which its elements refer.

Now, with this measurement tool in hand, let’s explore what Python’s built-in objects are hiding.

1. Empty objects are not empty!

For some of the empty objects we are familiar with, such as empty strings, empty lists, empty dictionaries, etc., I wonder if you have ever wondered, whether you have ever thought about these questions: does an empty object take up no memory? If it takes up memory, how much? Why is it distributed this way?

Let’s look at the size of empty objects for some basic data structures:

import sys
sys.getsizeof("")      # 49
sys.getsizeof([])      # 64
sys.getsizeof(())      # 48
sys.getsizeof(set())   # 224
sys.getsizeof(dict())  # 240

# as a reference:
sys.getsizeof(1)       # 28
sys.getsizeof(True)    # 28
Copy the code

As you can see, these objects are empty, but they are not “empty” in memory allocation, and they are allocated quite large (remember these numbers, we will check later).

In order: base number < empty tuple < empty string < empty list < empty set < empty dictionary.

How to explain this little secret?

Because these empty objects are containers, we can understand them abstractly: some of their memory is used to create the skeleton of the container, to record information about the container (reference counts, usage information, and so on), and some of their memory is pre-allocated.

2, memory expansion is not uniform!

Empty objects are not empty, in part because the Python interpreter preallocates some initial space for them. Each new element uses the existing memory without exceeding the original memory, thus avoiding the need to apply for new memory.

So, if the initial memory is allocated, how is the new memory allocated?

import sys
letters = "abcdefghijklmnopqrstuvwxyz"

a = []
for i in letters:
    a.append(i)
    print(f'{len(a)}, sys.getsizeof(a) = {sys.getsizeof(a)}')
    
b = set()
for j in letters:
    b.add(j)
    print(f'{len(b)}, sys.getsizeof(b) = {sys.getsizeof(b)}')

c = dict()
for k in letters:
    c[k] = k
    print(f'{len(c)}, sys.getsizeof(c) = {sys.getsizeof(c)}')
Copy the code

Add 26 elements to each of the three mutable objects and see what happens:

This shows the secret of mutable objects when they are augmented:

  • Overallocation mechanism: New memory is allocated not on demand, but more, so that when a few more elements are added, there is no need to immediately allocate new memory
  • Non-uniform allocation mechanism: The three types of objects apply for new memory at different frequencies, but the over-allocated memory of the same type of objects is not uniform, but gradually expanded

A list is not a list!

The above variable objects have a similar allocation mechanism when expanding, and the effect can be clearly seen in dynamic expansion.

Does this work for statically created objects? Is it different from the dynamic expansion ratio?

Let’s start with sets and dictionaries:

Statically create an object
set_1 = {1.2.3.4}
set_2 = {1.2.3.4.5}
dict_1 = {'a':1.'b':2.'c':3.'d':4.'e':5}
dict_2 = {'a':1.'b':2.'c':3.'d':4.'e':5.'f':6}

sys.getsizeof(set_1)  # 224
sys.getsizeof(set_2)  # 736
sys.getsizeof(dict_1) # 240
sys.getsizeof(dict_2) # 368
Copy the code

Looking at this result and comparing the screenshot from the previous section, you can see that statically created collections/dictionaries take up exactly the same amount of memory as dynamically expanded collections with the same number of elements.

Does this apply to list objects? Take a look:

list_1 = ['a'.'b']
list_2 = ['a'.'b'.'c']
list_3 = ['a'.'b'.'c'.'d']
list_4 = ['a'.'b'.'c'.'d'.'e']

sys.getsizeof(list_1)  # 80
sys.getsizeof(list_2)  # 88
sys.getsizeof(list_3)  # 96
sys.getsizeof(list_4)  # 104
Copy the code

The screenshot from the previous section shows that the list is 96 bytes for each of the first four elements and 128 bytes for each of the five elements, which clearly contradicts this.

So, the secret is clear: with the same number of elements, statically created lists can take up less memory than dynamically expanded lists!

In other words, the two lists look the same but are actually different! A list is not a list!

Eliminating elements does not free memory!

As mentioned earlier, when scaling mutable objects, it is possible to request new memory.

So, if you shrink the mutable object in reverse, after subtracting a few elements, will the newly allocated memory be automatically reclaimed?

import sys
a = [1.2.3.4]
sys.getsizeof(a) # Initial value: 96
a.append(5)      [1, 2, 3, 4, 5]
sys.getsizeof(a) # 128
a.pop()          [1, 2, 3, 4]
sys.getsizeof(a) # reduced: 128
Copy the code

As the code shows, the list expands and shrinks, but the memory space is not automatically freed up. The same goes for other mutable objects.

Here’s the Python secret: It’s easy for skinny people to get fat and lose weight, but they can’t lose weight

Empty dictionary is not equal to empty dictionary!

Using the pop() method only shrinks the elements in the mutable object, but it does not free up allocated memory.

There is also the clear() method, which clears all elements of the mutable object. Let’s try it:

import sys
a = [1.2.3]
b = {1.2.3}
c = {'a':1.'b':2.'c':3}

sys.getsizeof(a) # 88
sys.getsizeof(b) # 224
sys.getsizeof(c) # 240

a.clear()        # empty: []
b.clear()        Set ()
c.clear()        Empty: {}, dict()
Copy the code

By calling the clear() method, we get several empty objects.

Their memory size was checked in the first section. (Previously said the examination, please write back to read)

However, if you check again at this point, you may be surprised to find that the empty objects are not exactly the same size!

# continue the previous emptying operation:
sys.getsizeof(a) # 64
sys.getsizeof(b) # 224
sys.getsizeof(c) # 72
Copy the code

Empty lists and empty tuples remain the same size, but the empty dictionary (72) is much smaller than the previous empty dictionary (240)!

In other words, lists and tuples, after being emptied, return to the original point of view, whereas dictionaries throw up what they ate and lose what they had.

The dictionary is a pretty deep secret. To be honest, I have just learned it, and I can’t figure it out…

These are some of the secrets of allocating memory in Python. Do you feel enlightened after reading them?

How many have you figured out? How many new mysteries have you created? Welcome to leave a message to exchange oh ~

For those little secrets that haven’t been fully explained, we’ll reveal them later…

About the author: Pea Flower cat, born in Guangdong province, graduated from Wuhan University, is now a programmer in Su Piao, some geek thinking, some humanistic feelings, some temperature, and some attitude. Public id: Python Cat (python_cat)