Common Python modules

Wechat public number search [program yuan xiaozhuang], pay attention to the halfway program yuan how to rely on Python development to support the family ~

preface

When developing Python, we often use different modules to help us achieve certain functions, so understanding the common ways of some common modules can help us speed up the development of programs.

Time module

Time is usually represented in Python in the following ways:

1. Timestamp: indicates the offset calculated in seconds from 00:00:00 on January 1, 1970. It is a floating point type and can be used to calculate the time interval.

>>> import time
>>> time.time()
1622362862.679771
Copy the code

2. The Format String is used to display the time.

>>> import time
>>> time.strftime('%Y-%m-%d %H:%M:%S %p')  # Y- year M- month D - day H- hour m- minute S- second p-AM/PM
'2021-05-30 16:22:44 PM'

# The above code can be abbreviated, using X to represent the minute and second
>>> time.strftime('%Y-%m-%d %X')
'the 2021-05-30 16:23:34'
Copy the code

3. Struct time

>>> print(time.localtime()) Struct_time for local time zone
>>> print(time.gmtime())    Struct_time for UTC time zone
Copy the code

Among the above three time formats, the only time format that computers can recognize is the timestamp, while the time that humans can understand is the formatted time string or structured time, so there is a conversion relationship between the three.

1. Timestamp –> Structured time –> formatted time

# timestamp ---- structured time
import time

time1 = time.time()  # timestamp
res = time.localtime(time1)  Convert the timestamp to structured time
print(res)

# Structured time ---- Formatted time
res = time.localtime()  # # Get structured time
time1 = time.strftime('%Y-%m-%d %X',res)  Convert structured time to formatted time
print(time1)
Copy the code

2. Format time –> Structure time –> timestamp

# Format time - Structure time
res = time.strftime('%Y-%m-%d %X',res)  Get the formatting time
print(res)

time1 = time.strptime(res,'%Y-%m-%d %X')  Convert formatted time to structured time
print(time1)

# Structuring time to timestamp
res = time.localtime()  Get structured time
time1 = time.mktime(res)
print(time1)
Copy the code

Datetime module

The Datetime module can be used to add or subtract time

import datetime

print(datetime.datetime.now())  Format output time
print(datetime.datetime.now() + datetime.timedelta(3)) # Current time +3 days
print(datetime.datetime.now() + datetime.timedelta(-3)) # Current time -3 days
print(datetime.datetime.now() + datetime.timedelta(hours=3)) Current time +3 hours
print(datetime.datetime.now() + datetime.timedelta(minutes=30)) # Current time +30 minutes
Copy the code

The random module

The random module can be used to generate random numbers, random verification codes and other functions.

import random
Get floating point numbers between 0 and 1 randomly
print(random.random())

Select an integer between 1 and 6 at random
print(random.randint(1.6))

Select an integer between 1 and 6 at random
print(random.randrange(1.6))

Get one of the parameters randomly
print(random.choice((111.'aaa'.456[1234.'qq')))# select 1 as the value of choice
print(random.sample([111.'aaa'.'ccc'.'ddd'].1))
print(random.sample([1.3.5.7.89,].3))

Shuffle Shuffles the original list
list1 = [1.2.3.5]
random.shuffle(list1)
print(list1)

# Use the random module to generate captcha
def func(n=6) :
    Get 26 random uppercase English letters according to ASC code table
    al_A = chr(random.randint(65.90))
    Get 26 random lowercase English letters according to ASC code table
    al_a = chr(random.randint(97.122))
    Get a random number
    num = random.randint(0.9)
    Generate random captcha
    res = ' '
    for i in range(n):
        res += random.choice([al_a,al_A,str(num)])
    return res
Copy the code

OS module

OS module provides a very rich method to deal with files and directories, is a very large and operating system interaction module, here introduce some commonly used methods.

import os

os.listdir(path)  Get all subfolder names and file names under a folder
os.remove(path)  Delete the file in the specified path
os.rename("oldname"."newname")  Rename files/directories
os.path.dirname(__file__)  Get the current file folder
os.path.basename(__file__)  Get the file name of the current file
os.path.split(path)  # return split path into directories and file names
os.path.exists(path)  # return True if path exists; If path does not exist, return False
os.listdir('dirname')  # list all files and subdirectories under the specified directory, including hidden files, and print them as a list
os.mkdir('dirname')   Create a single level directory under the current file path
os.getcwd()  Get the current working directory, that is, the path of the current working Python script
os.path.abspath(path)  Return path normalized absolute path
os.path.join('a'.'b')  # concatenate file paths
Copy the code

Sys module

sys.version       Get the Python interpreter version information
sys.path          # return the module's search path, initialized with the value of the PYTHONPATH environment variable
sys.platform      Return the operating system platform name
Copy the code

Serialization and deserialization modules

The JSON module and the pickle module are used to serialize and deserialize data.

Serialization refers to converting an in-memory data type to content in a particular format that can be used for storage or transfer to other platforms (other programming languages). Deserialization refers to converting data from other platforms to a data format that Python recognizes.

Deserialization: data type in memory ---- deserialization ---- Specific format (JSON /pickle format) Deserialization: Data type in memory ---- Deserialization ---- Specific format (JSON /pickle format)Copy the code

Data serialization serves two purposes:

First, it can persist the data generated during the running of the program and write the data to disk.

Second, after serialization, serialized content can not only be written to disk, but also be transmitted to other machines through the network. If the receiver and receiver agree to use a serialized format, the limitation brought by platform/language differentiation can be broken, and cross-platform data interaction can be realized.

Serialization and deserialization can be done with json and pickle modules.

Json module

Json data format is all programming languages and general data type, if we want to transfer between different programming language objects, you must put the object serialization to a standard format, such as XML, but a better approach is serialized as json format, because the json format that is a string that can be read by all languages, It can also be easily stored to disk or transmitted over the network. However, JSON cannot be used for serialization or deserialization of language-specific data types, such as Python’s collection types.

Note the differences between json strings and Python syntax. For example, json strings do not support single quotes, but dictionaries do

import json

# serialization
res = json.dumps([1.2.3.4.True.False])
print('Serialized result',res)  # Serialize result [1, 2, 3, 4, true, false], 'true'/false-- string type
Serialized results can be stored in files: complex methods
with open('json.txt'.'w', encoding='utf-8') as f:
    f.write(res)

Easy way to write serialized results to file:
with open('json.txt'.'w', encoding='utf-8') as f:
    json.dump([1.2.3.4.True.False],f)
    
# deserialize
res1 = json.loads(res)
print('deserialization',res1,type(res1))
Deserialize data in files: complex approach
with open('json.txt'.'r', encoding='utf -8') as f:
    json_res = f.read()
    res = json.loads(json_res)
    print(res)

Easy way to deserialize
with open('json.txt'.'r', encoding='utf -8') as f:
    res = json.load(f)
    print(res,type(res))
Copy the code

The pickle module

The pickle module is used in the same way as the JSON module, except that it can only be used in Python. You can serialize python-specific data types.

import pickle
# serialization
res = pickle.dumps({1.2.3})
Collections are python specific and can be serialized using pickle
print(res)
# deserialize
res1 = pickle.loads(res)
print(res)
Copy the code

Hashlib module

Hash is an algorithm that computes a string of hash values after receiving the incoming content. The hash values have the following features:

I As long as the input content is the same, the hash value must be the same.

II cannot return the hash value to the content.

III No matter how large the content is, as long as the hash algorithm used remains unchanged, the hash value obtained will have a certain length.

Based on the above features, hash can be used to transfer and verify passwords. The encryption algorithm commonly used is MD5.

import hashlib

m = hashlib.md5()  # build a factory to receive data that needs to be encrypted
m.update('hello'.encode('utf-8'))  Receive encrypted data
m.update('world'.encode('utf-8'))  Can receive encrypted data multiple times
res = m.hexdigest()   # process the data 'helloWorld' that needs to be encrypted and get the encrypted result
print(res)  # fc5e038d38a57032085441e7fe7010b0

m1 = hashlib.md5('he'.encode('utf-8'))
m1.update('llo'.encode('utf-8'))
m1.update('w'.encode('utf-8'))
m1.update('orld'.encode('utf-8'))
res = m1.hexdigest()  # # Process the data 'helloWorld' that needs to be encrypted, and get the encrypted result
print(res)  # fc5e038d38a57032085441e7fe7010b0

Update a long piece of data multiple times, the same as a single update of a long piece of data.
Copy the code

Although the above encryption algorithm cannot be reversed, it can be reversed through the collision library. The reverse solution can be achieved by comparing the encrypted password with the known MD5 string, as shown in the following code:

cryptograph='aee949757a2e698417463d47acac93df'  # Known md5 encrypted string
import hashlib

# create a password field
passwds=[
    'asdfgh'.'asdfh66'.'asdfh6678'
]

dic={}
for p in passwds:
    res=hashlib.md5(p.encode('utf-8'))
    dic[p]=res.hexdigest()

# Simulate crash library to get password
for k,v in dic.items():
    if v == cryptograph:
        print('Database hit successfully, clear text password: %s' %k)
        break
Copy the code

Therefore, customized keys can be added to the encryption algorithm for encryption, commonly known as adding salt, adding salt can increase the cost of collision database.

import hashlib

m = hashlib.md5()

m.update('king'.encode('utf-8'))
m.update('hello'.encode('utf-8'))
m.update('Geiger'.encode('utf-8'))
m.update('world'.encode('utf-8'))
print(m.hexdigest()
Copy the code

Logging modules

Log logging module can be used to record the program runs, the log is a very important part in program development, online production environment or in a test environment can be log in to check whether the program is running normally, if the program is unusual, can through the log quickly locate the abnormal code location, so logging need as detailed as possible, Logging is possible in Python development with the logging module.

import logging

# Number size represents different levels
CRITICAL = 50 
ERROR = 40
WARNING = 30 
INFO = 20
DEBUG = 10
NOTSET = 0 


logging.debug('debug debug')
logging.info('news info')
logging.warning('warning warn')
logging.error('error error')
logging.critical('serious critical')

WARNING:root warn ERROR:root: ERROR CRITICAL:root: CRITICAL"
Copy the code

By default, logs of warning or higher levels (waring error critical) can be output to the terminal, logs should be recorded in a file for easy viewing, global configuration can be specified for the logging module through logging.basicconfig (), and logs can be saved to a file.

import logging

logging.basicConfig(filename='test.log'.format='%(asctime)s - %(name)s - %(levelname)s -%(module)s: %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S %p',
                    level=10Filename: creates a FiledHandler with the specified filename and prints it to the file. Datefmt: specifies the date and time format. Level: log level set by numberformatPossible formatting strings in the argument: %(name)s Logger name %(LevelNo)s Digital log level %(LevelName)s Text log level %(pathName)s Full pathname of the module that invokes the log output function, There may not be %(filename)s Name of the module that calls the log output function %(module)s Name of the module that calls the log output function %(funcName)s Name of the function that calls the log output function %(lineno)d Line of code where the statement that calls the log output function resides %(asctime)s Indicates the current time in the string format. The default format is"2003- 07-0816:49:45.896". After the comma is milliseconds %(thread)d the ID of the thread. There may be no %(threadName)s threadName. There may be no %(process)d process ID. There may be no %(message)s message output by the userCopy the code

The logging module uses Formatter, Handler, Logger, and Filter objects:

Logger: the object that generates logs.

Filter: indicates the object for filtering logs.

Handler: Receives logs and controls the printing to different places. FileHandler is used to print to a file. StreamHandler is used to print to a terminal.

Formatter object: You can customize different log format objects and bind them to different Handler objects to control the log format of different handlers.

With all this said, how do you use the logging module for logging in a Python project? In project development, just remember the following usage.

First, configure the log format in the project’s configuration file (settings.py) :

# settings.py

# log file
LOG_PATH = os.path.join(BASE_DIR,'log')
LOG_UER_PATH = os.path.join(LOG_PATH,'access.log')

Define three log output formats
standard_format = '% (asctime) s (threadName) - % s: % (thread) d - log name: % (name) s (filename) - % s: % d (lineno) -' \
                  '%(levelname)s - %(message)s'

simple_format = '[%(levelname)s][%(asctime)s][%(filename)s:%(lineno)d]%(message)s'

test_format = '%(asctime)s] %(message)s'

# log configuration dictionary
LOGGING_DIC = {
    'version': 1.'disable_existing_loggers': False.'formatters': {
        'standard': {
            'format': standard_format
        },
        'simple': {
            'format': simple_format
        },
        'test': {
            'format': test_format
        },
    },
    'filters': {},
    Handlers are the receivers of the logs, and different handlers output the logs to different locations
    'handlers': {
        Logs printed to the terminal
        'console': {
            'level': 'DEBUG'.'class': 'logging.StreamHandler'.# Print to screen
            'formatter': 'simple'
        },
        'default': {
            'level': 'DEBUG'.'class': 'logging.handlers.RotatingFileHandler'.Save to file
            # 'maxBytes': 1024*1024*5, # Log size 5M
            # 'maxBytes': 1000,
            # 'backupCount': 5,
            'filename': LOG_UER_PATH,  # os.path.join(os.path.dirname(os.path.dirname(__file__)),'log','a2.log')
            'encoding': 'utf-8'.'formatter': 'standard',},Print logs to files, collect info and above logs
        'other': {
            'level': 'DEBUG'.'class': 'logging.FileHandler'.Save to file
            'filename': LOG_UER_PATH, # os.path.join(os.path.dirname(os.path.dirname(__file__)),'log','a2.log')
            'encoding': 'utf-8'.'formatter': 'test',}},# loggers are producers of logs that are passed to handlers to control the output
    'loggers': {
        # logging.getLogger(__name__) gets the logger configuration. If we want logger objects with different logger names to share the same configuration, we can't define n keys in the loggers subdictionary. GetLogger (__name__). Different files have a different __name__. This ensures that logs are printed with different identifiers, but the key names in loggers are not found. The default configuration is key="
        ' ': {
            'handlers': ['default',].The log is written to the file and printed to the screen
            'level': 'DEBUG'.Handlers -> Handlers -> Handlers -> Handlers
            'propagate': False.# Default True, pass up (higher level loggers), usually set to False, otherwise a log will be passed up one layer}},}Copy the code

Then, use the logging module in other files to log:

# test.py

# logging is a package that needs to use config and getLogger under it
from logging import config
from logging import getLogger

Load log dictionary configuration
logging.config.dictConfig(settings.LOGGING_DIC)

Generate a log object
logger = logging.getLogger('test')

# Log
logger.info('Record a log')
Copy the code

The logging module looks very complex, but it works the same way as above. At project development time, you just need to copy it and use it.