Directory | in the previous section (8.1) | in the next section (8.3 debugging)

8.2 log

This section provides a brief introduction to the Logging Module.

Logging modules

The Logging module is the Python standard library module for logging diagnostic information. The logging module is very large and has many complex functions. We will show a simple example to illustrate its usefulness.

Abnormal revisited

In this exercise, we create a parse() function:

# fileparse.py
def parse(f, types=None, names=None, delimiter=None) :
    records = []
    for line in f:
        line = line.strip()
        if not line: continue
        try:
            records.append(split(line,types,names,delimiter))
        except ValueError as e:
            print("Couldn't parse :", line)
            print("Reason :", e)
    return records
Copy the code

Look at the try-except statement. In the except block, what should we do?

Should a warning message be printed?

try:
    records.append(split(line,types,names,delimiter))
except ValueError as e:
    print("Couldn't parse :", line)
    print("Reason :", e)
Copy the code

Or ignore the warning messages?

try:
    records.append(split(line,types,names,delimiter))
except ValueError as e:
    pass
Copy the code

Neither approach is satisfactory, and in general we need both (user optional).

The use of logging

The Logging module solves this problem:

# fileparse.py
import logging
log = logging.getLogger(__name__)

def parse(f,types=None,names=None,delimiter=None) :.try:
        records.append(split(line,types,names,delimiter))
    except ValueError as e:
        log.warning("Couldn't parse : %s", line)
        log.debug("Reason : %s", e)
Copy the code

Modify the code so that the program can issue a warning message when it encounters a problem, or a special Logger object. The Logger object is created using logging.getLogger(__name__).

Log based

Create a Logger object.

log = logging.getLogger(name)   # name is a string
Copy the code

Issue a log message:

log.critical(message [, args])
log.error(message [, args])
log.warning(message [, args])
log.info(message [, args])
log.debug(message [, args])
Copy the code

Different methods represent different levels of severity.

All methods create formatted log messages. Args is used with the % operator to create messages.

logmsg = message % args # Written to the log
Copy the code

The log configuration

Configuration:

# main.py.if __name__ == '__main__':
    import logging
    logging.basicConfig(
        filename  = 'app.log'.# Log output file
        level     = logging.INFO,   # Output level
    )
Copy the code

Generally, log configuration is one-time at startup. This configuration is separate from logging calls.

instructions

Logging can be configured arbitrarily. You can tweak any aspect of the logging configuration: output files, levels, message formats, and so on without worrying about impacting code that uses the logging module.

practice

Exercise 8.2: Adding logs to the module

In fileparse.py, there is some error handling related to exceptions that are caused by incorrect input. As follows:

# fileparse.py
import csv

def parse_csv(lines, select=None, types=None, has_headers=True, delimiter=', ', silence_errors=False) :
    ''' Parse a CSV file into a list of records with type conversion. '''
    if select and not has_headers:
        raise RuntimeError('select requires column headers')

    rows = csv.reader(lines, delimiter=delimiter)

    # Read the file headers (if any)
    headers = next(rows) if has_headers else []

    # If specific columns have been selected, make indices for filtering and set output columns
    if select:
        indices = [ headers.index(colname) for colname in select ]
        headers = select

    records = []
    for rowno, row in enumerate(rows, 1) :if not row:     # Skip rows with no data
            continue

        # If specific column indices are selected, pick them out
        if select:
            row = [ row[index] for index in indices]

        # Apply type conversion to the row
        if types:
            try:
                row = [func(val) for func, val in zip(types, row)]
            except ValueError as e:
                if not silence_errors:
                    print(f"Row {rowno}: Couldn't convert {row}")
                    print(f"Row {rowno}: Reason {e}")
                continue

        # Make a dictionary or a tuple
        if headers:
            record = dict(zip(headers, row))
        else:
            record = tuple(row)
        records.append(record)

    return records
Copy the code

Notice the print statement that issues the diagnostic message. Replacing these print statements with logging operations is relatively simple. Modify the code as follows:

# fileparse.py
import csv
import logging
log = logging.getLogger(__name__)

def parse_csv(lines, select=None, types=None, has_headers=True, delimiter=', ', silence_errors=False) :
    ''' Parse a CSV file into a list of records with type conversion. '''
    if select and not has_headers:
        raise RuntimeError('select requires column headers')

    rows = csv.reader(lines, delimiter=delimiter)

    # Read the file headers (if any)
    headers = next(rows) if has_headers else []

    # If specific columns have been selected, make indices for filtering and set output columns
    if select:
        indices = [ headers.index(colname) for colname in select ]
        headers = select

    records = []
    for rowno, row in enumerate(rows, 1) :if not row:     # Skip rows with no data
            continue

        # If specific column indices are selected, pick them out
        if select:
            row = [ row[index] for index in indices]

        # Apply type conversion to the row
        if types:
            try:
                row = [func(val) for func, val in zip(types, row)]
            except ValueError as e:
                if not silence_errors:
                    log.warning("Row %d: Couldn't convert %s", rowno, row)
                    log.debug("Row %d: Reason %s", rowno, e)
                continue

        # Make a dictionary or a tuple
        if headers:
            record = dict(zip(headers, row))
        else:
            record = tuple(row)
        records.append(record)

    return records
Copy the code

When you’re done, try using this code on the wrong data:

>>> import report
>>> a = report.read_portfolio('Data/missing.csv')
Row 4: Bad row: ['MSFT'.' '.'51.23']
Row 7: Bad row: ['IBM'.' '.'70.44'] > > >Copy the code

If you do nothing, you will only get log messages above WARNING. The output looks like a simple print statement. However, if you configure the logging module, you will get other information about the logging level, the module, and so on. Please follow the following steps to view:

>>> import logging
>>> logging.basicConfig()
>>> a = report.read_portfolio('Data/missing.csv')
WARNING:fileparse:Row 4: Bad row: ['MSFT'.' '.'51.23']
WARNING:fileparse:Row 7: Bad row: ['IBM'.' '.'70.44'] > > >Copy the code

You will notice that there is no output from the log.debug() operation. To change the log level, follow these steps:

>>> logging.getLogger('fileparse').level = logging.DEBUG
>>> a = report.read_portfolio('Data/missing.csv')
WARNING:fileparse:Row 4: Bad row: ['MSFT', '', '51.23']
DEBUG:fileparse:Row 4: Reason: invalid literal for int() with base 10: ''
WARNING:fileparse:Row 7: Bad row: ['IBM', '', '70.44']
DEBUG:fileparse:Row 7: Reason: invalid literal for int() with base 10: ''
>>>
Copy the code

Only critical log messages are left and other log messages are disabled.

>>> logging.getLogger('fileparse').level=logging.CRITICAL
>>> a = report.read_portfolio('Data/missing.csv')
>>>
Copy the code

Exercise 8.3: Adding logs to the program

To add logs to your application, you need some mechanism to initialize logs in the main module. One way is to use code that looks like this:

# This file sets up basic configuration of the logging module.
# Change settings here to adjust logging output as needed.
import logging
logging.basicConfig(
    filename = 'app.log',            # Name of the log file (omit to use stderr)
    filemode = 'w',                  # File mode (use 'a' to append)
    level    = logging.WARNING,      # Logging level (DEBUG, INFO, WARNING, ERROR, or CRITICAL)
)
Copy the code

Again, you need to put the logging configuration code in the program startup step. For example, where do I put it in the report.py program?

Directory | in the previous section (8.1) | in the next section (8.3 debugging)

Note: The full translation can be found at github.com/codists/pra…