1. Introduction

In real projects, you are often exposed to a variety of configuration files that can enhance the maintainability of your project

Common accessory file processing methods, including: JSON, INI/config, YAML, XML, etc

In this article, we’ll talk about the configuration file summary of Python data processing

​2.JSON

Python has a built-in JSON module that makes it easy to manipulate JSON data

The four common methods are:

  • json.load(json_file)

    Parse the JSON file and convert it to the corresponding data type in Python

  • json.loads(json_string)

    Parse a string in JSON format, resulting in a dictionary in Python

  • json.dump(python_content,file_path)

    Write Python data, including dict and list, to a file

  • json.dumps(python_dict)

    Convert a dict in Python to a string in JSON format

Take this JSON configuration file as an example:

# config. Json {" mysql ": {" host" : "198.0.0.1", "port" : 3306, the "db" : "xh", "username" : "root", "password" : "123456", "desc": "Mysql configuration file "}}Copy the code

1. Read the configuration file

The configuration file can be read in either of the following ways:

Read the configuration file directly using json.load()

Alternatively, the contents of the configuration file are read first and then converted to Python data types using json.loads()

It should be noted that JSON configuration files at a complex level can be read using JsonPath; Jsonpath is similar to xpath in that it allows you to read data quickly through regular expressions

Import json def read_jSON_file (file_path): """ Read the JSON file :param file_path: :return: """ with open(file_path, 'r', encoding='utf-8') as file: # result = json.load(file) # result = json.loads(file) # result = json.loads(file) # result = json.loads(file) # result = json.loads(file) # result = json.loads(file) # result = json.loads(file) # result = json.loads(file Result ['mysql']['host'] port_mysql = result['mysql']['port'] db = result['mysql']['db'] print(' mysql') ', host_mysql, ", port_mysql, ", database :", db) return resultCopy the code

2. Save the configuration file

You can write a dictionary to a JSON file using the json.dump() method in JSON

Def write_content_to_jSON_file (output_file, content): """ Write to a JSON file: param output_file: :param content: :return: """ with open(output_file, 'w') as file: Ensure_ascii =False json.dump(content, file, ensure_ASCII =False) content_dict = {'mysql': {' host ':' 127.0.0.1 'and' port: 3306, 'db' : 'xh', 'username' : 'admin' and 'password', '123456', 'desc' : }} write_content_to_json_file('./output.json', content_dict)Copy the code

3. Modify the configuration file

To modify the configuration file, read the content from the configuration file, modify the content, and save the modified content in the configuration file

Def modify_json_file(): """ Change the JSON configuration file :return: "" "result = read_json_file ('. / config. Json ') # modified result [' mysql '] [' host '] = '198.0.0.1' write_content_to_json_file('./config.json', result)Copy the code

3.ini/config

Ini configuration files and Config configuration files are parsed in similar ways, but the file suffixes are different

Here we use the INI configuration file as an example

Ini [mysql] host = 139.199.1.1 username = root password = 123456 port = 3306Copy the code

The ini file consists of three parts: Section, Key, and Value

There are two common ways Python handles INI files, including:

  • Use the built-in configParser standard module

  • Use the Configobj third-party dependency library

Let’s start with the built-in ConfigParser module

3.1.1 Reading the Configuration File

Instantiate a ConfigParser parser object and read the INI configuration file using the read() method

From configParser import configParser # CFG = configParser () #Copy the code

Using the sections() function, you can get a list of all nodes

# sections() get all sections and return them as a list = cfg.sections() print(sections)Copy the code

To get all the keys under a node, use the options(section_name) function

# CFG. Options (section_name) keys = CFG. Options ('mysql') print(keys)Copy the code

The items(section_name) function allows you to retrieve all key and value pairs under a node

Items = cfg.items("mysql") print(items)Copy the code

Get (section_name,key_name) ¶ Get (section_name,key_name)

Host = cfg.get("mysql", "host") print(host)Copy the code

3.1.2 Writing to the Configuration File

Similar to reading a configuration file, you need to instantiate a ConfigParser parser object

First, add a node using the add_section(section_name) function

# add a node to cfg.add_section("redis")Copy the code

You can then use the set(section_name,key,value) function to add a key-value pair to a node

# to nodes, add the key value of CFG. Set (" redis ", "host", "127.0.0.1) CFG. Set (" redis", "port", "12345")Copy the code

Finally, write to the configuration file using the write() function

Write (open('./raw/output.ini', 'w'))Copy the code

3.1.3 Modifying a Configuration File

To modify the configuration file, read the configuration file, set(section_name,key,value), and write it to the file using the write() function

Def modify_ini_file(file_path): """ "" "CFG. Read (file_path) CFG. Set (" mysql ", "host", "139.199.11.11") # write CFG. Write (open (file_path, "w"))Copy the code

Next, let’s talk about the process of manipulating ini configuration files using Configobj

First install the Configobj dependency library

# pip3 install configobjCopy the code

3.2.1 Reading the Configuration File

Use the ConfigObj class to construct an object using the ini configuration file path directly as a parameter

Config = configobj (file_path, encoding='UTF8')Copy the code

Looking at the source code, you can see that ConfigObj is a subclass of the Section node, which in turn is a subclass of the Dict dictionary

Therefore, the node and Key value can be obtained directly from the Key name Key

Print (type(config['mysql'])) # <class 'configconfig.section '> print(type(config['mysql']) Print (config['mysql']) print(config['mysql'])Copy the code

3.2.2 Modifying a Configuration File

You only need to read the configuration file, modify the ConfigObj object directly, and then use the write() method to modify the configuration file

Def modify_ini_file(file_path): """ Config ['mysql']['host'] = '139.199.1.1' Del config['mysql']['db'] except Exception as e: print(' db') pass #Copy the code

3.2.3 Writing to the configuration file

To write a configuration file, you first need to instantiate a ConfigObj object and pass in the path to the file

Then, the node is set, and the key-value pair is set against the node

Finally, write to the configuration file by calling the write() method

Def write_to_ini_file(output): """ write to ini file :param output: :return: """ config = ConfigObj(output, Encoding = 'UTF8) config [' website'] = {} config [' website '] [' url '] = "www.baidu.com" config [' website '] [' name '] = "baidu" # save  config.write()Copy the code

4.YAML

Python operates YAML files in two common ways: Pyyaml and ruamel.yaml

Install dependencies using PIP

# install ruamel.yaml # install ruamel.yaml # install ruamel.yamlCopy the code

The following uses a simple YAML configuration file as an example and is illustrated in two ways

# # Fruits Fruits: Apple, Apple: name: Apple price: 1 address: guangdong # oranges - Orange: name: Orange price: 3 address: # banana-banana: name: Banana price: 2 address: HainanCopy the code

Let’s look at Pyyaml first

4.1.1 Reading the Configuration File

First, the configuration file is read and the data is loaded using yaml.safe_load(). The data type obtained is a dictionary

import yaml with open(file_path, "r") as file: Data type: dict result = yaml.safe_load(data) print(result)Copy the code

You can then retrieve the key values through the hierarchy of the YAML configuration file

Name = result['Fruits'][0]['Apple']['name'] price = result['Fruits'][0]['Apple']['price'] address = Result [' Fruits'] [0] [' Apple '] [' address '] print (" name: ", the name, "price:", price, ", address: the address)Copy the code

4.1.2 Writing to the Configuration File

Using the dump() method in YAML, you can write a dictionary to a YAML configuration file

Note that allow_Unicode =True is required to ensure that Chinese writes can be displayed properly

Def write_to_yaml_file(content, file_path): """ Write to yamL file :param content: :param file_path: :return: With open(file_path, 'w', encoding=' utF-8 ') as file: yaml.dump(content, file, default_flow_style=False, encoding='utf-8', Allow_unicode =True) # Define a dictionary content = {" websites": [{"baidu": {" url ": "www.baidu.com", "name ": "price": 100}}, {" alibaba ": {" url" : "www.taobao.com", "name" : "taobao", "price" : 200}}, {" tencent ": {" url" : "www.tencent.com", "name" : "Tencent", "price" : 300}},]} write_to_yaml_file (the content, the ". / raw/new. Yaml ")Copy the code

4.1.3 Modifying a Configuration File

To modify the type of the INI file, read the configuration file first, then modify the contents of the dictionary, and finally use the above writing method to modify the configuration file

Def modify_yaml_file(): """ """ Content = read_yaml_file('./raw/norm.yaml') print(content) # Dict content['Fruits'][0]['Apple']['price'] = 10086 # Write_to_yaml_file (content, './raw/output.yaml')Copy the code

Next, let’s talk about the process of manipulating YAML configuration files using Ruamel

Ruamel is a derivative of PyYAMl, adding RoundTrip mode to traditional PyYAMl to ensure that the read and write order of YAML configuration files is consistent

Therefore, it is similar to PyYAML in the way it reads, modifies, and writes

4.2.1 Reading the Configuration File

From ruamel import yaml def read_yaml_file(file_path): """ read yaml :param file_path: :return: """ with open(file_path, 'r', encoding='utf-8') as file: Data = file.read() ordereddict result = yaml.load(data, Loader=yaml.RoundTripLoader) name = result['Fruits'][0]['Apple']['name'] price = result['Fruits'][0]['Apple']['price'] Address = result [' Fruits'] [0] [' Apple '] [' address '] print (" name: ", the name, "price:", price, ", address: ", address) return resultCopy the code

4.2.2 Writing to the Configuration File

Def write_to_yaml_file(filepath, data): """ write to the yamL file :param filepath: :param data: :return: """ with open(filepath, 'w', encoding='utf-8') as file: yaml.dump(data, file, Dumper=yaml.RoundTripDumper, allow_unicode=True)Copy the code

4.2.3 Modifying a Configuration File

Def modify_yaml_file(): """ """ Content = read_yaml_file('./raw/norm.yaml') print(content) # Dict content['Fruits'][0]['Apple']['price'] = 10086 # Write back to a new YAML file write_to_YAMl_file ('./raw/output.yaml', content)Copy the code

5.XML

XML, as a markup language, is designed to store and transfer data, and many projects often use XML as a configuration file and data transfer type

Python’s built-in XML module makes it easy to process XML configuration files

Take the following configuration file as an example:

<? The XML version = "1.0" encoding = "utf-8"? > <dbconfig> <host>127.0.0.1</host> <port>3306</port> <dbname>test</dbname> <username>root</username> <password>4355</password> </mysql> </dbconfig>Copy the code

First, the configuration file is parsed with xml.dom.minidom.Parser (file_path), and the documentElement attribute is used to obtain the XML root node

Import xml.dom.minidom.parse("./raw. XML" dom.documentElementCopy the code

Next, you get a node using the getElementsByTagName(tag_name) method

# access mysql node node_mysql = root. GetElementsByTagName (' mysql ') [0]Copy the code

Finally, the childNodes attribute is used to traverse the childNodes of the Node and get the Node name and value

For node in node_mysql. ChildNodes: # 1: Element # 2: Attribute # 3: Text # print(node.nodetype) if node.nodetype == 1: print(node.nodeName, node.firstChild.data)Copy the code

6. The final

This is the end of all Python data buckets!

I have uploaded all the source code to the background, follow the public number “AirPython” reply “dball” can get all the source code

If you think the article is good, please like, share, leave a message, because this will be my strongest power to continue to output more high-quality articles!

Recommended reading

Data processing in Python

Sqlite: Data processing in Python

Data processing in Python (Redis)

Data processing in Python

Mongo: Data processing in Python