You can easily manage and access folders, files, and paths using the Python OS module.

1 introduction

In recent development, I have often needed to read, iterate, modify files, etc. To do this quickly and easily, I chose Python. With Python’s standard built-in OS modules, you can do what you want with just a few lines of code. After the use of OS, this paper summarizes the common operations of OS module, mainly divided into the following divisions:

  • Folder operation: namely folder creation, modification (rename/move), query (view, traverse), delete and so on.
  • File operations: create, modify, read, and delete files.
  • (Folder/file) Path operation: Path operation of a folder or file, such as absolute path, file name and path splitting, and extension splitting

This article deals with the use of common OS functions, mainly using Python interactive mode for code illustration. The OS module has been imported by default as follows:

import os
Copy the code

2 Folder Operations

Use the local E://pythontest directory as the demo directory. The current files in this directory are as follows:

├ ─ sci-1.txt │ ├ ─ sci-1.txtCopy the code

Test and test-1 are folders, test. TXT and test-1 are files.

2.1 Query Operations

If you are familiar with Linux, you should be familiar with ls/PWD/CD. Python also has corresponding methods, including:

  • Listdir: list of files and directories
  • Getcwd: obtains the current directory
  • Chdir: changes the directory
  • Stat: indicates basic information about files and directories
  • Walk: Recursively traverses the directory
>>> os.chdir("E://pythontest")  # change directory
>>> os.getcwd()                 Get the current directory
'E:\\pythontest'
>>> os.listdir("test")          # List of files and directories, relative path
['test-1'.'test.txt']          
>>> os.listdir("E://pythontest/test")  List of files and directories, absolute path
['test-1'.'test.txt']
>>> os.stat("test")             Get directory information
os.stat_result(st_mode=16895, st_ino=4503599627377599, st_dev=266147611, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1590833033, st_mtime=1590832647, st_ctime=1590832207)
>>> os.stat("test/test.txt")    Get file information
os.stat_result(st_mode=33206, st_ino=2251799813692354, st_dev=266147611, st_nlink=1, st_uid=0, st_gid=0, st_size=4, st_atime=1590832653, st_mtime=1590832609, st_ctime=1590832598)
Copy the code

The stat function returns basic information about a file or directory, as follows:

  • St_mode: inode protection mode
  • St_ino: indicates the number of the inode node.
  • St_dev: device where the inode resides.
  • St_nlink: indicates the number of links in inodes.
  • St_uid: user ID of the owner.
  • St_gid: indicates the group ID of the owner.
  • St_size: indicates the size of a common file in bytes
  • St_atime: time of last access.
  • St_mtime: indicates the time of the last modification.
  • St_ctime: creation time.

In daily use, we usually use st_size, st_ctime and st_mtime to obtain the file size, creation time and modification time. In addition, we see that the output time is seconds, which is mentioned here for the date conversion processing.

(1) Seconds to date time format character string

>>> import time                              # Introduce the time module
>>> timestruct = time.localtime(1590803070)  # Convert to a time structure
>>> print(timestruct)
time.struct_time(tm_year=2020, tm_mon=5, tm_mday=30, tm_hour=9, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=151, tm_isdst=0)
>>> time.strftime("%Y-%m-%d %H:%M:%S",timestruct)   # format time
'the 2020-05-30 09:44:30'
Copy the code

(2) Format Date time string number of seconds

>>> import datetime               # Introduce datetime module
>>> timeobject = datetime.datetime.strptime("The 2020-05-23 10:00:00"."%Y-%m-%d %H:%M:%S") Parse the time string as a time object
>>> timeseconds=time.mktime(timeobject.timetuple())  Get the number of seconds
>>> print(int(timeseconds))       # convert to int display
1590199200
Copy the code

  • Traversal operation

    The walk function performs recursive traversal of the directory, returning root, dirs, files, corresponding to the current traversal directory, subdirectories and files in this directory respectively.

>>> data = os.walk("test")               Pass through the test directory
>>> for root,dirs,files in data:         # Recursive traversal and output
.   print("root:%s" % root)
.   for dir in dirs:
.      print(os.path.join(root,dir))
.   for file in files:
.      print(os.path.join(root,file))
...
root:test
test\test- 1
test\test2 -
test\test.txt
root:test\test- 1
test\test- 1\test1.txt
root:test\test2 -
test\test2 -\test- 2.txt
Copy the code

2.2 Creating a Vm

  • Mkdir: creates a directory. If the parent directory does not exist, the creation fails

  • Makedirs: Create multiple directories. If the parent directory does not exist in the directory path, it will be created automatically

>>> os.mkdir("test")
>>> os.mkdir("test1/test1-1")          The parent directory does not exist
Traceback (most recent call last):
  File "<stdin>", line 1.in <module>
FileNotFoundError: [WinError 3] The system could not find the specified path. :'test1/test1-1'
>>> os.makedirs("test1/test1-1")       The parent directory does not exist
>>> os.listdir("test1")
['test1-1']
Copy the code

2.3 Deleting a vm

  • Rmdir: Delete a single empty directory. If the directory is not empty, an error is reported
  • Removedirs: Deletes an empty recursive multi-level directory by path. If the directory is not empty, an error message is displayed
>>> os.rmdir("test1")                         If the directory is not empty, an error is reported
Traceback (most recent call last):
  File "<stdin>", line 1.in <module>
OSError: [WinError 145] The directory is not empty. :'test1'
>>> os.rmdir("test1/test1-1")
>>> os.removedirs("test1/test1-1")            Delete multi-level empty directories
>>> os.listdir(".")
['test']
Copy the code

Due to the limitation of deleting empty directories, it is more possible to delete non-empty directories and their files using the RMtree function in the Shutil module.

2.4 Modification Operations

  • Rename: You can rename a directory or file. You can change the path of a file or directory. If the target file directory does not exist, an error message is displayed.
  • Renames: renames a directory or file. If the destination file directory does not exist, it is automatically created
>>> os.makedirs("test1/test1-1") >>> os.rename("test1/test1-1","test1/test1-2") # test1-1 rename to test1-2 >>> Listdir ("test1") ['test1-2'] >>> os.rename("test1/test1-2","test2/test2-2") # Traceback (most recent call last): File "<stdin>", line 1, in <module> FileNotFoundError: [WinError 3] The system could not find the specified path. : 'test1/test1-2' -> 'test2/test2-2' >>> os.renames("test1/test1-2","test2/test2-2") # renames can automatically create non-existent directories >>> os.listdir("test2") ['test2-2']Copy the code

If the destination path file already exists, both os.rename() and os.renames() report an error: FileExistsError: [WinError 183] Cannot be created while the file already exists.

3 File Operations

3.1 Query Operations

  • Open /read/close: files are read
  • Stat: File information. For details, see the stat description in the previous folder
>>> f = os.open("test/test.txt", os.O_RDWR|os.O_CREAT)  # Open file
>>> str_bytes = os.read(f,100)                          Read 100 bytes
>>> str = bytes.decode(str_bytes)                       # byte to string
>>> print(str)
test write data
>>> os.close(f)                                         # close file
Copy the code

Note that the open/read/close operation needs to be operated together, where the open operation needs to specify the mode. The above operation opens the file in read/ write mode, and creates the file if the file does not exist. The modes are as follows:

Flags, this parameter can be the following options, multiple use “|” separated by:

  • O_RDONLY: open in read-only mode
  • O_WRONLY: open in write only mode
  • O_RDWR: open in read/write mode
  • O_NONBLOCK: does not block when opened
  • O_APPEND: Opens in append mode
  • O_CREAT: Creates and opens a new file
  • O_TRUNC: Open a file and truncate it to zero length (must have write permission)
  • O_EXCL: Returns an error if the specified file exists
  • O_SHLOCK: automatically obtains the shared lock
  • O_EXLOCK: Automatically obtains an independent lock
  • O_DIRECT: eliminates or reduces caching effects
  • O_FSYNC: Synchronizes data writes
  • O_NOFOLLOW: Do not track soft links

3.2 Creating a Vm

As mentioned earlier, use open to specify the schema and create the file if it does not exist. Somewhat similar to touch in Linux operations.

>>> f = os.open("test/test.txt", os.O_RDWR|os.O_CREAT)   If the file does not exist, create it
>>> os.close(f)
Copy the code

3.3 Modifying Operations

  • Open /write/close: writes file content
  • Rename, renames: The same as the rename and move operations described earlier.
>>> f = os.open("test/test.txt", os.O_RDWR|os.O_CREAT)     # Open file
>>> os.write(f,b"test write data")                         # write content
15
>>> os.close(f)                                   # close file
Copy the code

3.4 delete

  • Remove: Delete files. Note: Do not delete directories (use rmdir/removedirs).
>>> os.remove("test/test-1")       Failed to delete the directory
Traceback (most recent call last):
  File "<stdin>", line 1.in <module>
FileNotFoundError: [WinError 2] The specified file cannot be found. :'test/test1'
>>> os.remove("test/test.txt")     # delete file
>>> os.listdir("test")
['test-1']
Copy the code

4 Path Operations

In the process of using files or directories, files and directory paths need to be processed frequently. Therefore, the OS has a submodule path to handle path operations. The main operations are as follows:

  • Abspath: Returns the absolute path
>>> os.path.abspath("test")
'E:\\pythontest\\test'
Copy the code
  • Exists: Determines whether a file or directory exists
>>> os.path.exists("test")
True
>>> os.path.exists("test/test.txt")
False
>>> os.path.exists("test/test-1/test-1.txt")
True
Copy the code
  • Isfile /isdir: checks whether it is a file or directory
>>> os.path.isdir("test")
True
>>> os.path.isfile("test/test-1/test-1.txt")
True
Copy the code
  • Basename/dirName: Obtains the tail and head of a path. It’s essentially the last one in the path/Is the separator and is divided into head and tail. Tail is the content returned by basename and head is the content returned by dirName. Often used to obtain file names, directory names and other operations
>>> os.path.basename("test/test-1/test-1.txt")   # filename
'test-1.txt'
>>> os.path.basename("test/test-1/")     # empty content
' '
>>> os.path.basename("test/test-1")      # directory name
'test-1'
>>> os.path.dirname("test/test-1/test-1.txt")   # Directory where the file resides
'test/test-1'
>>> os.path.dirname("test/test-1/")   # directory path
'test/test-1'
>>> os.path.dirname("test/test-1")   Parent directory path
'test'
Copy the code
  • Join: Synthesize path, that is, join two parameters using the system path separator to form a complete path.
>>> os.path.join("test"."test-1")   # Connect two directories
'test\\test-1'
>>> os.path.join("test\\test-1"."test-1.txt")   # connect directory and file name
'test\\test-1\\test-1.txt'
Copy the code
  • Split: Splits the file name and folder, that is, splits the path into head and tail with the last slash “/” as the separator, and returns a (head, tail) tuple.
>>> os.path.split("test/test-1")     # split directory
('test'.'test-1')
>>> os.path.split("test/test-1/")    # Directory split ending in /
('test/test-1'.' ')
>>> os.path.split("test/test-1/test-1.txt")  # split file
('test/test-1'.'test-1.txt')
Copy the code
  • Splitext: Split pathname and file extension, separating path with the last extension “. Split, cut into head and tail, and return as a (head, tail) tuple. Note that the difference with split is the separator.
>>> os.path.splitext("test/test-1")  
('test/test-1'.' ')
>>> os.path.splitext("test/test-1/") 
('test/test-1/'.' ')
>>> os.path.splitext("test/test-1/test-1.txt")  # distinguish between filename and extension
('test/test-1/test-1'.'.txt')
>>> os.path.splitext("test/test-1/test-1.txt.tmp") # use the last "." as the split point
('test/test-1/test-1.txt'.'.tmp')
Copy the code

5 Example Application

The following is a comprehensive use of the previous operation functions with some common scenarios.

5.1 Changing file Names in Batches

def batch_rename(dir_path):
    itemlist = os.listdir(dir_path)
    Get a list of directory files
    for item in itemlist:
        Connect to the full path
        item_path = os.path.join(dir_path, item)
        print(item_path)
        # change file name
        if os.path.isfile(item_path):
            splitext = os.path.splitext(item_path)
            os.rename(item_path, splitext[0] + "- a copy of the" + splitext[1])
Copy the code

5.2 Traversal all files with the specified extension in a directory and subdirectories


def walk_ext_file(dir_path,ext):
    # traversal
    for root, dirs, files in os.walk(dir_path):
        Get file name and path
        for file in files:
            file_path = os.path.join(root, file)
            file_item = os.path.splitext(file_path)
            Print the path to the file with the specified extension
            if ext == file_item[1]:
                print(file_path)
Copy the code

5.3 Sorting files in a specified directory by modification time

def sort_file(dir_path):
    # before ordering
    itemlist = os.listdir(dir_path)
    print(itemlist)
    # forward sort
    itemlist.sort(key=lambda filename: os.path.getmtime(os.path.join(dir_path, filename)))
    print(itemlist)
    # reverse sort
    itemlist.sort(key=lambda filename: os.path.getmtime(os.path.join(dir_path, filename)), reverse=True)
    print(itemlist)
    Get the latest modified file
    print(itemlist[0])
Copy the code

6 summarizes

Python is a quick and easy choice when you need to manipulate files or directories. This article introduces the common methods of python’s standard built-in OS module and its submodule os.path, and finally makes comprehensive use based on application scenarios. I believe I have met most of your requirements for file and directory operations.

The resources

  • OS module for Python:https://www.cnblogs.com/yufeihlf/p/6179547.html
  • Python OS file/directory method: https://www.runoob.com/python/os-file-methods.html
  • Python OS. The path () module: https://www.runoob.com/python/python-os-path.html

The articles

  • Distributed deployment of MinIO
  • Use MinIO to easily build static resource services
  • SpringBoot Multiple data sources (3) : Parameterize change sources
  • SpringBoot Multiple Data Sources (2) : dynamic data sources
  • SpringBoot Multiple Data Sources (1) : Multiple source policies
  • Java development must learn: dynamic proxy
  • Good books to read in 2019

My official account (search Mason technical records) for more technical records: