This article aims to provide an in-depth discussion of Linux file system concepts. This article is neither intended as a foundational description of a particular type of file system, such as ext4, nor as a tutorial on file system commands.

Every general-purpose computer needs to store various types of data on a hard disk drive (HDD) or some equivalent device, such as a USB. There are several reasons to store it on these devices. First,RAM loses its contents when the computer is powered off, although there are also non-volatile types of memory that can maintain data storage after power is turned off (such as Flash memory, which is the USB and solid-state drives used for flash memory), But Flash memory is much more expensive than standard, volatile memory such as DDR3 and other similar types.

The second reason data needs to be stored on a hard drive is that even standard RAM is more expensive than disk space. Both RAM and disk costs are falling rapidly, but RAM is still higher in terms of cost per byte. A quick calculation of the per-byte cost, based on the cost of 16GB of RAM and 2TB of hard drive, shows that RAM is about 71 times more expensive than a hard drive. Currently, the typical cost of RAM is about 0.0000000043743750 per byte.

To put it more bluntly, in the early days of computers, a kind of memory was based on points on the CRT screen, at about $1 per bit, which was very, very expensive!

define

You may hear the word file system often talked about in different or confusing ways. The word itself can have multiple meanings, and you may need to discern the true meaning from the context of the discussion or document.

I’ll try to define the various meanings of “file system” based on how I use it in different situations. Note that in trying to follow the standard “official” meaning, my intention is to define the term in terms of its various uses. These implications are discussed in more detail in the later sections of this article.

  1. The wholeLinuxThe directory structure starts at the top (/) root directory.
  2. Various specific types of data storage formats, such asEXT3,EXT4,BTRFS,XFSAnd so on.LinuxSupport near100Types of file systems, including some very old file systems, as well as some of the latest file systems. Each file system type uses its own metadata structure to define how data is stored and accessed.
  3. A partition or logical volume formatted as a specific type of file system that can be mounted toLinuxAt the specified mount point on the file system.

Basic file system functionality

Disk storage is required, and it brings some interesting and inescapable details. Obviously, a file system is designed to provide space for non-volatile storage of data, which is its most fundamental function. But it also has many other important features that meet different needs.

All file systems need to provide a namespace — a method of naming and organizing. It defines how files are named, specifically the length of the file name and a subset of the characters available for the file name, which can be taken from the full set of characters. It also defines the logical structure of the data on disk, such as using directories to organize files instead of concentrating them in a single, huge set of files.

Once a namespace is defined, a metadata structure is needed to provide the logical foundation for that namespace. Metadata includes data structures needed to support hierarchical directory structures. Structures used to determine which blocks of space on the disk are used and which are available; Allows the structure of file and directory names to be maintained; Information about files, such as their size and time, such as when they were created, modified, or last accessed, etc. And the location of the data belonging to the file on the disk. Other metadata is used to store advanced information about disk partitioning, such as logical volumes and partitions. This higher-level metadata and the structure it represents contains information that describes the file system stored on the drive or partition, separate from the generic file system metadata mentioned above.

File systems also require API interfaces to provide access to system function calls that manipulate file system objects such as files and directories. APIs provide interfaces for things like creating, moving, and deleting files. It also provides algorithms to determine where files are placed on the file system. These algorithms are also used to determine speed or minimize disk fragmentation.

Modern file systems also provide a secure schema, which is a scheme for defining access permissions for files and directories. Linux file system security mode helps ensure that users can only access their own files, not those of other people or the operating system itself.

The final building blocks are the software needed to implement all of these functions. To improve system and programmer efficiency, Linux uses a two-part software implementation.




The first part of this two-part implementation is the Linux virtual file system. This virtual file system provides the kernel and developers with a set of commands to access all types of file systems. Virtual file system software invokes specific device drivers to connect to various types of file systems. File system specific device drivers are the second part of the implementation. The device driver translates and interprets a standard set of file system commands based on the type of file system on a particular partition or logical volume.

The directory structure

As a normally very organized Virgo, I like to keep things in small, organized places rather than a big bucket. Using directories helps me store and locate the files I’m looking for. Directories are also known as folders because they can be thought of as the folders that hold files on your desk in real life.

In Linux and many other operating systems, directories can be constructed in a tree-like hierarchy. The Linux directory structure is well defined and documented in the Linux File System Hierarchy Standard (FHS). Referencing these directories when accessing them is done by using deep-order directory names (/) connected by a slash (/), such as /var/log and /var/spool/mail. These are called paths.

The following table provides a very short list of standard, well-known, and top-level Linux directories and their uses.




Table 1 shows directories and subdirectories and subdirectories of subdirectories, where directories with a blue background are considered indispensable components of the root file system. That is, they cannot be created as separate file systems and installed at startup. This is because they (and especially their content) must appear at boot time in order for the system to boot properly.

The /media and/MNT directories are part of the root file system, but they should not contain any data. Instead, they are temporary mount points. The rest of the directories, which have no background color in Table 1, do not need to appear in the boot sequence, but will be installed later, preparing the host to perform useful work in the boot sequence.

You can learn more about these directories and their many subdirectories by referring to the official Linux File System Hierarchy Standard (FHS) Web page. Wikipedia also has a good description of FHS. This standard should be followed as closely as possible to ensure operational and functional coherence. This hierarchical directory structure is the same regardless of the type of file system used on the host.

Linux unified directory structure

On some non-Linux PC operating systems, if there are multiple physical hard disks or partitions, each disk or partition is assigned a drive letter. A drive letter, such as C: or D:, is required to locate the hard disk where a file or program resides. Then, issue the drive letter D: in the form of a command, for example, to change to D: drive, and then use the CD command to change to the correct directory to locate the desired files. Each hard drive has its own separate and complete directory tree.

Linux file systems unify all physical hard disks and partitions into a single directory structure. It all starts at the top (/) directory. All other directories and their subdirectories are located under the Linux root. This means that there is only a single directory tree to search for files and programs.

They all work because file systems such as /home, / TMP, /var, /opt, /usr can be created on a different partition on a separate physical hard disk, or a different logical volume /(root) file system, and then installed on a mount point (directory) as part of the root file system tree. Even removable drives, such as flash drives or external USB or ESATA hard drives, are installed into the root file system and become an integral part of the directory tree.

File systems can do this for a good reason when upgrading from one Linux distribution to another, or switching from one distribution to another. In general, with the exception of upgrade tools such as Fedora’s DNF upgrade, it is wise to occasionally reformat the hard drive that contains the operating system, because during the upgrade process the hard drive will wipe out anything that has accumulated over time. If /home is part of the root file system, it will be reformatted and then must be restored from the backup. By formatting /home as a separate file system, the root file system is recognized as a separate file system when it is formatted, and the current step can be skipped. This also applies to /var for databases, email inboxes, web sites, and other variable user and system data stores.

There are other reasons to maintain parts of the Linux directory tree as separate file systems. For example, long ago, when I wasn’t aware of the potential problems surrounding all the required Linux directories being part of the /(root) file system, I filled my home directory with a lot of very large files. Since the /home and/TMP directories are not separate file systems, but subdirectories of the root file system, the entire root file system is filled. The operating system does not have room to create temporary files or extend existing data files. At first, applications start complaining about not having room to save files, and then the operating system itself starts to get really weird. Booting into single-user mode and clearing out the problem files in my home directory allowed me to start over. I then reinstalled Linux using a fairly standard multi-file system setup and was able to prevent a complete system crash from happening again.

I also had a situation where the Linux host continued to run but prevented the user from logging in using the GUI desktop. I was able to use the command line interface (CLI) locally and SSH remotely using a virtual console. The problem was that the/TMP file system was full, and some temporary files needed for the GUI desktop could not be created at login. Since CLI login does not require file creation in/TMP, the lack of space did not prevent me from using CLI login. In this case, the/TMP directory is a separate file system with a large amount of available space in the volume group, of which the/TMP logical volume is part. I simply expanded the/TMP logical volume to a large enough size (essentially LVM capacity expansion) to accommodate my new requirements for the amount of temporary file space that the host required and solved the problem. Note that this solution does not require a reboot, and when the/TMP file system is enlarged, the user can log in to the desktop.

Logical volume extension can also be referred to the article I summarized earlier, which is concise and straightforward: LVM dynamic extension

Another situation occurred when I was working as a lab administrator at a large technology company. One of our developers installed the application in the wrong location (/var) (I personally don’t think it was installed in the wrong location, just the space available in the wrong location). The application crashed because the /var file system was full and the log file stored on /var/log/ could not add new messages due to lack of space. However, the system kept running because the key /(root) and/TMP file systems were not populated. Removing the offending application and reinstalling it into the /opt file system solves the problem. (You can also solve this problem by dynamically expanding the file system using LVM, either by increasing the size of the file system or by replacing it with a larger file system.)

File System Type

Linux supports reading of about 100 partition types, and it can create or write files on only a few of them, not all of them. However, it is possible and very common to mount different types of file systems on the same root file system. In this context, we are talking about the structure and metadata needed to store and manage user data on partitions of hard disks or logical volumes. A complete list of file system partition types recognized by the Linux fdisk command is provided here, so you can see the high degree of compatibility between Linux and many types of systems.

0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 84 OS/2 hidden or c6 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx 5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data 6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility 8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt 9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access a OS/2 Boot Manag 50  OnTrack DM 94 Amoeba BBT e3 DOS R/O b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi ea Rufus alignment e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD eb BeOS fs f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ee GPT 10 OPUS 55 EZ-Drive a7 NeXTSTEP ef EFI (FAT-12/16/ 11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f0 Linux/PA-RISC b 12 Compaq diagnost 5c Priam Edisk a9 NetBSD f1 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f4 SpeedStor 16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ f2 DOS secondary 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fb VMware VMFS 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fc VMware VMKCORE 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fd Linux raid auto 1c Hidden W95 FAT3 75 PC/IX bc Acronis FAT32 L fe LANstep 1e Hidden W95 FAT1 80 Old Minix be Solaris boot ff BBTCopy the code

The main purpose of having the ability to support reading so many partition types is to allow compatibility and some interoperability with the file systems of other computer systems. The options available when creating a new file system using Fedora are shown below.

  • btrfs
  • cramfs
  • ext2
  • ext3
  • ext4
  • fat
  • gfs2
  • hfsplus
  • minix
  • msdos
  • ntfs
  • reiserfs
  • vfat
  • xfs

Other distributions support the creation of different file system types. For example, CentOS 6 supports only the file systems shown in bold.

mount

In Linux, the term to mount refers to early computers when a tape or removable disk package needed to be physically mounted on the appropriate drive. After the disks are physically placed, the file systems on the disk package are logically mounted by the operating system to make the content accessible to the operating system, applications, and users.

A mount point is simply a directory, just like any other directory, that is created as part of the root file system. For example, the home file system is installed on the directory /home. File systems can be mounted at mount points on other non-root file systems, but this is not common. The Linux root file system is installed on the root directory (/) very early boot sequence. Other file systems are installed later by Linux boot programs, whether rc under SystemV or systemd in newer Linux versions. Mounting file systems during boot is managed by the /etc/fstab configuration file. One easy way to remember is that fstab stands for “file system table,” which is a list of file systems to mount, along with the mount points they specify, and any options that a particular file system might need.

The file system is installed on an existing directory (mount point) using the mount command. In general, any directory that is used as a mount point should be empty and not contain any other files in it. Linux does not prevent users from installing a file system on an existing file system, or on a directory containing files. If you install a file system on an existing directory or file system, the original contents will be hidden and only the contents of the newly mounted file system will be visible.

conclusion

I hope this article clears up some of the possible confusion surrounding the term “file system.” It took me a long time to really understand and appreciate the complexity, elegance, and functionality of the Linux file system.

If you have questions, please add them to the comments below and I’ll try to answer them.

Next month,

Another important concept is that with Linux, everything is a file. This concept provides some interesting and important real-world applications for users and system administrators. I mention this because you might want to read my Everything files article before my article in the /dev directory next month.

The original address: opensource.com/life/16/10/…

I also translated the article “Everything is a document”

Translation address: www.tony-yin.top/2017/12/21/…