Topic:
- How can a large number of files be transferred between Windows and the local MAC computer without a USB flash drive?
- Why PNPM? What are hard links and soft links?
The cause of
Recently, my office equipment has been changed from ThinkPad to macbook PRO14. Due to the powerful performance of M1 chip, my work efficiency has been greatly improved. I even carry it home every day without feeling tired. It suddenly occurred to me that there were still many project codes on the computer before that had not been migrated, so I could clone them directly, and the rest were some old codes of historical projects that were out of print. I had U disk and portable hard disk on hand, but MAC did not have USB interface, and the adapter was also in the company. And the hard drive isn’t directly MAC accessible, requiring additional software to handle formatting. Well, on second thought, there’s something like this.
Fast spread website
Occasionally used cows before coming fast transmission zip files, because network under certain NPM package was not down, so use this website to my colleagues, but that is too slow to limit too much, and the solution to file away himself again they server, not suitable for transmission privacy file, not goozyet. pass
LAN Sharing
I remember that most companies have Intranet sharing, which stores access to shared folders and can be copied locally. Is this supported between MAC and Windows? After a bit of searching, it works (happily), so give it a try.
First, ensure that Windows and MAC devices are on the same network (the easiest way is to connect to the same wifi). Right-click the folder to share and select “Share this folder”. By default, Everyone => Apply and confirm. At this point, the folder is configured as a shared folder, which can be accessed by other devices in the current LAN using the device’S IP + user account password.
Open Finder under MAC, menu bar => Go to => Connect server => enter SMB :// IP address => Connect => If the link is successful, you will need to enter the device account password for verification, after passing, you will see a new device option in the Finder’s sidebar. Open it to see the previously shared folder, which can be accessed and moved around like regular files.
At this point, I thought I would be happy to copy and paste, but I found that there are too many projects, dozens of large and small projects. The worst thing is that there are many projects with node_modules folder. The code is only a few meters, but node_modules has 200+M. So basically each project has a separate copy, well, now you have to delete node_modules first, and then move the project code, which is pretty quick.
If you have used M1 before, you may have heard of the concept of SSD swap. It is one of the reasons why MacOS system is more smooth than Windows system under the same memory specifications. When the system has too many applications, Windows will stall significantly when memory is low, but macOS will do much better because it will move unused application caches from ram to SSD to make room for memory.
This process is no perception so you will see clearly opened several large application theory has long been beyond the memory size, but still not have obvious card, it is for this reason, especially with the augmentation of the SSD performance as well as the change of the M1 chip architecture, the use of this feature is better, on the one hand, this is a good thing, It takes full advantage of SSD performance and uses less memory for a better experience than ever before, but there are two sides to the story. Swap’s philosophy means that it allows for more SSD reads and writes, whereas SSDS have a lifetime, which means that there is a maximum number of writes.
The life of a hard disk is usually expressed in TBW (maximum write capacity). A 1T SSD solid state usually has a nominal warranty life of 600TBW. You may have noticed that the warranty life here is not the service life, yes, everyone says that the actual life is much greater than the warranty life. My 512GB hard disk is the nominal write capacity of 300TBW, which is enough to use originally. According to ordinary people’s usage habits, even I, as a developer, can use it for 10 hours a day for office and daily entertainment. This is based on my usage habits in the past 50 days, the latest statistics was on February 1st, the average monthly 3T, 3*12 = 36T/ year, nearly ten years of life.
However, after last night’s operation, I checked the company today and found that the swap volume increased by 6T (1T==1024GB) within a week. It was very scary, I really couldn’t figure it out. I suspected that it was the data read and write surge caused by last night’s file operation. So the disk reads the number of files and processes them sequentially, consuming more disk than a single compressed file; Node_modules may have some missing files that have not been deleted. The multi-tree organization is very complex, and the process of indexing the files must be disk consuming.
Although I have many projects, the size of 6T is really incredible. It is amazing that SWAP can reach 6144G. I have no idea for the time being, but I will record it first.
What can PNPM do
At that time, I was thinking, if my project uses PNPM to manage packages, node_modules only saves a complete copy of the package, when I need to batch backup code directly compressed package on the line, do not need to delete each project cache of package files, a lot of worry.
How does PNPM do it
This involves two concepts in the file system: hard links and soft links;
Hard links
In The Linux file system, files stored in disk partitions are assigned an Inode number (Inode Index) regardless of their type. In Linux, multiple file names are allowed to point to the same Inode, and these connections are usually hard links.
The function of hard links is to allow a file to have multiple valid pathnames, so that users can establish hard links to important files to prevent “accidental deletion” function. The reason for this is as described above, because there is more than one link to the index node that should be the directory. Deleting only one link does not affect the index node itself or the other links. Only when the last link is deleted will the data block of the file and the link of the directory be released. In other words, the condition for a file to be deleted is that all hard-linked files related to it are deleted.
Syntax: ln filename [linkname]
- Create a hard link to add a file name to the same inode (essentially add a file name mapping to the same inode in the directory entry).
- When a new hard link is created, the number of links increases. If the file name mapped to the inode does not exist, the inode will be recycled.
- Hard-linked files and original files share data, but are independent of each other. (If you modify the data of any one of the files, the data of other files will be changed. If you delete the hard link file, the number of corresponding links will be reduced. If it is the last link number, the file will be deleted directly.)
- Hard links cannot be created across partitions and devices
- Hard links cannot be created to directories (directories can have up to three hard links: the directory itself,. Under the directory, and… under subdirectories).
You are advised to run the following commands on a Linux VM or MacOS:
1. Create poetryFile $touch poetryFile 2. $echo '$echo '>poetryFile $cat poetryFile Use ln to create a hard link file in the same path. HardPoetryFile $ln poetryFile hardPoetryFile 4. Modify hard linked file, add 'hardPoetryFile' $echo 'hardPoetryFile $cat poetryFileCopy the codeSoft links
A bit like the shortcut in Window, it is also a file itself, but save is the full path of the file it points to, access will be through it to access the file path to open the specified file, so when deleting the source file, open it will report an error indicating no related path.
Ln -s filename [linkname]
- The essence of a soft link is to create a shortcut to a file. The data stored is the name of the original file and the data size is the number of bytes of the original file name. Access to the original file data through the file name
- Soft links are supported across partitions
- You can create directory soft links
- Soft-linked files depend on the original file; If the original file is deleted, the soft link file becomes invalid
Example: Create a soft link based on poetryFile
$ln -s poetryFile softPoetryFile $cat softPoetryFile $cat softPoertFile $cat softPoertFile $cat softPoertFile $cat softPoertFile $cat softPoertFile I was born to be usefulCopy the codeIn the current directory of the file, you can view the information through lS-HL. It can be found that the hard link file and the source file have the same information, but the soft link file only stores the pointing information, so you can see the obvious difference.
total 16
-rw-r--r--@ 2 zhoumingjie staff 66B 2 9 10:24 hardPoetryFile
-rw-r--r--@ 2 zhoumingjie staff 66B 2 9 10:24 poetryFile
lrwxr-xr-x 1 zhoumingjie staff 10B 2 9 10:21 softPoetryFile -> poetryFile
Copy the codePNPM to hard link, soft link application
When using NPM or Yarn, if you have 100 projects using a dependency, 100 copies of that dependency are saved on hard disk. With PNPM, dependencies are stored in content-addressable storage, so if you use different versions of a dependency, only the different files are added to the repository. For example, if a package has 100 files and its new version changes only one of them. PNPM update only adds a new file to the storage center, instead of copying the contents of the entire new version of the package for just one file change. All files are stored somewhere on the hard drive. When a package is installed, the files in the package are hardlinked to this location without taking up additional disk space. This allows you to share the same version of dependencies across projects. As a result, you save a lot of space on disk, proportional to the number of projects and dependencies, and install much faster! From: PNPM. IO/useful/motivati…
Schematic diagram of official introduction:
Before the appearance of PNPM, NPM and YARN adopted a flat pair policy to improve package loading efficiency and reuse rate, that is, all dependent packages are stored in the root directory. As a result, the packages in node_modules differ greatly from those defined in package.json. This causes ghost dependencies (ghost dependencies are packages used in a project that are not defined in its package.json file).
As shown in figure:
ls -hlLooking at its details, you can see that it points to the Express file under.pnpm, PNPM /[package_name]@version/node_modules/[package_name].
Total 0 lrwxr-xr-x 1 Zhoumingjie staff 41B 29 13:29 Express ->. PNPM /[email protected]/node_modules/ Express lrwxr-xr-x 1 Zhoumingjie staff 47B 29 13:29 MIme-types ->. PNPM /[email protected]/node_modules/mime-typesCopy the codePNPM file is a hard link to the source file. To verify this, first find the express file path under the pnpm-test project, and read the details of the terminal command stat -s index.js. St_ino =5206568; express =5206568; st_ino=5206568; You can see that index.js under Express is actually being reused.
PNPM config set store-dir /path/to/. Pnpm-store: PNPM config set store-dir /path/to/. Pnpm-store: PNPM config set store-dir /path/to/
conclusion
PNPM == P (Performant) + NPM, which stands for high performance NPM, I think the availability is good enough so far, although there may be some migration issues like ghost dependencies, it is still worth upgrading. Especially for new projects, use it mindlessly.
Reference links:
- PNPM document
- Linux file index node knowledge
- Linux soft and hard links
- Understand the front-end package management tool from soft link & hard link