Recently the company’s business to do a report management system, which involves massive file storage problems. For file storage, it’s usually simple to configure a storage directory in your Django configuration file and upload or download files according to the rules.

Actually, when there are fewer files, Django can handle it. But when it comes to massive files, Django doesn’t work that well, so Fast DFS was introduced.

FastDFS is an open source distributed file system. It manages files, including file storage, file synchronization, and file access (file upload and download). It solves the problems of large-capacity storage and load balancing. It is especially suitable for online services with file as the carrier, such as photo album website, video website and so on. It was born for the Internet, for big data.

The FastDFS server has two roles: tracker and storage. The tracker mainly does scheduling work and plays a load balancing role in access. Storage nodes store files and perform all functions of file management, including storage, synchronization, and access interfaces. FastDFS also manages meta data of files. Both the tracker and the storage node can consist of multiple servers. Servers in both the tracker and the storage node can be added or taken offline at any time without affecting online services. All servers in the tracker are peer and can be added or reduced at any time according to the stress of the server.

Reasons to use FastDfs:

1 Provides mass storage and convenient storage capacity expansion. 2 solve the file content duplication, if the user uploads the file duplication (file fingerprint is the same), then the system only stores a piece of data, it is worth mentioning that this technology is currently widely used in the network disk. 3 combine Nginx to improve the efficiency of website reading pictures.

If we build fastdFS server from scratch then it is too low, there are a lot of Docker images for you to choose from, so it is time to take advantage of the advantages of Docker.

Download the FastdFS image

sudo docker pull delron/fastdfs
Copy the code

2. View the downloaded image

sudo docker images
Copy the code

Just over 400 megabytes are loaded with Nginx and FastDFS services

3. Then use docker image to build tracker container (tracking server, playing the role of scheduling), where tracker service will be automatically mapped to the host

docker run -d --network=host --name tracker -v /root:/var/root delron/fastdfs tracker
Copy the code

Use the docker mirror build storage containers (backup storage server, provide the capacity and service), storage containers need to rely on the tracker service here, to you the IP address of the tracker service, the default port is 22122, the IP address is also your hosting IP

docker run -d --network=host --name storage -e TRACKER_SERVER=192.16899.100.:22122 -v /root:/var/root -e GROUP_NAME=group1 delron/fastdfs storage
Copy the code

At this point, enter sudo docker ps on the command line to see that both sets of services have been started

4, Now that the deployment is complete, we enter the background running storage container, test

sudo docker exec -it storage /bin/bash
Copy the code

Download any image, don’t worry about that, because in the container, the image will not be saved unless it is submitted to the repository

wget https://upload-images.jianshu.io/upload_images/11693390-a26b21909429f7d2.png
Copy the code

5. Upload the image to the distributed system by command

/usr/bin/fdfs_upload_file /etc/fdfs/client.conf 11693390-a26b21909429f7d2.png
Copy the code

The image has been uploaded to the file system, and the network address where the image is stored is returned after this statement is executed

This image is a static resource through the nginx proxy. By default, nginx listens on port 8888, so it needs to add the port number. If it is deployed on Ali Cloud, it needs to expose the external port 8888

As you can see, there is no problem. Similarly, if it is a video resource, you can also upload it to Fastdfs. Done.

Now that you’re done uploading files to the host, Python can also execute commands from the command line. You can use the URL from the command to communicate with Fastdfs, using python’s os.popen method. You can simply get the fastdfs network address returned from the command line, avoiding the Python client that has to install Fastdfs, which is not very python3 friendly. The code is as follows:

import os
import re

std = os.popen("docker exec -i storage /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /var/root/test.mp4").read()
print('*********** fastdfs excute start ***********')
print(std.strip())
print('*********** fastdfs excute end ***********')
Copy the code

In this way, when you upload a file from Django, you can upload it to Fastdfs by command, obtain the return address, and then store it in the library. This operation uses the features of Docker to the extreme, which shows that docker is widely used, and actually improves the development efficiency.