Use FastDFS to build file service Docker one click launch integration version - Running on the Shoulders of Giants series

What is FastDFS

Recently, the company’s business needs to build a file server, after various consulting and searching, decided to use FastDFS. So what are the advantages of FastDFS?

FastDFS is an open source distributed file system written in C language. FastDFS is tailor-made for the Internet. It takes into account redundant backup, load balancing, and linear expansion, and emphasizes high availability and performance. It is easy to set up a high-performance file server cluster to provide file uploading and downloading services.

For more details, you can refer to: articles from ITyouKnow: click to view

How to use docker image

In order to integrate into existing services more easily, we need a docker image that can run with one click, but there are many on the network, some have separated libfastCommon, some have cumbersome configuration, and none of them are well integrated or do not meet business requirements. So, let’s write the Dockerfile step by step to implement one-click operation.

If you’re lazy, just run the image below:

docker run -d annoak/fastdfs:latest \
    -p 8888:8888 \
    -p 22122:22122 \
    -e TZ=Asia/Shanghai \
    -v /data/fdfs:/var/local/fdfs \
    --restart=always
Copy the code

If you need to customize the port

docker run -d annoak/fastdfs:latest \
    -p 8888:8888 \
    -p 22122:22122 \
    -e TZ=Asia/Shanghai \
    -e NGINX_PORT=8888 \
    -e FDFS_PORT=22122 \
    -v /data/fdfs:/var/local/fdfs \
    --restart=always
Copy the code

If you need to customize the HOST

The first type: [Pass in the network name, automatically obtain the IP address]

docker run -d annoak/fastdfs:latest \
    -p 8888:8888 \
    -p 22122:22122 \
    -e TZ=Asia/Shanghai \
    -e NET_VAR=eth0 \
    -v /data/fdfs:/var/local/fdfs \
    --restart=always
Copy the code

Type 2: [Direct incoming IP address]

docker run -d annoak/fastdfs:latest \
    -p 8888:8888 \
    -p 22122:22122 \
    -e TZ=Asia/Shanghai \
    -e HOST_IP=xxx.xxx.xxx.xxx \
    -v /data/fdfs:/var/local/fdfs \
    --restart=always
Copy the code

If you do not want to map ports directly use the host network:

docker run -d annoak/fastdfs:latest \
    --net=host \
    -e TZ=Asia/Shanghai \
    -v /data/fdfs:/var/local/fdfs \
    --restart=always
Copy the code

If you need to customize nginx versions

docker run -d annoak/fastdfs:latest \
    --net=host \
    -e TZ=Asia/Shanghai \
    -eNGINX_VERSION = 1.17.0 \ - v/data/FDFS: / var /local/fdfs \
    --restart=always
Copy the code

If you’re using Docker-compose

services:
    fastdfs:
        image: annoak/fastdfs:latest
        container_name: my-fastdfs
        environment:
            - TZ=Asia/Shanghai
            - NGINX_PORT=8888
            - FDFS_PORT=22122
            - HOST_IP=192.168.198.107
            - NET_VAR=eth0
        volumes:
            - /data/fdfs:/var/local/fdfs
        #ports:# No mapping required if host network is used
        # - "8888-8888"
        # - "22122-22122"
        Add the root permission to the file
        privileged: true
        network_mode: "host"
        restart: always
Copy the code

If you want to know how it came about, keep reading:

How to write dockerfile

The first step, of course, is to create a DockerFile and look at the version of the software we need to use.

Libfastcommon and Fastdfs are master versions, nginx defaults to 1.17.0, you know.

Specific version view:

Libfastcommon: Click to jump

Fastdfs: Click to jump

Nginx: Click to jump
Step 2, we need to select an environment, well, we select the minimum Linux -> alpine, now dockerfile looks like this:

The FROM alpine: 3.7Copy the code

Step 3: Define some global variables

# Working directory
ENV HOME=/root/fastdfs \
    # nginx versionNGINX_VERSION = 1.17.0 \# Nginx port default
    NGINX_PORT=8888 \
    The default value of the network where the IP address resides
    NET_VAR=eth0 \
    # fastdfs Port default value
    FDFS_PORT=22122
Copy the code

Step 4, we need to install some compile-time dependencies and update the system software

Create a directory
RUN mkdir -p ${HOME}

Upgrade package
RUN apk update

# Install the necessary software and add -- virtual. mybuilds,
RUN apk add --no-cache --virtual .mybuilds \
    bash \
    gcc \
    make \
    linux-headers \
    curl \
    gnupg \
    gd-dev \
    pcre-dev \
    zlib-dev \
    libc-dev \
    libxslt-dev \
    openssl-dev \
    geoip-dev
Copy the code

Why add — virtual. mybuilds and what is it? When you install packages, they are not added to the global package. And it can be easily restored. For example, I need GCC to compile a program, but once the program is compiled, I don’t need GCC anymore.

Step 5, we need to check the directory, then download the Fastdfs dependency libFastCommon and compile and install it

Download and install libfastCommon
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/libfastcommon/archive/master.tar.gz -o fastcommon.tar.gz \
    && tar zxf fastcommon.tar.gz \
    && cd ${HOME}/libfastcommon-master/ \
    && ./make.sh \
    && ./make.sh install
Copy the code

Step 6: Download, compile, and install Fastdfs

Download and install Fastdfs
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/fastdfs/archive/master.tar.gz -o fastfs.tar.gz \
    && tar zxf fastfs.tar.gz \
    && cd ${HOME}/fastdfs-master/ \
    && ./make.sh \
    && ./make.sh install
Copy the code

Step 7: Configure fastdfs and replace /home/yuq/fastdfs

# configuration fastdfs
RUN cd /etc/fdfs/ \
    && cp storage.conf.sample storage.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/storage|g" /etc/fdfs/storage.conf \
    && cp tracker.conf.sample tracker.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/tracker|g" /etc/fdfs/tracker.conf \
    && cp client.conf.sample client.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/storage|g" /etc/fdfs/client.conf 
Copy the code

Step 8, ok, next let’s install nginx and the nginx plug-in

# Download the nginx plugin
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/fastdfs-nginx-module/archive/master.tar.gz -o nginx-module.tar.gz \
    && tar zxf nginx-module.tar.gz

# download nginx
RUN cd ${HOME}/ \
    && curl -fSL http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o nginx-${NGINX_VERSION}.tar.gz \
    && tar zxf nginx-${NGINX_VERSION}.tar.gz
Copy the code

Step 9, we will compile and install nginx brothers together

# Compile nginx and Fastdfs nginx plugins
RUN cd ${HOME} \
    && chmod u+x ${HOME}/fastdfs-nginx-module-master/src/config \
    && cd nginx-${NGINX_VERSION} \
    && ./configure --add-module=${HOME}/fastdfs-nginx-module-master/src \
    && make && make install
Copy the code

No such file or directory #include “common_define. H”

Dockerfile: fastdfs-nginx-module-master/ SRC /config: fastdfs-nginx-module-master/ SRC /config: fastdfs-nginx-module-master/ SRC /config:

ngx_addon_name=ngx_http_fastdfs_module
if test -n "${ngx_module_link}"; then
    ngx_module_type=HTTP
    ngx_module_name=$ngx_addon_name
    ngx_module_incs="/usr/local/include"
    ngx_module_libs="-lfastcommon -lfdfsclient"
    ngx_module_srcs="$ngx_addon_dir/ngx_http_fastdfs_module.c"
    ngx_module_deps=
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
    . auto/module
else
    HTTP_MODULES="$HTTP_MODULES ngx_http_fastdfs_module"
    NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_fastdfs_module.c"
    CORE_INCS="$CORE_INCS /usr/local/include"
    CORE_LIBS="$CORE_LIBS -lfastcommon -lfdfsclient"
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
fi
Copy the code

Then, before compiling that section in the dockerfile, add

# Fix fatal bug where compilation could not find #include "common_define.h
ADD ./config ${HOME}/fastdfs-nginx-module-master/src
Copy the code

Step 10, configure the Nginx and FastdFS environment. Configure nginx

# Configure nginx and FastdFS
RUN cp ${HOME}/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs/ \
    && sed -i "s|^store_path0.*$|store_path0=/var/local/fdfs/storage|g" /etc/fdfs/mod_fastdfs.conf \
    && sed -i "s|^url_have_group_name=.*$|url_have_group_name=true|g" /etc/fdfs/mod_fastdfs.conf \
    && cd ${HOME}/fastdfs-master/conf/ \
    && cp http.conf mime.types anti-steal.jpg /etc/fdfs/ \
    && echo -e "events {\n\ worker_connections 1024; \n\ }\n\ http {\n\ include mime.types; \n\ default_type application/octet-stream; \n\ server {\n\ listen \$NGINX_PORT; \n\ server_name localhost; \n\ location ~ /group[0-9]/M00 {\n\ ngx_fastdfs_module; \n\ }\n\ }\n\ }">/usr/local/nginx/conf/nginx.conf
Copy the code

Step 11, we’re almost there. Let’s clean up the files and software we don’t need anymore

Clean up temporary software and files
RUN rm -rf ${HOME}/*
RUN apk del .mybuilds
RUN apk add bash pcre-dev zlib-dev
Copy the code

Step 12, we seem to need a startup script. Next, write a startup script. Note the NET_VAR variable here, which I’ve made a hole in. The default is eth0, which is suitable for Linux, but if it’s a MAC, you need to assign it the value en0, or you can specify HOST_IP directly

Create a startup script
RUN     echo -e "mkdir -p /var/local/fdfs/storage/data /var/local/fdfs/tracker; \n\ ln -s /var/local/fdfs/storage/data/ /var/local/fdfs/storage/data/M00; \n\n\ sed -i \"s/listen\ .*$/listen\ \$NGINX_PORT; /g\" /usr/local/nginx/conf/nginx.conf; \n\ sed -i \"s/http.server_port=.*$/http.server_port=\$NGINX_PORT/g\" /etc/fdfs/storage.conf; \n\ if [ \"\$HOST_IP\" = \"\" ]; then \n\ HOST_IP=\$(ifconfig \$NET_VAR | grep \"inet\" | grep -v \"inet6\" | awk '{print \$2}' | awk -F: '{print \$2}')\n\ fi \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/storage.conf;  \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/client.conf; \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/mod_fastdfs.conf; \n\ /etc/init.d/fdfs_trackerd start; \n\ /etc/init.d/fdfs_storaged start; \n\ /usr/local/nginx/sbin/nginx; \n\ tail -f /usr/local/nginx/logs/access.log \ ">/start.sh \
&& chmod u+x /start.sh
Copy the code

Step 13, ok, expose the port, configure the startup script, and call it a day

# Exposed port
EXPOSE ${NGINX_PORT} ${FDFS_PORT}
ENTRYPOINT ["/bin/bash"."/start.sh"]
Copy the code

Summary: The complete Dockerfile

FROM Alpine :3.7 MAINTAINER Oak <[email protected]># define variables
ENV HOME=/root/fastdfs \
    # nginx versionNGINX_VERSION = 1.17.0 \# Nginx port default
    NGINX_PORT=8888 \
    # Network where the IP address resides
    NET_VAR=eth0 \
    # fastdfs Port default value
    FDFS_PORT=22122

Create a directory
RUN mkdir -p ${HOME}

Upgrade package
RUN apk update

Install the necessary software
RUN apk add --no-cache --virtual .mybuilds \
    bash \
    gcc \
    make \
    linux-headers \
    curl \
    gnupg \
    gd-dev \
    pcre-dev \
    zlib-dev \
    libc-dev \
    libxslt-dev \
    openssl-dev \
    geoip-dev

Download and install libfastCommon
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/libfastcommon/archive/master.tar.gz -o fastcommon.tar.gz \
    && tar zxf fastcommon.tar.gz \
    && cd ${HOME}/libfastcommon-master/ \
    && ./make.sh \
    && ./make.sh install

Download and install Fastdfs
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/fastdfs/archive/master.tar.gz -o fastfs.tar.gz \
    && tar zxf fastfs.tar.gz \
    && cd ${HOME}/fastdfs-master/ \
    && ./make.sh \
    && ./make.sh install

# configuration fastdfs
RUN cd /etc/fdfs/ \
    && cp storage.conf.sample storage.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/storage|g" /etc/fdfs/storage.conf \
    && cp tracker.conf.sample tracker.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/tracker|g" /etc/fdfs/tracker.conf \
    && cp client.conf.sample client.conf \
    && sed -i "s|/home/yuqing/fastdfs|/var/local/fdfs/storage|g" /etc/fdfs/client.conf 

# Download the nginx plugin
RUN cd ${HOME}/ \
    && curl -fSL  https://github.com/happyfish100/fastdfs-nginx-module/archive/master.tar.gz -o nginx-module.tar.gz \
    && tar zxf nginx-module.tar.gz

# download nginx
RUN cd ${HOME}/ \
    && curl -fSL http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o nginx-${NGINX_VERSION}.tar.gz \
    && tar zxf nginx-${NGINX_VERSION}.tar.gz

# Fix fatal bug where compilation could not find #include "common_define.h
ADD ./config ${HOME}/fastdfs-nginx-module-master/src

# Compile nginx and Fastdfs nginx plugins
RUN cd ${HOME} \
    && chmod u+x ${HOME}/fastdfs-nginx-module-master/src/config \
    && cd nginx-${NGINX_VERSION} \
    && ./configure --add-module=${HOME}/fastdfs-nginx-module-master/src \
    && make && make install

# Configure nginx and FastdFS
RUN cp ${HOME}/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs/ \
    && sed -i "s|^store_path0.*$|store_path0=/var/local/fdfs/storage|g" /etc/fdfs/mod_fastdfs.conf \
    && sed -i "s|^url_have_group_name=.*$|url_have_group_name=true|g" /etc/fdfs/mod_fastdfs.conf \
    && cd ${HOME}/fastdfs-master/conf/ \
    && cp http.conf mime.types anti-steal.jpg /etc/fdfs/ \
    && echo -e "events {\n\ worker_connections 1024; \n\ }\n\ http {\n\ include mime.types; \n\ default_type application/octet-stream; \n\ server {\n\ listen \$NGINX_PORT; \n\ server_name localhost; \n\ location ~ /group[0-9]/M00 {\n\ ngx_fastdfs_module; \n\ }\n\ }\n\ }">/usr/local/nginx/conf/nginx.conf

Clean up temporary software and files
RUN rm -rf ${HOME}/*
RUN apk del .mybuilds
RUN apk add bash pcre-dev zlib-dev

Create a startup script
RUN     echo -e "mkdir -p /var/local/fdfs/storage/data /var/local/fdfs/tracker; \n\ ln -s /var/local/fdfs/storage/data/ /var/local/fdfs/storage/data/M00; \n\n\ sed -i \"s/listen\ .*$/listen\ \$NGINX_PORT; /g\" /usr/local/nginx/conf/nginx.conf; \n\ sed -i \"s/http.server_port=.*$/http.server_port=\$NGINX_PORT/g\" /etc/fdfs/storage.conf; \n\ if [ \"\$HOST_IP\" = \"\" ]; then \n\ HOST_IP=\$(ifconfig \$NET_VAR | grep \"inet\" | grep -v \"inet6\" | awk '{print \$2}' | awk -F: '{print \$2}')\n\ fi \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/storage.conf;  \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/client.conf; \n\ sed -i \"s/^tracker_server=.*$/tracker_server=\$HOST_IP:\$FDFS_PORT/g\" /etc/fdfs/mod_fastdfs.conf; \n\ /etc/init.d/fdfs_trackerd start; \n\ /etc/init.d/fdfs_storaged start; \n\ /usr/local/nginx/sbin/nginx; \n\ tail -f /usr/local/nginx/logs/access.log \ ">/start.sh \
&& chmod u+x /start.sh

# Exposed port
EXPOSE ${NGINX_PORT} ${FDFS_PORT}

ENTRYPOINT ["/bin/bash"."/start.sh"]
Copy the code

File structure:

How do I view logs

Script one:

#! /bin/bash
STORAGE=/var/local/fdfs/storage/logs/storaged.log
TRACKER=/var/local/fdfs/tracker/logs/trackerd.log
NGINX=/usr/local/nginx/logs/access.log

ID=$(docker ps|grep fastdfs|awk '{print $1}')
USER_PRINT=The $1
LOG=""
if [[ "${USER_PRINT}" = "tracker"]].then
    LOG=${TRACKER}
elif [[ "${USER_PRINT}" = "storage"]].then
    LOG=${STORAGE}
else 
    LOG=${NGINX}
fi

docker exec -it $ID /usr/bin/tail -f ${LOG}
Copy the code

Use method (remember to assign execution rights to the script) :./log.sh tracker or storage or nginx

After the

No official account, no QR code, no ads, that’s all.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Use FastDFS to build file service Docker one click launch integration version – Running on the Shoulders of Giants series

What is FastDFS

How to use docker image

How to write dockerfile

How do I view logs

After the

Use FastDFS to build file service Docker one click launch integration version – Running on the Shoulders of Giants series

What is FastDFS

How to use docker image

How to write dockerfile

How do I view logs

After the

Related Posts

General technology and framework for front – end separation projects

JVM garbage collection algorithms and garbage collectors

Ali cloud OSS upload error: InvalidAccessKeyIdError | August more text challenge