I used to use Nginx simply as a WebServer, but recently I took a look at its plugins and found that there is a lot to play with, in particular that a lot of work can be done by installing and configuring plugins without writing code. This article takes setting up a document server as an example to demonstrate how to use the Nginx plug-in.

Description of project

This project tries to use several modules of Nginx to build a document server to realize file uploading and browsing. A pure Nginx implementation is intended to make file uploading an independent base module, which can be used directly if other business modules need file management functions.

Project address: github.com/javanan/tms…

Installing a plug-in

Configure Nginx

Environmental parameters

You need to add the following Settings to the nginx.conf file, using the env directive to specify the environment variables you want to use, and using the js_set directive to create and assign variables (there is no way to use environment variables directly in the configuration file).

env REDIS_KEY_UPLOAD_START_TIME;
env REDIS_KEY_UPLOAD_COUNTER;
env REDIS_CHANNEL_UPLOAD;
Copy the code

js_include /usr/local/nginx/njs/upload.js;
js_set $redis_key_upload_start_time var_redis_key_upload_start_time;
js_set $redis_key_upload_counter var_redis_key_upload_counter;
js_set $redis_channel_upload var_redis_channel_upload;
Copy the code

Access to the directory

We hope that we can through the Nginx direct access to a directory of files, for example: Nginx working directory files directory, in Nginx. Conf to add the following content:

location = /files/ {
  root .;
  autoindex on;
}
Copy the code

Upload a file

We use the nginx-upload-module plugin to upload files. After the installation is complete, add the following configuration to nginx.conf:

location /upload/ {
  # specify where to store the uploaded file
  upload_store /usr/local/nginx/files;

  Set the forwarding information
  upload_set_form_field $upload_field_name.name "$upload_file_name";
  upload_set_form_field $upload_field_name.content_type "$upload_content_type";
  upload_set_form_field $upload_field_name.path "$upload_tmp_path";
  upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
  upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";

  # Forward requests after file upload
  upload_pass @upload_response;
  # add_header "Content-Type" "text/html; charset=UTF-8";
  # echo "echo: Upload done";
}
Copy the code

The upload_pass command forwards basic information about a file to a specified address. The address that receives the uploaded file information is named location, which is not matched by a regular and is used for internal forwarding (the NJS module explains how to do this later).

The uploaded file will be placed at the location specified by the upload_store directive. The file is an ascending string of digits, such as 0000000001 0000000002. If no further action is required, you can run the Echo command to return the result without forwarding the uploaded file data.

Handle file

Since you can’t specify the name of the uploaded file (no extension), and Nginx will start naming the file from scratch every time you restart it, possibly overwriting existing files, it is not convenient to use, so you want to change the name of the uploaded file. Here we use the NJS module and add the following configuration to nginx.conf:

js_include /usr/local/nginx/njs/upload.js;
Copy the code

location @upload_response {
  js_content handle;
}
Copy the code

The file name consists of three parts: service startup time + number of files uploaded since startup + 1+ extension, for example, 20191224_124808_8.mp4. To achieve this requirement, the service startup time and file upload times need to be recorded. The simplest way to think about it is to make global variables, but there is no way to do that in Nginx. The idea was to write the data to a local file, but there would be concurrent reading problems, so Redis was used to save the data.

Access to Redis

Redis stores two data: 1. Startup time; 2. Counters. Each time a file is uploaded, a file name is created by combining the startup time and the counter. Get startup time with get command, get counter with incr command. Using the redis2-nginx-module module to turn nginx into a Redis client, we added the following configuration to nginx.conf:

location = /redis/counter {
  redis2_query get $redis_key_upload_start_time;
  redis2_query incr $redis_key_upload_counter;
  redis2_pass redis:6379;
}
Copy the code

In addition to storing shared information, Redis is also used for event notification. After each file is processed, Redis uses its publish and subscribe mechanism to send a message about the uploaded file, so that other systems can receive this event if they need to be extended. Add the following configuration to nginx.conf:

location = /redis/publish {
  set_unescape_uri $message $arg_message;
  redis2_query publish $redis_channel_upload $message;
  redis2_pass redis:6379;
}
Copy the code

The container is changed

For ease of use, we made the entire project into a Docker container. There are three containers: Nginx container, Redis container, and Redis-cli container, where Redis-cli is for setting initial values in Redis.

Set the time zone

The default time zone for alpine mirror is UTC, eight hours behind ours. As mentioned earlier, you need to have the service startup time as part of the upload file name, so you need to change the loss to Asia/Shanghai. Add the following content to the Dockerfile:

RUN sed -i 's? http://dl-cdn.alpinelinux.org/?https://mirrors.aliyun.com/?' /etc/apk/repositories && \
    apk add -U tzdata && \
    cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    apk del tzdata
Copy the code

Redis initial value

We need to record the startup time in Redis every time the service is started. The simplest way to think of is to run the following command with Redis – CLI after the service is started.

date +'%Y%m%d_%H%M%S' | xargs redis-cli -h redis set uploadBootAt
Copy the code

The environment variable

Specify the value of the environment variable in the docker-comemage. yml file, which can be overridden by the docker-comemage. overlay. yml file if desired.

environment:
  - REDIS_KEY_UPLOAD_BOOT_AT=upload:start_time
  - REDIS_KEY_UPLOAD_COUNTER=upload:counter
  - REDIS_CHANNEL_UPLOAD=upload:event
Copy the code

other

volumes:
  - ./nginx/nginx.conf:/usr/local/nginx/conf/nginx.conf:ro
  - ./nginx/html:/usr/local/nginx/html
  - ./nginx/njs:/usr/local/nginx/njs
  - ./upload:/usr/local/nginx/files
Copy the code

Here’s a lesson to share. I wanted to upload the file to TMP and rename it to the directory named volumes. However, this requires the file to be moved between the two devices, which is not allowed, so the file is directly uploaded to the specified directory and renamed on the spot.

Set the connection to Redis in nginx.conf.

location = /redis/counter {
  redis2_query get $redis_key_upload_start_time;
  redis2_query incr $redis_key_upload_counter;
  redis2_pass redis:6379;
}
Copy the code

This requires a connection between the two containers, but we don’t know what the address is beforehand. The solution is to specify it in docker-comemage.yml.

Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias was specified.

nginx:
  ...
  links:
      - redis
Copy the code

JS code description

Nginx. Conf file

env REDIS_KEY_UPLOAD_START_TIME;
Copy the code

js_set $redis_key_upload_start_time var_redis_key_upload_start_time;
Copy the code

njs/upload.js

//
function var_redis_key_upload_start_time(r) {
  return process.env.REDIS_KEY_UPLOAD_START_TIME;
}
Copy the code

The js_set directive solves the problem of assigning values to environment variables that cannot be referenced in the nginx.conf file.

r.subrequest("/redis/counter", { method: "GET" }, function(res) {
  ......
});
Copy the code

You can make a call with the subRequest method, but not with a named location.

var fs = require("fs");
fs.renameSync(oFileData.path, newpath);
Copy the code

The FS module enables simple file operations. However, you cannot use require to call your own modules.