This is the 11th day of my participation in the August More Text Challenge

One, foreword

Back-end services generally provide file upload functions, which are faced with the following scenarios:

  1. Distributed scenario: Multiple services can be deployed to improve availability
  2. Microservices scenario: Services are broken down and calls between services become more complex

Basic user operations are as follows:

  1. upload
  2. download

Then the following analysis is based on two scenarios.


(1) Distributed scenario

Benefits of distributed scenarios:

  1. Multiple units can be deployed to improve availability
  2. Share the load of a single server to access more requests

It is assumed that the uploaded files are stored on the server where the file service resides:

  • File upload: Uploading a file service (stored on a server)

  • File download: To find the uploaded file, visit the specific file service (host server)

Of course, IP Hash can be used to meet the two requirements, as shown in the following figure:

But I think each file service is the same and can be scaled horizontally so that distributed storage can be introduced, as shown in the figure below:


(2) Micro-service scenario

In order to provide the same functions and unified management, microservices are introduced, and each service deals with a single function.

Now there is A requirement that users upload and download files with the authorization of service A to access the file service.

There are three ways to address this requirement:

  1. Brute force: copy, direct access to distributed services

  2. Inter-service access: Users directly access service A, which then invokes the file service

  3. Client direct: The client uploads directly to the file service

  4. Brute force: copy, direct access to distributed services

This way, it breaks the microservices unbundling principle.

  1. Inter-service access: Users directly access service A, which then invokes the file service

This approach is simple and files are also invoked through inter-service access. However, general interservice access, such as Dubbo and Feign, is only applicable to tabloid articles. Feign, for example, has a BUG passing files and stores them in memory.

  1. Client direct: The client uploads directly to the file service

This way can be divided:

  1. Client authorization is obtainedtoken
  2. The client relies ontokenUpload a file
  3. The application service gets the file service callback




Distributed storage

Currently, common distributed storage devices include FastDFS, Ceph, HBase, Go-FastDFS, and MiniO.

Distributed storage has the following features:

  1. Metadata storage
  2. Data redundancy
  3. Data validation
  4. Data compression
  5. Data encryption
  6. Data to heavy

A third component is involved, increasing the complexity of service invocation but solving the distributed access scaling problem.

We won’t explore how distributed storage components implement these features here, assuming it is available. Just focus on how it is invoked at the application level.

Full callback retransmission mechanism: