This is the 30th day of my participation in the August Text Challenge.More challenges in August
preface
Today I encountered an error when calling Minio’s shard merge interface, which bothered me all day. Because I always look for reasons on myself, I once suspected that there was something wrong with my logic processing, and I thought that the most likely reason was that MY parameters were passed wrong. However, it never occurred to me that the problem was Minio servicing itself.
The body of the
Without further ado, let’s look at the merge error message:
Your proposed upload is smaller than the minimum allowed object size
Error screenshot:
Talk about the background of the project
Each terminal (including the Web terminal, PC terminal, mobile APP, and vehicle terminal) slices the uploaded video files and sends them to the storage gateway, which then dumps them to the Minio object store. The reason for this problem is that the fragment size uploaded by each terminal is 1MB or 2MB, which has been fixed, but the minimum fragment size required by Minio object storage is 5MB. Therefore, when merging all fragments of a file, Your proposed upload is smaller than the minimum allowed object size is always prompted.
The solution
After identifying the merge error caused by this, I quickly sorted out all possible solutions, and altogether, there were three possible solutions.
Solution 1: Modify the fragment size synchronously on each end
The obvious method is to change the fragment size to 5MB for all terminals. However, this method has two disadvantages: first, all terminals need to synchronize the fragment size, which involves a large range. 2. The object storage service is not flexible enough. When the fragment size is fixed, incompatibility may occur.
Scheme 2. Storage gateway reconstructs sharding
The second solution is also easy to think of, which is to temporarily store the requests for small fragments and forward them to Minio storage service when the cache size reaches 5MB. What we need to worry about is the alignment of the size fragments. Possible problems: In multi-instance deployment, some shards may not be joined together.
Solution 3: Modify the Minio fragment size limit
The ultimate solution, directly modify Minio source code, adjust the fragment size limit, and then recompile.
The fragment size is written dead in the Minio source code as follows:
globalMinPartSize = 5 * humanize.MiByte
The modification method is also very simple, I am currently directly changed to 1MB.
At the end
Ok, the problem of merging Minio shards is finally solved. In fact, Minio source code has a lot of experience can be used for reference, welcome to taste fresh. My name is Liuzhen007, China Bond, a code-tapping Bond, welcome to comment and add “one key three connect”.
