Minio high performance Kubernetes native object storage

[toc]

MinIO is an object storage service based on the Apache License V2.0 open source protocol. Compatible with Amazon S3 cloud storage service interface, it is very suitable for storing large amount of unstructured data, such as pictures, videos, log files, backup data and container/VM images, and an object file can be any size, ranging from a few KB to a maximum of 5 TB.

MinIO is a very lightweight service that can be easily combined with other applications such as NodeJS, Redis or MySQL.

The characteristics of

  • High performance MiniO is the world’s fastest object storage (min.io/)
  • Elastic capacity expansion You can flexibly expand a cluster
  • Natural cloud native services
  • Open source and free, most suitable for enterprise customization
  • S3 Fact standard
  • Simple and powerful
  • Storage mechanism (Minio uses erasure code and checksum to protect data from hardware failures and silent data corruption. You can recover data even if you lose half (N/2) of your hard drives.

The installation

Minio consists of a server and a client. The server is deployed through minio, and the client is just a binary command (MC), through which you can operate the object storage (add, delete, search, etc.). Of course, MINIo also provides SDK in various languages

The server can be installed in independent single-machine deployment mode or distributed installation mode. The distributed installation mode is similar to the single-machine installation mode, but varies according to parameter transfer

stand-alone

  • Docker container installation
docker pull minio/minio     
docker run -p 9000:9000 minio/minio server /data     
Copy the code
  • macOS
brew install minio/stable/minio     
minio server /data     
Copy the code
  • Linux
wget https://dl.min.io/server/minio/release/linux-amd64/minio     
chmod +x minio     
./minio server /data     
Copy the code
  • Windows
wget https://dl.min.io/server/minio/release/windows-amd64/minio.exe     
minio.exe server D:\Photos     
Copy the code

distributed

Distributed Benefits Distributed Minio lets you combine multiple hard disks (even on different machines) into an object storage service. Because hard disks are distributed on different nodes, distributed Minio avoids single points of failure

Data protection Distributed Minio uses erasure codes to protect against multiple node outages and bit rot. Distributed Minio requires at least four hard disks. The distributed Minio automatically introduces erasure codes.

The high-availability standalone Minio service has a single point of failure. Conversely, if you have a distributed Minio with N disks, as long as N/2 disks are online, your data is safe. But you need at least N/2+1 hard drive to create new objects. For example, a 16-node Minio cluster with 16 disks per node would still be readable even if eight servers went down, but you would need nine servers to write data. Note that you can combine different nodes and several hard disks per node, as long as you adhere to distributed Minio’s limitations. For example, you could have two nodes with four disks per node, or you could have four nodes with two disks per node, or something like that.

Consistency Minio In distributed and single-machine mode, all read and write operations strictly follow the Read-after-write consistency model.

Erasure codes Minio uses erasure codes and checksum to protect data from hardware failures and silent data damage. Even if you lose half the number of hard drives (N/2), you can still recover data.

What is Erasure code? Erasure code is a mathematical algorithm for recovering lost and damaged data. Minio uses Reed-Solomon code to split objects into N/2 data and N/2 parity blocks. This means that if you have 12 disks, an object will be divided into 6 data blocks and 6 parity blocks. You can lose any 6 disks (whether they are stored as data blocks or parity blocks) and still recover data from the rest of the disks.

Why are erasure codes useful? The working principle of erasure correcting codes is different from RAID or replication. For example, RAID6 can not lose data even when two disks are lost, while Minio can ensure data security even when half of disks are lost. In addition, the Minio erasure correction code is applied to the object level and can restore one object at a time, while RAID is applied to the volume level and data recovery takes a long time. Minio encodes each object individually. Once storage services are deployed, there is usually no need to replace disks or repair them. Minio erasure codes are designed for performance and use hardware acceleration wherever possible.

What is bit-decay bit ROT protection? Bit decay, also known as Data Corruption or Silent Data Corruption, is a serious Data loss problem of hard disk Data. Data on hard drives can be corrupted without anyone noticing, and there is no error log. Is the so-called open enemy is easier to hide, hidden enemies are harder to defend, this kind of secretly made mistakes than the hard disk directly kaka down is also dangerous. But fear not, Minio erasure codes employ a high-speed HighwayHash based checksum to protect against bit decay.

Distributed deployment :GNU/Linux and macOS Example 1: Start the distributed Minio instance on eight nodes with one disk on each node. Run the following command on all eight nodes.

export MINIO_ACCESS_KEY=<ACCESS_KEY>     
exportMINIO_SECRET_KEY = < SECRET_KEY > minio server http://192.168.1.11/export1 http://192.168.1.12/export2 \     
               http://192.168.1.13/export3 http://192.168.1.14/export4 \     
               http://192.168.1.15/export5 http://192.168.1.16/export6 \     
               http://192.168.1.17/export7 http://192.168.1.18/export8     
     
Copy the code

Distributed deployment: Kebernetes

# Helm install by Googlehelm install minio --set mode=distributed,numberOfNodes=4,imagePullPolicy=IfNotPresent,accessKey=v9rwqYzXXim6KJKeyPm344,secretKey=0aIRBu9KU7gAN0l uoX8uBE1eKWNPDgMnkVqbPC,service.type=NodePort,service.nodePort=25557 googleapis/minio -n veleroCheck the pods status after the installation. If the Pods are in the READY state, the installation is successful
kubectl get pods -n velero -o wide     

If the Pods are not in the READY state, check logs
kubectl logs minio-0 -n velero     

If the disk is in a waiting state, restart the PODS to checkKubectl delete pods -n velero minio-{0,1,2,3}The default is cluser. For convenience, I use nodeport
Copy the code


As shown above, when I created a distributed MiniO with four nodes, I used the default PVC to create the storage. By default, each node creates 1 10GB storage (can be customized).

Client MC installation and use

The installation

wget https://dl.min.io/client/mc/release/linux-amd64/mc     
chmod +x mc     
./mc --help     
Copy the code

MC Command Guide

Ls Lists files and folders. MB Creates a bucket or folder. Cat displays file and object contents. Pipe redirects a STDIN to an object or file or STDOUT. Share generates the URL for sharing. Cp Copies files and objects. Mirror Mirrors buckets and folders. Find finds files based on parameters. Diff compares differences between two folders or buckets. Rm deleted files and objects. Events Notifications of managed objects. Watch listens for events on files and objects. Policy Manages access policies. Session Indicates the session saved by cp command management. Config Manages MC configuration files. Update Checks software updates. Version Displays the version information.Copy the code

MC command practice

Check the minio server configuration
mc config host ls     
     
Add minio server configuration
mc config host add minio  http://minio.vaicheche.com:25555  v9rwqYzXXim6KJKeyPm344 0aIRBu9KU7gAN0luoX8uBE1eKWNPDgMnkVqbPC --api s3v4     
     
# Check the minio bucket
mc ls minio     
     
# create bucket
mc mb minio/backup     
     
# upload local directory (file without r)
mc cp -r  ingress minio/backup/     
     
# download remote directory (file without r)
mc cp -r  minio/backup .     
     
# mirror a local folder to minio(similar to rsync)
mc mirror localdir/ minio/backup/     
     
# keep listening on local folder mirroring to minio(similar to rsync)
mc mirror -w localdir/ minio/backup/     
     
Continually find all JPEG images from miniO buckets and copy them to miniO "Play /bucket" buckets
mc find minio/bucket --name "*.jpg" --watch --exec "mc cp {} play/bucket"     
     
# delete directory
mc rm minio/backup/ingress  --recursive --force     
     
# delete file
mc rm minio/backup/service_minio.yaml     
     
Delete all incomplete uploaded objects from mybucket
mc rm  --incomplete --recursive --force play/mybucket     
     
Delete objects created 7 days ago
mc rm --force --older-than=7 play/mybucket/oldsongs     
     
Dump MySQL database file to minio
mysqldump -u root -p ******* db | mc pipe minio/backups/backup.sql     
     
# directing a backup
mongodump -h mongo-server1 -p 27017 -d blog-data --archive | mc pipe minio1/mongobkp/backups/mongo-blog-data-`date +%Y-%m-%d`.archive     
     
     
     
Copy the code

Optimization practice of MINIO in K8S

As in the k8S practice of MINIo, in my practice environment. After I installed the distributed system through helm, I adopted NFS as storeagesclasses by default. A total of 4 nodes were created and 4 PVCS were automatically created. After I deleted the data of 1 PVC, miniO could still read and write normally and the data could still exist. Refer to the below


One of the biggest problems is that if you are using a self-built shared storage like NFS, minio can keep data secure even with four nodes. But you only have one NFS disk, in case your NFS goes down and the disk is damaged, your data is lost. So to keep the data safe. You are advised to use hostPath to save data on each node. This way, if the node goes down and the disk is corrupted, your data will not be lost. Data is also read and written faster using local nodes. Of course you need additional management nodes for local storage.

Minio in K8ShostPathThe deployment of practice

Environment description: 5 node K8S environment, use 4 nodes as MINO, and use node host network

#1. Label 4 of the nodes because I want to deploy minio on nodes labeled miniO-server =true
kubectl get node --show-labels=true     
kubectl label nodes node-hostname1  minio-server=true     
kubectl label nodes node-hostname2  minio-server=true     
kubectl label nodes node-hostname3  minio-server=true     
kubectl label nodes node-hostname3  minio-server=true     
     
#2. Add hosts to each host. If your hostname resolves automatically, do not change it
echo "host1 [IP1] >> /etc/hosts"     
echo "host2 [IP2] >> /etc/hosts"     
echo "host3 [IP3] >> /etc/hosts"     
echo "host4 [IP4] >> /etc/hosts"     
     
# 3. Create a namespace
You can also use other custom namespaces, but you need to modify the following YAML files
kubectl create ns velero     
     
#4. Download Headless, Daemonset, Service
wget https://download.osichina.net/tools/k8s/yaml/minio/minio-distributed-headless-service.yaml     
wget https://download.osichina.net/tools/k8s/yaml/minio/minio-distributed-daemonset.yaml     
wget https://download.osichina.net/tools/k8s/yaml/minio/minio-distributed-service.yaml     
     
#5. Modify and create corresponding Service and DaemonsetYaml 'hostPath: specifies that you need to use the node local path MINIO_ACCESS_KEY and MINIO_SECRET_KEY: Define your secret key. For security, change the url to host name after args: startup parameter: http://host{1... 4}/data/minio 'minio-distributed-service.yaml' is an external service, which is ClusterIP by default and can be accessed with ingress or nodePort-f minio-distributed-statefulset.yaml      
kubectl create -f minio-distributed-daemonset.yaml     
kubectl create -f minio-distributed-service.yaml     
     
Copy the code

The practice process is as follows: