Li Hang: Distributed storage Ceph introduction and principle architecture sharing

1. File File that the user needs to read and write. File – > Object mapping:

A. ino (metadata of File, unique ID of File).

B. ono(Serial number of an object generated by File segmentation, by default, a block size of 4M).

C. Oid (Object ID: ino + ono).

2. Object is the Object required by RADOS. Ceph assigns a static hash function to calculate the value of OID, maps OID to a pseudo-random value with approximately uniform distribution, and then gets PGID by phase with mask. Object – > PG mapping:

A) Hash (oid) & mask-> pgid

B) mask = total PG m(m is an integer power of 2)-1.

3. Placement Group (PG) is used to organize and map the storage of objects. (Similar to the concept of slot in Redis Cluster) A PG contains many objects. CRUSH algorithm is adopted to substitute pGID into it, and then a group of OSD is obtained. PG – > OSD mapping:

A) CRUSH (a pgid) – > (osd1, osd2, osd3).

2.4 Ceph IO pseudo-code process

1 locator = object_name
2
3 obj_hash =  hash(locator)
4
5 pg = obj_hash % num_pg
6
7 osds_for_pg = crush(pg)    # returns a list of osds
8
9 primary = osds_for_pg[0]
10
11 replicas = osds_for_pg[1:]Copy the code

The process of writing data to the OSD node is as follows:

• 1. In the form of LIBRbd, librBD is used to create a block device and write data to the block device.

• 2. The librados interface is called locally on the client, and then layer by layer mapping through pool, RBD, Object and PG. In the PG layer, the data can be stored on the three OSD nodes, which have a master-slave relationship.

• 3. The client establishes SOCKET communication with the Primay OSD node. The data to be written is sent to the Primary OSD node, which then sends the data to another replica OSD node.

2.7 Ceph Pool and PG distribution

Description:

• A. Pool is a logical partition used by CEPH to store data. It acts as a namespace.

• B. Each pool contains a configurable number of PGS.

• Objects in c. PG are mapped to different OSD nodes.

• D. The pool is distributed to the entire cluster.

• E. Pool can be used as a fault isolation domain and can be isolated according to different user scenarios.

2.8 Ceph data expansion PG distribution

Scenario data migration process:

A. Status Three OSD nodes and four PGS

B. Expand the capacity to four OSD nodes and four PGS

Status:

After the expansion:

Description:

Many PGS are distributed on each OSD node, and each PGS are automatically distributed on different OSD nodes. If the capacity is expanded, PGS will be migrated to the new OSD node to balance the number of PGS.

3. Ceph heartbeat mechanism

3.1 Heartbeat introduction

Heartbeat is used to check whether each node is faulty, so that the faulty node can be found in a timely manner and the corresponding troubleshooting process can be started.

Question:

A. Balance the fault detection time with the load caused by heartbeat packets.

B. If the heartbeat frequency is too high, too many heartbeat packets affect system performance.

C. If the heartbeat frequency is too low, the system availability is affected.

A fault detection strategy should be able to:

Timely: The cluster can detect node anomalies, such as downtime or network interruption, within an acceptable time range.

Appropriate pressure: including pressure on nodes, and pressure on the network.

Tolerate network jitter: The network is occasionally delayed.

Diffusion mechanism: The metamodel changes caused by the node status changes need to spread to the whole cluster through some mechanism.

3.2 Ceph heartbeat detection

The OSD node listens on public, Cluster, front, and back ports

· Public port: Listens for connections from Monitor and Client.

· Cluster port: listens for connections from OSD peers.

· Front port: a network adapter used by clients to connect to clusters, temporarily heartbeat between clusters.

· Back port: a network card for internal use of the customer cluster. Heartbeat is performed between clusters.

· HBClient: Messenger to send ping heartbeat.

3.3 Ceph OSD mutual heartbeat detection

Steps:

A. The OSD nodes in a PG heartbeat with each other and send PING and PONG messages to each other.

B. Check every 6s (actually a random time will be added to this basis to avoid the peak).

C. No heartbeat reply is detected at 20s and the system joins the Failure queue.

3.4 Ceph OSD and Mon heartbeat detection

OSD reports to Monitor:

• A. An event, such as a fault or PG change, occurs on the OSD.

• B. Within 5 seconds after starting itself.

• C. OSD periodically reports the OSD to Monito

• D. OSD Checks the failure information about the partner OSD in failure_queue.

• E. Send a failure report to the Monitor, add the failure information to the failure_pending queue, and remove it from the failure_queue.

• F. When receiving the heartbeat message from the OSD node in failure_queue or failure_pending, it is removed from the two queues and tells the Monitor to cancel the failure report.

• G. When reconnecting to the Monitor network occurs, the failure_pending error report is added back to the failure_queue and sent to the Monitor again.

• H. Monitor Collects statistics on the offline OSD

• I. Monitor collects partner failure reports from the OSD.

• J. If the faulty OSD node exceeds a certain threshold and enough OSD nodes report failures, take the OSD node offline.

3.5 Ceph heart detection summary

Ceph determines whether an OSD node fails by reporting the failed node to its partner OSD node or by monitoring the heartbeat traffic from the OSD node.

In a timely manner:

The partner OSD node detects node failures at the second level and reports to the Monitor. The Monitor takes the failed OSD node offline in a few minutes.

Appropriate pressure:

Because of the partner OSD reporting mechanism, the heartbeat statistics between the OSD and the Monitor are used as an insurance measure. Therefore, the interval for the OSD to send heartbeat messages to the Monitor can be as long as 600 seconds, and the monitoring threshold can be as long as 900 seconds. Ceph actually distributes the pressure of the central node during fault detection to all OSD nodes, so as to improve the reliability of the central node Monitor and further improve the scalability of the whole cluster.

Tolerate network jitter:

After receiving the report from the OSD node to its partner OSD node, the Monitor does not immediately take the target OSD node offline. Instead, the Monitor periodically waits for the following conditions:

1. The failure time of the target OSD node is greater than the threshold dynamically determined by a fixed amount of OSd_heartbeat_GRACE and historical network conditions.

2. Reports from different hosts reach MON_osd_min_down_reporters.

3. The failure report is not canceled by the source OSD before the preceding two conditions are met.

Diffusion:

The Monitor, as the central node, does not broadcast notifications to all OSD nodes and clients after OSDMap updates. Instead, the Monitor lazily waits for OSD nodes and clients to obtain OSDMap updates. This reduces Monitor stress and simplifies interaction logic.

Ceph Communication Framework

4.1 Ceph communication framework categories introduction

There are three different implementations of the network communication framework:

Simple thread mode

Features: For each network connection, two threads are created, one for receiving and one for sending.

Disadvantages: A large number of links can generate a large number of threads, which can consume CPU resources and affect performance.

I/O multiplexing mode of Async events

Features: This is the current network communication widely used in the way. Version K already uses Asnyc by default.

XIO is implemented using accelio, an open source network communication library

Features: This method relies on third-party library Accelio stability, currently in the experimental stage.

4.2 Ceph communication framework design model

Subscribe/Publish design pattern:

The subscription publishing pattern, also known as the Observer pattern, is intended to “define a one-to-many dependency between objects,

When the state of an object changes, all objects that depend on it are notified and automatically updated.

4.3 Ceph communication framework flow chart

Steps:

• A. Accepter calls SimpleMessenger::add_accept_pipe() to create a Pipe to SimpleMessenger:: Pipes to process the request.

• B. Pipe is used to read and send messages. This class has two main components, Pipe::Reader and Pipe::Writer, which handle message reading and sending.

• C. Messenger acts as the publisher of the message and each Dispatcher subclass acts as the subscriber of the message. After receiving the message, Messenger reads the message through Pipe and forwards it to Dispatcher for processing.

• D. Dispatcher is the subscriber’s base class. The subscription backend inherits this class and registers with Messenger:: Dispatcher via Messenger::add_dispatcher_tail/head. Received the message.

• E. DispatchQueue This class is used to cache incoming messages and wake up the DispatchQueue::dispatch_thread thread to find the dispatch_thread on the back end to process the messages.

4.4 Ceph communication framework map

4.5 Ceph communication data format

Communication protocol format requires both parties to agree on the data format.

The content of the message is divided into three parts:

· Header // Message header, the envelope of the type message

· User data // Actual data to be sent

O Payload // Operation saves metadata

O Middle // Reserved field

O data // Reads and writes data

O footer // The end of the message

1 class Message : public RefCountedObject { 2 protected: 3 ceph_msg_header header; // Message header 4 ceph_msg_footer footer; // Message bottom 5 //"front" unaligned blob
6    bufferlist       middle;   // "middle" unaligned blob
7    bufferlist       data;     // data payload (page-alignment will be preserved where possible)
8 
9  /* recv_stamp is setwhen the Messenger starts reading the 10 * Message off the wire */ 11 utime_t recv_stamp; /* dispatch_stamp is the time stamp from which data is receivedsetwhen the Messenger starts calling dispatch() on 13 * its endpoints */ 14 utime_t dispatch_stamp; /* throttle_stamp is the point atwhichwe got throttle */ 16 utime_t throttle_stamp; // Get timestamp 17 /* time at for slot throttlewhich message was fully read*/ 18 utime_t recv_complete_stamp; 20 ConnectionRef connection; // Uint32_t magic = 0; // list_member_hook<> dispatch_q; // Boost :: Intrusive; 26 27 struct ceph_msg_header { 28 __le64 seq; // The unique sequence number of the message in the current session 29 __le64 tid; // The globally unique id of the message is 30 __le16type; // Message type 31 __le16 priority; // Priority 32 __le16 version; // version number 33 34 __le32 front_len; // The length of the payload 35 __le32 middle_len; // The length of middle is 36 __le32 data_len; // The length of data is 37 __le16 data_off; Struct ceph_entity_name SRC; struct ceph_entity_name SRC; // Oldest code we think can decode this.unknownifzero. */ 43 __le16 compat_version; 44 __le16 reserved; 45 __le32 crc; /* header crc32c */ 46 } __attribute__ ((packed)); 47 48 struct ceph_msg_footer { 49 __le32 front_crc, middle_crc, data_crc; // CRC check code 50 __le64 sig; // The message's 64-bit signature 51 __u8 flags; // End flag 52} __attribute__ ((Packed));Copy the code

5. Ceph CRUSH algorithm

5.1 Data distribution algorithm challenges

Data distribution and load balancing:

1. Data is evenly distributed to each node.

2. Load balancing to balance the load of data access and read operations among nodes and disks.

Flexible cluster scaling:

1. The system can easily add or delete node devices and handle node failures.

2. After nodes are added or deleted, data is automatically balanced and data is migrated as little as possible.

Support for large-scale clusters:

1. The metadata maintained by the data distribution algorithm is relatively small, and the calculation amount should not be too large. With the increase of cluster size, the overhead of data distribution algorithm is relatively small.

5.2 Ceph CRUSH algorithm description

CRUSH algorithm: Controlled Scalable Decentralized Placement of Replicated Data

The mapping process algorithm from PG to OSD is called CRUSH algorithm. An Object needs to be stored in three osd nodes.

CRUSH algorithm is a pseudo-random process. It can randomly select an OSD set from all OSD nodes. However, the random selection result of a PG is unchanged, that is, the mapped OSD set is fixed.

5.3 Ceph CRUSH algorithm principle

CRUSH algorithm factor:

Hierarchical Cluster Map

Indicates the physical topology of the storage system. Defines the hierarchical static topology of an OSD cluster. The OSD hierarchy enables the CRUSH algorithm to realize rack-awareness when selecting OSD. In other words, replicas can be distributed on different racks and in different equipment rooms through rule definition to ensure data security.

Placement Rules

Rules that determine how copies of objects for a PG are selected. These rules allow users to customize the distribution of copies in the cluster.

● 5.3.1 Hierarchical Cluster Map

CRUSH Map is a tree structure. OSDMap records OSDMap attributes (epoch/ FSID /pool information, OSD IP address, etc.).

The leaf node is a device (also known as OSD), and the other nodes are called buckets. These buckets are imaginary nodes that can be abstracted based on the physical structure. Of course, the tree structure has only one final root node, which is called root node. The intermediate virtual bucket nodes can be data center abstract, machine room abstract, rack abstract, and host abstract.

● 5.3.2 Data distribution strategy Placement Rules

The main features of data distribution strategy Placement Rules are:

1. Search for a node in the CRUSH Map

2. Use the node as the fault isolation domain

3. Search mode of location copy (breadth-first or depth-first)

1 rule replicated_ruleset  Pool name = pool name = pool name = pool name
2 
3 {
4
5    ruleset 0                #rules specifies the number of the set6, 7type replicated          Define pool type as replicated(and erasure schema)
8
9     min_size 1                The minimum number of specified replicas in the pool cannot be less than 1
10 
11    max_size 10               The maximum number of replicas specified in the pool cannot be greater than 10
12 
13    step take default         The bucket entry point is usually a root bucket
14 
15    step chooseleaf  firstn  0  type  host Select a host and recursively select the osd node
16
17    step emit        # end18 19}Copy the code