0 Album Overview

Etcd is an important foundation component in the cloud native architecture and is incubated and hosted by CNCF. Etcd can be used not only as service registration and discovery, but also as middleware for key-value storage in microservices and Kubernates clusters.

“Thoroughly Understand ETCD series of articles” will be from the basic function of ETCD practice, API interface, implementation principle, source code analysis, as well as the implementation of the pit experience and other aspects of the specific introduction of ETCD. It is expected that there will be about 20 articles, I will continue to update every week, welcome to pay attention to.

1 the Compact method

The Compact method compresses the event history in the ETCD key-value pair store. The key-value pair store should be compressed periodically, otherwise the event history will continue to grow indefinitely.

rpc Compact(CompactionRequest) returns (CompactionResponse) {}
Copy the code

The message body of the request is the CompactionRequest, which compresses the key-value pairs stored into the given revision. All keys that are smaller than the compressed revision will be removed:

Message CompactionRequest {// Revised version of the key-value store for comparison int64 Revision = 1; bool physical = 2; }Copy the code

When Physical is set to true, RPC will wait until the compression is physically applied to the local database, at which point the compressed item will be removed from the back-end database entirely. The body of the reply message, CompactionResponse, is defined as:

message CompactionResponse {
  ResponseHeader header = 1;
}
Copy the code

CompactionResponse has only one generic response header.

2 Watch service

The Watch API provides an event-based interface for asynchronously monitoring key changes. The ETCD3 monitor continuously monitors key changes from a given revision (current or historical) and streams key updates back to the client.

The event

Each key change is represented by an event message. Event messages provide both update data and update type. The mvccpb.Event message body is defined as follows:

message Event { enum EventType { PUT = 0; DELETE = 1; } EventType type = 1; KeyValue kv = 2; // prev_kv holds the key-value pair before the event. }Copy the code

Type is the event type. If the type is PUT, the new data has been stored in the key. If the type is DELETE, the key has been deleted.

Kv holds the KeyValue for the event. The PUT event contains the current KV key pair. The PUT event of kv.version =1 indicates the creation of the key. The DELETE/EXPIRE event contains the deleted key whose modified revised version is set to the deleted revised version.

Monitor the flow

The Watch API provides an event-based interface for asynchronously monitoring key changes. The ETCD monitor waits for key changes by continuously monitoring from a given revised version (current or historical) and streams the key updates back to the client.

Monitoring runs continuously and uses gRPC to stream event data. Monitoring flows are bidirectional, with clients writing to the stream to establish monitoring events and reading to receive monitoring events. A single monitor flow can duplicate many different observations by marking events with each observer identifier. This multiplexing helps reduce memory footprint and connection overhead on etCD clusters.

The Watch event has the following three features:

  • Ordered, events are ordered in the order of revision; If the event precedes the published event, it will never appear on the watch.
  • Reliable, event sequences never discard any event subsequences; If there are three events a < B < C in chronological order, the Watch can be guaranteed to receive B if it receives events A and C.
  • Atom, ensuring that the list of events contains a complete revision; Updates made with multiple keys in the same revision are not split into multiple event lists.

Watch the service definition

In rpc.proto, Watch service is defined as follows:

service Watch {
  rpc Watch(stream WatchRequest) returns (stream WatchResponse) {}
}
Copy the code

Watch watches events that are about to happen or have already happened. Inputs and outputs are streams; The input stream is used to create and cancel observations, while the output stream sends events. An observation RPC can be observed on multiple key ranges at once and fluidize events for multiple observations. The entire event history can be observed from the last compressed revision. The WatchService has only one Watch method.

The WatchRequest message body of the request is defined as follows:

message WatchRequest { oneof request_union { WatchCreateRequest create_request = 1; WatchCancelRequest cancel_request = 2; }}Copy the code

Request_union is either a request to create a new observer or a request to cancel an existing observer. The request to create a new observer WatchCreateRequest:

Message WatchCreateRequest {// Key is the key to register to watch. Bytes Key = 1; bytes range_end = 2; // start_revision is an optional revision to start (including) the observation. If start_revision is not set, the value is now. Int64 start_revision = 3; bool progress_notify = 4; Enum FilterType {// Filter out the PUT event NOPUT = 0; NODELETE = 1; } // Filter to filter out events before the server sends them back to the observer. repeated FilterType filters = 5; // If prev_kv is set, the created observer gets the previous KV before the event occurs. // If the previous KV has been compressed, nothing is returned bool prev_kv = 6; }Copy the code

If range_end is not set, only the key argument will be observed; if range_end is equal to ‘\0’, All keys greater than or equal to the parameter key are observed; if range_end is 1 greater than the given key, all keys prefixed with the given key are observed.

The progress_notify setting so that if no recent event has occurred, the ETCD server will periodically send a WatchResponse without any event to the new observer. Useful when the client wants to restore disconnected observers from the last known revision. The ETCD server will determine how often it sends notifications based on the current load.

Cancel the WatchCancelRequest for an existing observer:

message WatchCancelRequest {
  int64 watch_id = 1;
}
Copy the code

Watch_id is the ID of the observer to cancel so that no more events are propagated. The body of the reply message WatchResponse:

message WatchResponse { ResponseHeader header = 1; // watch_id is the ID of the observer associated with the response int64 watch_id = 2; bool created = 3; bool canceled = 4; int64 compact_revision = 5; String cancel_reason = 6; // cancel_reason specifies a reason to cancel the observer. repeated mvccpb.Event events = 11; }Copy the code

Created is set to true if the reply is used to create the observer request. The client should record the Watch_id and expect to receive events for the created observer from the same stream. All events sent to the created observer will carry the same watch_id; Canceled is set to true if the reply is used to cancel the observer request. No more events are sent to the cancelled observer.

Compact_revision is set to a minimum index if an observer tries to view the compressed index. Occurs when an observer is created on a compressed revision or the observer is unable to keep up with the progress of the key-value pair store. The client should treat the observer as cancelled and should not attempt to recreate any observer with the same start_Revision.

3 summary

This paper mainly introduces the KEY-value compression and Watch API involved in Etcd API, which are two commonly used function APIS provided externally. Understanding the key-value compression and Watch API is very helpful for us to better use Etcd.

Subscribe to the latest articles, welcome to follow my official account

Recommended reading

  1. Comparison between ETCD and other K-V components such as Zookeeper and Consul
  2. Thoroughly understand ETCD series article (1) : first know ETCD
  3. Thoroughly understand etCD series article (2) : ETCD installation posture
  4. Thoroughly understand etCD series (3) : ETCD cluster operation and maintenance deployment
  5. Thoroughly understand etCD series article (four) : ETCD security
  6. Thoroughly understand the etCD series article (five) : the use of ETCDCTL
  7. Thoroughly understand etCD series article (6) : ETCD core API V3
  8. Etcd gRPC service API

] (blueskykong.com/2020/08/27/…). 9. Thoroughly understand the ETCD series (8) : ETCD transaction API

reference

etcd docs