0 Album Overview

Etcd is an important foundation component in the cloud native architecture and is incubated and hosted by CNCF. Etcd in microservices and Kubernates clusters can be registered with discovery not only as a service, but also as a middleware for key-value storage.

“Thoroughly Understand ETCD series of articles” will be from the basic function of ETCD practice, API interface, implementation principle, source code analysis, as well as the implementation of the pit experience and other aspects of the specific introduction of ETCD. It is expected that there will be about 20 articles, I will continue to update every week, welcome to pay attention to.

Introduction to etcd

Etcd is an open source project launched by the CoreOS team in June 2013. Its goal is to build a highly available distributed key-value database. It has the following characteristics:

  • Simple: It is simple to install and configure, and it provides an HTTP API for interaction and is simple to use
  • Key-value pair storage: Storing data in hierarchical directories, as in a standard file system
  • Monitor changes: Monitor specific keys or directories for changes and react to changes in values
  • Security: Supports SSL certificate authentication
  • Fast: A single instance supports 2k+ reads per second, according to official benchmark data
  • Reliability: Raft algorithm is adopted to achieve the availability and consistency of distributed system data

Written in the Go language, ETCD has excellent cross-platform support, small binaries, and a strong community. Communication between ETCD machines is handled by Raft algorithm.

Etcd is a highly consistent distributed key-value store that provides a reliable way to store data that needs to be accessed by distributed systems or clusters of machines. It gracefully handles leader elections during network partitions in case of machine failures, even when the Leader node fails.

Any complex application, from simple Web applications to Kubernetes clusters, can read data from or write data to ETCD.

Etcd officially joined the Cloud Native Computing Foundation (CNCF) in December 2018 and is supported by CNCF. CNCF is a vendor-neutral foundation and a leader in the promotion and adoption of cloud native technologies.

2 Application Scenarios

The most common use of ETCD is for service registration and discovery. In addition, it can be used for key-value pair storage, where applications can read and write data from ETCD.

A simple use case is to store database connection details or functional flags in etCD as key-value pairs. We can observe these values so that our application can reconfigure itself as it changes. Advanced use is to use etCD’s consistency guarantee to enforce a database leader election or to perform distributed locking across a group of followers.

2.1 Key-value pair storage

A highly-available key value store for shared configuration and service discovery. A key-value storage system for configuring sharing and service discovery.

After all, ETCD is a key and value store component, and other applications are based on its key and value store capabilities. Etcd storage has the following features:

  • Kv type data storage, generally faster than the relational database.
  • Supports both dynamic storage (memory) and static storage (disk).
  • Distributed storage that can be integrated into a multi-node cluster.
  • Storage mode, using a similar directory structure.
    • Only leaf nodes can actually store data, the equivalent of files.
    • The parent of a leaf node must be a directory, which cannot store data.

The latency of the ETCD leader is the most important metric to track, and the built-in dashboard has a view dedicated to this. In our tests, severe delays caused instability within the cluster because Raft was only as fast as the slowest machine on most machines. We can mitigate this problem by tweaking the cluster appropriately. Etcd has been pretuned on cloud providers with highly variable networks.

2.2 Service Registration and Discovery

Service Registration and Discovery addresses one of the most common problems in distributed systems: how processes or services in the same distributed cluster can find each other and establish connections. Essentially, service discovery is about knowing if a process in the cluster is listening on UDP or TCP ports and can be found and linked by name.

Solving the problem of service discovery requires three pillars, one of which is indispensable.

  • Highly consistent and highly available service storage directories. Etcd, based on the Raft algorithm, is naturally a highly consistent and highly available directory for storing services.

  • A mechanism for registering services and service health. Users can register services on the ETCD, configure key TTL for the registered services, and maintain the heartbeat of the services periodically to monitor the health status.

  • A mechanism for finding and connecting services. Services registered under the topic specified by etCD can be found under the corresponding topic. To ensure connectivity, we can deploy a Proxy mode ETCD on each service machine to ensure that services accessing the ETCD cluster are connected to each other.

The ETCD/RAFT library, introduced in ETCD 2, is one of the most stable and feature-rich open source conformance protocols available. As the core data replication engine for etCD, TiKV, CockcorachDB, Dgraph and other well-known distributed databases, ETCD/RAFT drives over 100,000 clusters and is one of the most widely adopted consistency protocol implementations. The multi-version control and transaction functions introduced in ETCD3 greatly simplify the development process of distributed applications and improve efficiency and stability. After five years of evolution, ETCD has also become the default storage option for various container orchestration systems. Kubernetes is a popular container platform, and Kubernetes clusters running in any environment rely on ETCD to provide stable and reliable storage services.

2.3 Message publishing and subscription

In distributed systems, one of the most suitable communication methods between components is message publishing and subscription. This is to build a configuration sharing center where data providers publish messages, and message consumers subscribe to topics they care about and notify subscribers in real time when a topic has a message published. In this way, centralized management and dynamic update of distributed system configuration can be achieved.

In distributed search services, the meta information of the index and the node state of the server cluster machines are stored in etCD for subscription by individual clients. Use etCD’s key TTL feature to ensure that machine state is updated in real time.

Distributed log collection system. The core of the system is to collect logs distributed across different machines. Collectors typically assign collection units by application (or topic), so you can create a directory P on ETCD named after the application (topic), store all the machine IP addresses of the application (topic) as subdirectories on directory P, and then set up an ETCD recursive Watcher. Recursively monitors changes to all information in the application (topic) directory. This enables the collector to be notified in real time when the machine IP (message) changes to adjust the assignment.

A situation in which information in a system requires dynamic automatic retrieval and manual intervention to modify the content of the information request. This is usually done by exposing an interface, such as a JMX interface, to get some runtime information. With the introduction of ETCD, instead of implementing a solution of your own, you can simply store the information in designated ETCD directories, which can be accessed externally via an HTTP interface.

2.4 Distributed Notification and coordination

The concept of distributed notification and coordination is similar to publishing and subscribing to messages. In distributed systems, one of the most suitable communication methods between components is message publishing and subscription. This is to build a configuration sharing center where data providers publish messages, and message consumers subscribe to topics they care about and notify subscribers in real time when a topic has a message published. In this way, centralized management and dynamic update of distributed system configuration can be achieved.

Watcher mechanism in ETCD is used here to realize notification and coordination between different systems in distributed environment through registration and asynchronous notification mechanism, so as to realize real-time processing of data changes. This is usually implemented like this: Different systems register for the same directory on etCD and set Watcher to watch for changes in that directory (recursive mode can be set if changes in subdirectories are needed). When a system updates the directory on ETCD, the system with Watcher will be notified and react accordingly.

Low coupling heartbeat detection via ETCD. The detecting system is associated with the detected system through a directory on etCD rather than directly, which can greatly reduce the coupling of the system.

The system is scheduled using etCD. A system is composed of a console and a push system. The responsibility of the console is to control the push system to carry out corresponding push work. Some of the actions the administrator makes on the console actually change the status of some directory nodes on etCD, and etCD notifies the push clients that have signed up for Watcher’s push system about these changes, and the push system makes corresponding push tasks.

Complete work report through ETCD. Most similar task distribution systems register a temporary working directory on etCD after a subtask is started and report its progress regularly (writing progress to this temporary directory) so that the task manager can know the progress of the task in real time.

2.5 Distributed Lock

In distributed systems, where there is only one copy (or a limit) of the data, locks are used to control the number of processes that modify the data at any given time. The lock in single-machine mode not only needs to ensure that the process is visible, but also needs to consider the network between the process and the lock in distributed environment.

Distributed locks can store tokens in memory, but the memory is not allocated by a process but in common memory such as Redis and Memcache. As for the use of databases, files, and so on to do the lock and stand-alone implementation is the same, as long as the mark can be mutually exclusive on the line.

Because ETCD uses the Raft algorithm to keep data consistent, the values stored in the cluster for an operation must be globally consistent, so distributed locking is easy to implement. The lock service can be used in two ways, either to remain exclusive or to control timing.

Retention exclusivity means that all users who acquire a lock end up with only one. Etcd provides a set of apis for implementing distributed lock atom operation CAS (CompareAndSwap). Setting prevExist ensures that if multiple nodes attempt to create a directory at the same time, only one of them succeeds. The successful user is considered to have acquired the lock.

Control timing, that is, all users who want to acquire locks are scheduled to do so, but the order in which they are acquired is also globally unique and determines the order in which they are executed. Etcd also provides a set of apis (automatic creation of ordered keys) to specify a POST action for a directory, so that ETCD will automatically generate a current maximum value in the directory as a key and store the new value (client number). You can also use the API to list all the keys in the current directory in order. The values of these keys are the timing of the client, and the values stored in these keys can be numbers representing the client.

3. Summary

This chapter introduces concepts related to ETCD and its application scenarios. Etcd is a powerful tool in distributed environments and has a wide range of applications for consistent storage. The next article will cover etCD installation and usage practices.

Quality book recommendation

Welcome to buy my book, now published:

Original is not easy, I hope you support, looking forward to exchange and learning with you.

Subscribe to the latest articles, welcome to follow my official account

reference

etcd