Etcd Gateway and GRPC-gateway

Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

Etcd Gateway pattern: Build a portal to an ETCD cluster

The ETCD gateway is a simple TCP proxy that forwards network data to an ETCD cluster. The gateway is stateless and transparent, does not check client requests or interfere with cluster responses, supports multiple ETCD server instances, and adopts a simple circular strategy. The ETCD gateway routes requests to the available endpoints and hides faults from the client so that the client does not perceive faults on the server side. Other access strategies, such as weighted polling, may be supported later.

When to use etCD gateway mode

When we use a client to connect to an ETCD server, each application accessing etCD must know the address of the cluster instance of etCD to be accessed, the address used to provide the client service: ETCD_LISTEN_CLIENT_URLS.

If multiple applications on the same server access the same ETCD cluster, each application still needs to know the client endpoint address of the broadcast of the ETCD cluster. If the ETCD cluster is reconfigured to have different endpoints, each application will also need to update its list of endpoints. In a large cluster environment, reconfiguration is both repetitive and error-prone.

All of the above problems can be solved by using the ETCD gateway as a stable local endpoint, so that client applications are not aware of changes to cluster instances. A typical ETCD gateway configuration is such that each computer running the gateway listens at a local address, and each ETCD application connects to the corresponding local gateway. When a change occurs to an ETCD cluster instance, the gateway only needs to update its endpoint, not the code implementation of each client application.

Of course, not all scenarios are suitable for etCD gateway, such as the following two scenarios.

Performance improvement

Etcd gateway is not designed to improve etCD cluster performance. It does not provide caching, Watch stream merging, or bulk processing. The ETCD team is currently developing a caching proxy aimed at improving cluster scalability.

Run the management system on the cluster

Advanced cluster management systems like Kubernetes themselves support service discovery. Applications can access the ETCD cluster using the system’s default DNS name or virtual IP address. For example, kube-Proxy, which provides intra-cluster Service discovery and load balancing for services, functions as an ETCD gateway.

In summary, to automatically propagate cluster endpoint changes, the ETCD gateway runs on each machine, providing multiple applications with access to the same ETCD cluster service.

Etcd Gateway pattern practice

The following is a real combat exercise based on etCD gateway mode, with the following environment:

Start the ETcd gateway to proxy these static endpoints with the etcd gateway command:

$etcd gateway start - endpoints = http://192.168.10.7:2379, http://192.168.10.8:2379, http://192.168.10.9:2379 # response results as follows: {" level ", "info", "ts" : 1607794339.7171252, "caller" : "tcpproxy/pulls. Go: 90", "MSG" : "ready to proxy client Requests, "" endpoints" : [" 192.168.10.7:2379 ", "192.168.10.8:2379", "192.168.10.9:2379"]}Copy the code

Note that — endpoints are comma-separated lists of etCD server targets used to forward client connections. The default value is 127.0.0.1:2379. Configurations similar to the following cannot be used:

- endpoints = https://127.0.0.1:2379Copy the code

Because gateways do not determine TLS.

Other common configurations are as follows:

• – Listen-addr Specifies the interface and port bound to accept client requests. The default value is 127.0.0.1:23790.
• – Retry-delay Delay in connecting to the failed endpoint. The default value is 1m0s.Note that the value is followed by units, similarly123Is not valid.An invalid parameter appears on the command line.

invalid argument "123" for "--retry-delay" flag: time: missing unit in duration 123
Copy the code

• — insecure-discovery accepts SRV records that are insecure or vulnerable to man-in-the-middle attacks. The default is false.
• – Trusted -ca-file is the path to the TLS CA file on the CLIENT of the ETCD cluster, which is used to authenticate endpoints.

In addition to statically specifying an endpoint, you can also use DNS to discover services and set DNS SRV entries. You can try it yourself based on the CLUSTER of DNS discovery patterns we explained earlier.

Grpc-gateway: provides HTTP interfaces for non-GRPC clients

Etcd V3 uses gRPC as the messaging protocol. Etcd project includes Go client based on gRPC and command line tool ETCDctl. The client communicates with ETCD cluster through gRPC framework. For client languages that do not support gRPC, ETCD provides JSON grPC-gateway, and provides RESTful proxies through grPC-gateway. Converts HTTP/JSON requests to gRPC Protocol Buffer format messages.

Note here that the JSON object in the HTTP request body contains key and value fields defined as byte arrays, so the content must be processed in the JSON object using Base64 encoding. For convenience, I use curl to make an HTTP request. Other HTTP/JSON clients (e.g., browsers, Postman, etc.) can do this.

Specifies the mapping between etCD versions and grPC-gateway interfaces

The interface path provided by grpc-gateway has been changed since ETCD V3.3:

• Etcd V3.2 and earlier can only use the [client-url]/v3alpha/* interface;
• Etcd v3.3 uses client-url /v3alpha/*;
• Etcd v3.4 uses client-url /v3beta/ and deprecates [client-url]/v3alpha/;
• Etcd V3.5 uses client-URL /v3beta/ only.

The interface path provided by grPC-gateway is different from the etCD version of the V3 API, so you need to pay attention to the etCD version you are using.

Below we will carry out key-value read and write, watch, transaction and security authentication practices based on the GRPC-gateway interface provided by ETCD.

Key-value read/write operations

Keys /v3/kv/range /v3/kv/put /v3/kv/put We write the key value object to etcd:

$ curl -L http://localhost:2379/v3/kv/put \ -X POST -d '{"key": "Zm9v", "value": "YmFy"}' #  {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"16","raft_term":"9"}}Copy the code

As you can see, we successfully wrote a key-value pair with a key of Zm9v and a value of YmFy via the HTTP request. The key-value pair is base64 encoded, and the actual key-value pair is written to foo:bar.

/v3/kv/range interface to read the key-value pair we just wrote:

$ curl -L http://localhost:2379/v3/kv/range \ -X POST -d '{"key": "Zm9v"}' #  {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"16","raft_term":"9"},"kvs ":[{"key":"Zm9v","create_revision":"13","mod_revision":"16","version":"4","value":"YmFy"}],"count":"1"}Copy the code

Through the range interface, the corresponding value of “Zm9v” can be obtained, which is completely in line with our expectations.

When we want to get a key-value pair prefixed with the specified value, we can use the following request:

$ curl -L http://localhost:2379/v3/kv/range \ -X POST -d '{"key": "Zm9v", "range_end": "Zm9w"}'  {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"16","raft_term":"9"},"kvs ":[{"key":"Zm9v","create_revision":"13","mod_revision":"16","version":"4","value":"YmFy"}],"count":"1"}Copy the code

We specified in the request that the key be in the range zM9V-ZM9W. The result returns only one key-value pair, as expected. Through the interfaces /v3/kv/range and /v3/kv/ PUT, we can easily read and write key-value pairs.

Watch the key value

Watch for key-value pairs is also a feature often used in ETCD. Etcd provides /v3/watch interface to monitor keys, let’s watch just write “Zm9v”, request as follows:

$ curl -N http://localhost:2379/v3/watch \ -X POST -d '{"create_request": {"key":"Zm9v"}}' & #  {"result":{"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"1","raft_term": "2"},"created":true}}Copy the code

We create a request with a monitor key of Zm9v, and the ETCD server returns the result of a successful creation.

Another request is made to update the key value as follows:

$ curl -L http://localhost:2379/v3/kv/put \

  -X POST -d '{"key": "Zm9v", "value": "YmFy"}' >/dev/null 2>&1
Copy the code

In the execution page of the watch request, we can see the following result:

When the key is written, the monitor event is triggered and the console outputs the details of the time. The HTTP request client establishes a long connection with the ETCD server and notifies the client of the event when the monitored key/value pair changes.

Implementation of ETCD transactions

Transactions are used to complete a set of operations by comparing the specified conditions, executing the corresponding operation in the case of success, and rolling back otherwise. Grpc-gateway provides an API interface to initiate a transaction through /v3/ KV/TXN interface.

We first compare the created version of the specified key-value pair and, if successful, perform the update operation.

To get the build version, we query the key-value pair before executing:

# of query key/value pair version $curl - http://localhost:2379/v3/kv/range - X POST - L d '{" key ": "Zm9v"}' # response result {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"20","raft_term":"9"},"kvs" Zm9v: [{" key ":" ", "create_revision" : "13", "mod_revision" : "20", "version" : "8", "value" : "YmFy}]," count ":" 1 "} # affairs, Comparison of the specified key/value pair to create version $curl - L http://localhost:2379/v3/kv/txn POST \ \ - X - d '{"compare":[{"target":"CREATE","key":"Zm9v","createRevision":"13"}],"success":[{"requestPut":{"key":"Zm9v","value":"YmF Y "}}]}' # {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"20","raft_term":"9"},"succ eeded":true,"responses":[{"response_put":{"header":{"revision":"20"}}}]}Copy the code

Compare is a list of assertions with multiple federated conditions. In this case, if the createRevision value is 13, the condition is met and the transaction can be successfully executed.

Here is a transaction that compares the version of the specified key-value pair. The HTTP request is implemented as follows:

# affairs, Comparison of the specified key/value pair version $curl - L http://localhost:2379/v3/kv/txn POST \ \ - X - d '{"compare":[{"version":"8","result":"EQUAL","target":"VERSION","key":"Zm9v"}],"success":[{"requestRange":{"key":"Zm9v"} }]}' # {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"6","raft_term":"3"},"succe eded":true,"responses":[{"response_range":{"header":{"revision":"6"},"kvs":[{"key":"Zm9v","create_revision":"2","mod_rev ision":"6","version":"4","value":"YmF6"}],"count":"1"}}]}Copy the code

The above command gets the key value of the specified VERSION. You can see that the enumeration value of target in Compare is VERSION. It is found that the version of Zm9v is indeed 8, so the query result is executed and the correct value YmF6 for Zm9v is returned.