The reason for adding “native” is that our development based on BFE has changed.

Routing and forwarding is the core function of BFE as a seven-layer traffic proxy service. BFE designs a routing and forwarding model that supports multi-tenant and multi-machine rooms.

Photo source:Inside The BFE)

The basic concept

In the native routing and forwarding function, BFE explains the concepts of tenant, cluster, subset group, and instance as follows:

  • Tenant: Product, different business lines/different product lines;
  • Cluster: a product will have multiple micro-services, for example, a mall product may have user micro-services, order micro-services, inventory micro-services, each micro-service is a cluster;
  • Subgroups: Usually a cluster has only one subcluster, unless there are multiple data centers;
  • Instance: Each subset can have multiple back-end service instances. “IP + port” is an instance, and there are as many instances as there are nodes deployed in a microservice.

Related configuration files

  • Host_rule: configures the mapping between tenants and labels and between labels and domain names to obtain labels based on domain names and determine tenants based on labels.
  • Vip_rule: Configures the mapping between IP addresses and tenants to determine tenants based on their IP addresses.
  • Route_rule: Configures routing rules, that is, how requests are routed to the cluster.
  • Cluster_rule: Common cluster configuration, such as heartbeat detection interface, request back-end timeout, maximum retry times, etc.
  • GSLB: Configure global load balancing, that is, configure the weight of subsets in a cluster to select subsets in a cluster for load balancing.
  • Cluster_table: Cluster -> subset -> instance profile;

Routing and forwarding process

Routing related process: Determine tenant -> Determine cluster -> Determine subgroup -> Determine instance

Forwarding related process: Get forwarders -> Forward requests

  • Determine the tenant

    • 1. According to the requested host (domain name), find the tenant mapped to the host from the host_rule configuration.
    • 2. If host cannot be mapped to a tenant, or if host is an IP address, find the tenant mapped to the IP address from the vip_rule configuration file.

  • To determine the cluster

    • 1. Obtain all routing rules of the tenant based on the tenant name from route_rule configuration.
    • 2. According to the rule configuration sequence, traverse the routing rules and obtain the first routing rule matching the current request (the matching logic is the same as that of traffic limiting).
    • 3. Obtain the cluster name configured with the matching routing rule.

  • Obtain transponder

    • 1. Locate the cluster_rule configuration file based on the cluster name.
    • 2. Obtain or create forwarders based on the cluster configuration.

  • Determine the subset group

    • 1. According to the cluster name, find all subgroups of the cluster from the GSLB configuration file, and select a subcluster according to the weight of the subgroup load balancing;
    • 2. Obtain all instances of this subset from the Cluster_TABLE configuration file.

  • To determine the instance

    • 1. Select an instance from the subset group according to the weighted load balancing;

  • Forward the request

    • 1. Use a forwarder, pass in the back-end address, and create connections or obtain idle connections according to the back-end address and request protocol;
    • 2. Use connections to forward requests;

Note: The steps to find/retrieve the XX configuration from the XX configuration file are only described for ease of understanding. In fact, BFE has loaded all the configuration files into memory at startup.

Case analysis

ShoppMall product line, the domain name is ShopPMall.com, and OrderService is deployed in Cn-IDC0, with three instances.

Related configurations are as follows:

host_rule:

{"Version": "1.0.0", "DefaultProduct": null, "Hosts": {// Corresponding to HostTags configuration "ShoppMallTag":["shoppmall.com"]}, "HostTags": {"ShoppMall":[// tag arbitrary "ShoppMallTag"]}}Copy the code

route_rule:

{"Version": "1.0.0", "ProductRule": {"ShoppMall": [{// requests starting with /order are routed to OrderService microservice "Cond": "req_path_prefix_in("/order")", "ClusterName": "OrderService" } ] } }Copy the code

cluster_rule:

{"Version": "init Version", "Config": {// OrderService Basic configuration of the cluster "OrderService": {// Request backend related: connection timeout, etc. "BackendConf": { "TimeoutConnSrv": 2000, "TimeoutResponseHeader": 50000, "MaxIdleConnsPerHost": 0, "RetryLevel": 0}, // Heartbeat check configuration "CheckConf": {"Schem": "HTTP ", "Uri": "/ healthCheck ", "Host": "example.org", "StatusCode": 200, "FailNum": 10, "CheckInterval": 1000}, // Global load balancing policy configuration "GslbBasic": {"CrossRetry": 0, "RetryMax": 2, "HashConf": {"HashStrategy": 0, "HashHeader": "Cookie:UID", "SessionSticky": false}}, // Client read-write related "ClusterBasic": { "TimeoutReadClient": 30000, "TimeoutWriteClient": 60000, "TimeoutReadClientAgain": 30000, "ReqWriteBufferSize": 512, "ReqFlushInterval": 0, "ResFlushInterval": -1, "CancelOnClientClose": false } } } }Copy the code

gslb:

{" Clusters ": {/ / order a subset of the group of cluster configuration" OrderService ": {" GSLB_BLACKHOLE" : 0, "CN - idc0" : 100}}, "Hostname" : ""," Ts ":" 0 "}Copy the code

cluster_table:

{" Version ":" 1.0.0 ", "Config" : {/ / cluster "OrderService" : {/ / the subset of "CN - idc0" : [{/ / instance "Addr" : "127.0.0.1", "Name" : "Example_hostname", "Port" : 8180, the "Weight" : 10}, {" Addr ":" 127.0.0.1 ", "Name" : "example_hostname", "Port" : 8181, "Weight" : 10}, {" Addr ":" 127.0.0.1 ", "Name" : "example_hostname", "Port" : 8182, the "Weight" : 10}]}}Copy the code

The client sends the POST request http://shoppmall.com/order/createOrder routing process is as follows:

  • 1. Route to tenant: ShoppMall according to host:shoppmall.com;
  • Obtain all routing rules of ShoppMall tenant, path “/order/createOrder” matches cluster OrderService;
  • 3. Obtain GSLB configuration according to the cluster and route to the subset group as CN-IDc0;
  • 4. Obtain three back-end instances: 127.0.0.1:8180, 127.0.0.1:8181, 127.0.0.1:8182 according to cluster: OrderService, subset: CN-idc0;
  • 5. Select one of the three instances for load balancing.

conclusion

In my opinion, there are too many routing configuration files in BFE, which are not intuitive and a little convoluted. If tenant configuration is not differentiated, a file will have a lot of configuration, which is not easy to manage.

Perhaps because of open source, BFE does not want to rely on other third party services such as configuration center, database, etc., so it has changed to using configuration file configuration, or was originally designed to use file configuration. This, like nginx, requires frequent configuration file changes.