Load Balancing is an open source feature of PaaS Rainbond, which is mainly implemented by the rainbond-entrance controller.

This article introduces rainbond-Entrance around the design architecture and implementation.

Why do you need load balancing

Rainbond internal network division supports multiple tenants. Each tenant has a private IP address segment, and the networks of different tenants are invisible to each other. When we deploy a containerized application to Rainbond, Rainbond assigns the container an internal IP for the communication of different applications within the cluster within the same tenant, which is not directly accessible outside the cluster. Therefore, we need to have a cluster entry controller so that users can easily access these applications.

In addition, each application deployed in Rainbond can have multiple instances, that is, if we deploy three instances of a WEB application, each of which shares a portion of the traffic, we need to add load balancing controllers in front of these three instances to distribute the traffic.

In addition to the above basic functions, a load balancing controller must support more functions, such as:

  • The entry controller can forward requests to specific applications based on packet information such as protocol, port number, host name, and so on
  • Detects application changes (such as adding custom domain names, certificates, and ports) in the cluster in real time and dynamically updates its own forwarding rules
  • It also supports HTTP, TLS, TCP, and UDP. Sometimes, not only WEB applications need to provide service, but also RPC and MySQL need to be opened to the outside world
  • High availability support

To sum up, we need a load balancing controller cluster that supports BOTH L4 and various application protocols (L7), and must be able to automatically detect application changes in the cluster to update its forwarding rules.

Load balancing in Rainbond

The overall architecture

  1. web: represents one application in Rainbond and has three instances
  2. api-server: Kubeneters kube-Apiserver component
  3. entrance: Common interface of Rainbond load balancing controller. Supports various load balancing plug-ins
Entrance to achieve

The load balancing in Rainbond is application-oriented. Different applications can use different load balancing. Rainbond’s Entrance component is designed to integrate multiple load balancing plug-ins, meaning that Rainbond supports not only the popular OpenResty but also other load balancing plug-ins, such as commercially available F5.

The main work of the Entrance is to monitor the changes of the information of application running nodes from Kube-Apiserver, such as Service, Endpoint, Pod, etc., and then abstract these resources into general load-balancing abstractions and cache them in ETCD. These general abstractions include:

  • Pool: Load balancing pool, which contains multiple nodes, corresponding to the three WEB instances in the preceding figure
  • Node: A node in the Pool, corresponding to one of the WEB instances in the figure above
  • Domain: domain name: The load balancing controller can identify the domain name in a data packet and forward the data to the corresponding Pool
  • VirtualService: a virtual host that listens on a port and specifies the port’s protocol name, which is used to handle L4 entry control and load balancing
  • Rule: forwarding rule, which describes the mapping between domain names and pools, specifies the protocol name and certificate information of ports, and handles L7 entry control and load balancing

When the resource changes, the Entrance converts the common resource to the resource of the corresponding plug-in and operates the load balancing controller based on the different plug-in drivers selected by the application.

As you can see from the architecture, there are two entrances and two OpenResty instances. Each Entrance holds all OpenResty addresses, and when there is information that needs to be updated, the Entrance will update the information to all OpenResty instances. How do the two entrances coordinate with each other? Here we make use of etCD’s own features to make distributed locks to ensure that only one Entrance has access to update information to OpenResty, thus making it highly available.

OpenResty plug-in

OpenResty is a WEB application that can use Lua scripts to process requests and tabular logic, and has many luA-related specifications and functions built in for developers. It is suitable for developing Restful API servers. We used OpenResty as a plug-in for the following reasons:

  1. Developed based on Nginx, it performs well in terms of stability and performance
  2. Close to the design goals of Rainbond, it has helped us compile the Lua module, making it easy to enrich the load balancing controller with Lua scripts
  3. Load balancing of L7 and L4 is supported

We have embedded a Rest API server on the OpenResty side that is written in Lua. As mentioned earlier, OpenResty integrates Lua scripting capabilities, so we can use Lua directly to handle requests. Here is part of the Nginx configuration file:

# custom api of upstream and server
server {
    listen 10002;

    location ~ /v1/upstreams/([-_0-9a-zA-Z.@]+) {
        set $src_name The $1;
        content_by_lua_file lua/upstream.lua;
    }

    location ~ /v1/servers/([-_0-9a-zA-Z.@]+) {
        set $src_name The $1; content_by_lua_file lua/server.lua; }}Copy the code

When we call the following API:

curl -s localhost:10002/v1/servers/app1 -X POST -d "$json_data"
Copy the code

Lua. As mentioned earlier, OpenResty has a lot of lua-related commands and functions built in to make Lua interact better with Nginx, so we’ll container handle the JSON data we receive in our scripts. Convert it to a configuration Nginx file, which is not posted due to Lua code. You can find the project address in the references section of this article.

One caveat here is that OpenResty needs to load configuration files frequently when receiving a large number of requests to modify the server and upstream, which can increase load and affect performance. In fact, there are many third-party plugins available for OpenResty. There is a plugin called dyups that can dynamically modify upstreams as follows:

Upstream: adds or updates upstream. Update ("upstream_name", [[server 127.0.0.1:8088;]]) -- delete(upstream)"upstream_name")
Copy the code

Once executed, it takes effect, so we don’t need to execute the nginx -s reload command, which makes it a little more efficient.

The changes to the server have not been dynamically modified by plug-ins, so in fact our load balancing controller has two cases. Updating the upstream configuration will take effect immediately, while updating the server configuration requires adding the nginx -s reload command.

conclusion

We use the Entrance plus OpenResty to achieve a pluggable and highly available load balancing controller. Overall, it is not complicated. I hope this article can give you some help.

So far we’ve split the Rainbond OpenResty plugin as a subproject and open source it on Github, which you can download and use separately: The Github address.

Citation and reference

  1. Openresty project
  2. Openresty has built-in Lua function usage documentation
  3. Dyups plug-in

Learn more about Rainbond V3.5.1

Rainbond is an application-centric open source PaaS that deeply integrates kubernetes-based container management, Service Mesh microservice architecture best practices, multi-type CI/CD application construction and delivery, multi-data center resource management and other technologies. To provide users with cloud native application life-cycle solutions, build an ecosystem of interconnection between applications and infrastructure, application to application, and infrastructure to meet the requirements of agile development, efficient operation, and lean management required to support rapid business development.

  • Website: https://www.rainbond.com
  • Try Rainbond public cloud: https://www.goodrain.com
  • Github: https://github.com/goodrain/rainbond
  • Yards cloud: https://gitee.com/rainbond/Rainbond
  • Wechat group: Add “Qingguo-wei” and accept invitations to join the group