Apisix source code parsing

Introduction to the

Apisix is a gateway controller written in Lua. Apisix is more like a controller than a gateway described in the official website. Because the code itself does not carry traffic.

Apache APISIX is a dynamic, real-time, high-performance API gateway.

Apisix runs on Top of OpenResty, which runs on top of Nginx.

The goal of OpenResty® is to make your Web services run directly within Nginx services, taking full advantage of Nginx’s non-blocking I/O model, Consistent high-performance responses not only to HTTP client requests, but also to remote backends such as MySQL, PostgreSQL, Memcached, and Redis.

In short, OpenResty extends Nginx sufficiently to introduce a number of hook points that you can implement using lua.

On the basis of OpenResty, Apisix implements many openResty hooks and realizes dynamic configuration of NGINx to cope with complex gateway requirements.

For example, the forwarding (load balancing) logic in Apisix actually uses lua-retsy-core/NGX /balancer

The directory structure

. ├ ─ ─ apisix# Apisix lua source│ ├ ─ ─ the admin# apisix admin api│ ├ ─ ─ balancer# upstream Load balancing│ ├ ─ ─ the cli# command line interface│ ├ ─ ─ control# control api│ ├ ─ ─ the coreSome configuration, ETCD interaction, request parsing, etc│ ├ ─ ─ the discoveryService discovery│ ├ ─ ─ HTTP# Route matching, etc│ ├ ─ ─ the plugins# Built-in plugins in this directory│ ├ ─ ─ SSL# SSL snI related│ ├ ─ ─ the stream# Stream processing related, includes some plugins for stream processing│ ├─ exercises │ ├─ Exercises ├─ Exercises# Develop apisix├ ─ ─ ci ├ ─ ─ the conf# Config file├── Example ├─ Kubernetes# kubernetes deployment file├ ─ ─ logos ├ ─ ─ rockspecLuarocks dependency description file├ ─ ─ t# test├ ─ ─ utils... └ ─ ─ a Makefile# make
Copy the code

Start the

Apisix uses Luarocks for package management. Because Lua is a scripting language, its execution requires the lua interpreter to interpret execution, and Lua also provides luajit just-in-time compilation.

Lua Luarocks must be installed when running Apisix for the first time, and Apisix depends on:

make deps
Copy the code

Make deps downloads all dependencies of Apisix to the deps in the current directory.

Up and running:

make run
Copy the code

The Makefile provides a quick start run with bin/apisix start, and apisix start internally calls Openresty.

When develop env.lua#L34, the Lua Package Loader is configured to look for dependencies in the deps folder. Lua # l46-l51

Using Make Run is very difficult to understand how Apisix really works. But apisix-Docker /alpine/Dockerfile shows how Apisix started and how it works with OpenResty.

bin/apisix

The apisix file is essentially a bash file. The internal logic to determine the current apisix file path is to look for the OpenResty path to determine the luajit location. Or use lua to start apisix. Lower-level calls to./apisix/cli/apisix.lua

#! /bin/bash.# use the luajit of openresty
    echo "$LUAJIT_BIN $APISIX_LUA$*. ""
    exec $LUAJIT_BIN $APISIX_LUA $*
elif [[ "$LUA_VERSION"= ~"The Lua 5.1"]].then
    OpenResty version is not 1.19, use Lua 5.1 by default
    echo "lua $APISIX_LUA$*. ""
    exec lua $APISIX_LUA$*...Copy the code

apisix/cli/apisix.lua

Apisix. lua contains the following commands:

Github.com/apache/apis…

Apisix init, which initializes the nginx configuration by readingconf/config.yamlGenerate the nginx config file. Used by OpenResty (nginx).
Apisix init_etcd initializes the ETCD configuration and is used to synchronize data with the ETCD.
Apisix start; Actually implementedopenresty -p /usr/local/apisix -g 'daemon off; 'Start openresty.

nginx.conf

Using make init will execute apisix init and generate the nginx configuration file. By reading the Nginx file, you can understand the apisix process.

The following nginx files extract key content from the generated nginx.conf file.

# Configuration File - Nginx Server Configs # This is a read-only file, do not try to modify it. # https://github.com/apache/apisix/blob/master/conf/config-default.yaml#L158-L175 # config.yaml The following section is the contents of the nginx_config.main_Configuration_snippet. # main configuration snippet starts # main configuration snippet ends http { ... # HTTP Configuration Snippet Starts # HTTP Configuration Snippet Ends https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md#http-subsystem # here is the NGX apisix balancer The implementation of the. The route selection of proxy bypass is implemented. Corresponds to the load balancing upstream function in Apisix. # apisix of load balancing in https://github.com/apache/apisix/tree/master/apisix/balancer, now provides 5 kinds of algorithms. Upstream apisix_Backend {server 0.0.0.1; Balancer_by_lua_block {# https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L709 apisix.http_balancer_phase() } ... } # https://openresty-reference.readthedocs.io/en/latest/Directives/#init_by_lua_block init_by_lua_block { require "Resty. Core "apisix = require("apisix") local dns_resolver = {"127.0.0.53", } local args = { dns_resolver = dns_resolver, } # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L61 apisix.http_init(args) } init_worker_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L90 apisix.http_init_worker() } exit_worker_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L137 Apisix.http_exit_worker ()} server {listen 127.0.0.1:9090; access_log off; location / { content_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L773 apisix.http_control() } } ... } server {listen 127.0.0.1:9091; access_log off; location / { content_by_lua_block { local prometheus = require("apisix.plugins.prometheus") prometheus.export_metrics() }} location = /apisix/nginx_status {allow 127.0.0.0/24; deny all; stub_status; } } server { listen 9080 default_server reuseport; listen 9443 ssl default_server http2 reuseport; listen [::]:9080 default_server reuseport; listen [::]:9443 ssl default_server http2 reuseport; server_name _; # apisix SSL server certificate configuration ssl_certificate cert/ ssl_place_holde.crt; ssl_certificate_key cert/ssl_PLACE_HOLDER.key; . # http server configuration snippet starts # http server configuration snippet ends location = /apisix/nginx_status { Allow 127.0.0.0/24; deny all; access_log off; stub_status; } location /apisix/admin { set $upstream_scheme 'http'; set $upstream_host $http_host; set $upstream_uri ''; Allow 127.0.0.0/24; deny all; content_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L752 apisix.http_admin() } } ssl_certificate_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L142 apisix.http_ssl_phase() } proxy_ssl_name $upstream_host; proxy_ssl_server_name on; location / { ... access_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L341 apisix.http_access_phase() } ... # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L709 proxy_pass $upstream_scheme://apisix_backend$upstream_uri; mirror /proxy_mirror; header_filter_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L575 apisix.http_header_filter_phase() } body_filter_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L612 apisix.http_body_filter_phase() } log_by_lua_block { # https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L670 Apisix.http_log_phase ()}} Location@grpc_pass {access_by_lua_block {# https://github.com/apache/apisix/blob/e7d26dc4f0bd690c288867a248a69f0efeaea733/apisix/init.lua#L553 apisix.grpc_access_phase() } grpc_set_header Content-Type application/grpc; grpc_socket_keepalive on; grpc_pass $upstream_scheme://apisix_backend; header_filter_by_lua_block { apisix.http_header_filter_phase() } body_filter_by_lua_block { apisix.http_body_filter_phase() } log_by_lua_block { apisix.http_log_phase() } } ... } # http end configuration snippet starts # http end configuration snippet ends }Copy the code

Processing phase & plug-in

You saw in the previous section what Apisix does by setting hooks at different stages in Nginx

The functions of Apisix are completed by different plug-ins. Apisix organizes these plug-ins (admin API) and calls them at different times according to the description of the plug-ins.

In the source code, all plug-ins have a unified entry.

function common_phase(phase_name)

Pass in the phase name at different stages to make the plug-in call for that phase. Plugin.run_plugin (Phase_name, nil, api_ctx) is called internally to run the plug-in.

For plug-ins, implement a function named phase to be called during that phase. For example:

Rewrite (conf, CTX); rewrite(conf, CTX);
Function _m. access(conf, CTX)

The known stages are:

preread
ssl
access
balancer
rewrite
header_filter
body_filter
log

Initialize the

Called when apisix starts:

Apisix init parses the configuration and generates the nginx.conf file
Apisix init_etcd, which initializes the storage directory in etCD. See apisix/the lua
Start the openresty (nginx)
Apisix.http_init (args), initialization related to nginx.
- Set the DNS resolver
- Start the “ring the agent”

Apisix.http_init_worker (), the core initialization logic of Apisix.

Initialize the OpenResty worker event

discovery.init_worker()

    local discovery = require("apisix.discovery.init").discovery
    if discovery and discovery.init_worker then
        discovery.init_worker()
    end
    require("apisix.balancer").init_worker()
    load_balancer = require("apisix.balancer")
    require("apisix.admin.init").init_worker()

    require("apisix.timers").init_worker()

    plugin.init_worker()
    router.http_init_worker()
    require("apisix.http.service").init_worker()
    plugin_config.init_worker()
    require("apisix.consumer").init_worker()

    if core.config= =require("apisix.core.config_yaml") then
        core.config.init_worker()
    end

    require("apisix.debug").init_worker()
    apisix_upstream.init_worker()
    require("apisix.plugins.ext-plugin.init").init_worker()

    local_conf = core.config.local_conf()

    if local_conf.apisix and local_conf.apisix.enable_server_tokens == false then
        ver_header = "APISIX"
    end
Copy the code

apisix.http_exit_worker()
- Stop ring “agent”

The request calls chain analysis

By looking at nginx.conf, we can analyze the processing flow of a request after it enters apisix.

Request to enter nginx
If the /apisix/admin path request is made, enter apisix.http_admin() and return after completion
Continue if it is a regular request (/)
The apisix.http_access_phase() phase is entered
- Initialize the API_CTX context
- Includes the CLIENT TLS authentication
- The request was matched with or without the registered path of Apisix, and the Radix Tree based Oenresty routing component was used internally for matching.
- Information such as the Apisix route and service matched by the request is injected into the context for subsequent use.
- Inject plug-ins related to the request into the context. For example, if the route corresponding to the request exists plug-ins, if the service corresponds to the request exists plug-ins defined by the service, and global plug-ins are added.
- Plug-ins that invoke the “rewrite” phase.
- Call the plug-in in the “Access” phase.
- Access to upstream
- Run the loadBalancer command to select a server
- Call the “balancer” phase plug-in
- Upstream: according to the upstream type, GRPC dubbo and others go to @grpc_pass @dubbo_pass and other follow-up processes. These configurations can be viewed in nginx.conf.
The apisix.http_balancer_phase() phase is entered
- This phase is the implementation of OpenResty Balancer. The main function of this phase is to perform the actual traffic forwarding configuration. Requests of HTTP GRPC and other types will pass through this phase
- If pick_server exists in the online document, that is, the server selected after loadbalancer is executed in the previous phase, configure nginx to directly forward the pick_server to this server. Note: NGINXproxy_passThe component actually performs the request forwarding function; Apisix only configurs it.
- If there is no pick_server, the loadBalancer is executed again
- Call openresty set_CURRENT_peer (server, CTX) to complete the proxy_pass configuration
- Perform proxy_pass
Enter the apisix.http_header_Filter_phase () phase, which mainly changes the response header
- Set head"Server", APISIX
- Set upstream status header:X-APISIX-Upstream-Status
- Execute the plug-in for the “header_Filter” stage
The apisix.http_body_filter_phase() phase is entered
- The plug-in that performs the body_filter phase
Enter apisix.http_log_phase()
- Execute the “log” phase of the plug-in
- Reclaim the API_CTX context

admin api

The Admin API is the control plane of Apisix, where the entire Apisix configuration is performed.

The entry is apisix/init.lua#L752, and routes configured in apisix/admin/init.lua#L375 are internally distributed.

Apisix /admin/init.lua#L44 distributes the request internally and routes the request to the corresponding module for processing.

Take the enable batch-requests plugin as an example:

The HTTP request: PUT http://127.0.0.1:9080/apisix/admin/plugin_metadata/batch-requests

Corresponding apisix/admin/plugin_metadata lua# L92

function _M.put(plugin_name, conf)
    local plugin_name, err = check_conf(plugin_name, conf)
    if not plugin_name then
        return 400, err
    end

    local key = "/plugin_metadata/". plugin_name core.log.info("key: ", key)
    local res, err = core.etcd.set(key, conf)
    if not res then
        core.log.error("failed to put plugin metadata[", key, "].", err)
        return 500, {error_msg = err}
    end

    return res.status, res.body
end
Copy the code

Write the plug-in name (key) and configuration (value) to the plugin_metadata directory in the ETCD

The plugin.plugin_metadata(plugin_name) is read when the plug-in is running.

control api

The control-API serves two purposes

It leads to the internal state
Controls the current behavior of Apisix

The listening address is 127.0.0.1:9090, which only allows local access because it contains sensitive data.

Entrance to apisix http_control ()

It executes registration of registered plug-ins that implement the control_api() method.

Take server_info as an example: apisix/plugins/server-info.lua#L194

function _M.control_api(a)
    return {
        {
            methods = {"GET"},
            uris ={"/v1/server_info"},
            handler = get_server_info,
        }
    }
end
Copy the code

The path /v1/server_info is registered on the Control API and specified to be processed using the get_server_info function.

Apisix source code parsing

Apisix source code parsing

Introduction to the

The directory structure

Start the

bin/apisix

apisix/cli/apisix.lua

nginx.conf

Processing phase & plug-in

Initialize the

The request calls chain analysis

admin api

control api

Related Posts

Interview Ali was asked, “How does your project handle repeated/concurrent requests?”

Linux system disk partition and mount – fdisk

02- Class-loading subsystems