This article mainly introduces how to build Kubernetes Ingress log analysis platform based on log service, and provides some simple hands-on experiments to facilitate you to quickly understand the log service related functions.

Deploy the Ingress logging scheme

  1. Log in to the Container Services Management Console.
  2. Save the CRD configuration above as a choreographer template. For details about the Choreographer template documentation, see Creating a Choreographer Template.
  3. To create an application based on this template, select the default namespace of your cluster.
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # your config name, must be unique in you k8s cluster
  name: k8s-nginx-ingress
spec:
  # logstore name to upload log
  logstore: nginx-ingress
  # product code, only for k8s nginx ingress
  productCode: k8s-nginx-ingress
  # logtail config detail
  logtailConfig:
    inputType: plugin
    # logtail config name, should be same with [metadata.name]
    configName: k8s-nginx-ingress
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: nginx-ingress-controller
            Stderr: false
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            Keys:
            - client_ip
            - x_forward_for
            - remote_user
            - time
            - method
            - url
            - version
            - status
            - body_bytes_sent
            - http_referer
            - http_user_agent
            - request_length
            - request_time
            - proxy_upstream_name
            - upstream_addr
            - upstream_response_length
            - upstream_response_time
            - upstream_status
            - req_id
            - host
            NoKeyError: true
            NoMatchError: true
            Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"] +)"\s(\d+)\s(\d+)\s"([^"] *)"\s"([^"] *)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S*).* SourceKey: contentCopy the code







Verify the Ingress logging scheme

  1. Log in to the Container Services console.
  2. On the routing page, click Ingress Overview. If the Ingress Overview page is displayed, the Ingress log scheme is created successfully.






Deploy the Ingress logging script

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-ingress
  namespace: default
  labels:
    app: mock-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mock-ingress
  template:
    metadata:
      labels:
        app: mock-ingress
    spec:
      containers:
      - name: mock-ingress
        image: registry.cn-shanghai.aliyuncs.com/log-service/mock-ingress:latest
        command: ["/data/replay_log"]
        args:
          - '-ecs=true'
          - '-project=k8s-log-{cluster-id}'
          - '-endpoint=cn-shanghai.log.aliyuncs.com'
          - '-source=/data/data.txt'
          - '-dist=/data/distribute.csv'Copy the code


  1. Log in to the Container Services console, copy the cluster ID from the cluster list and save it to Notepad.
  2. Replace {cluster-id} in the configuration template with the actual cluster ID.
  3. Use the configuration template to create a stateless application.



View Ingress detailed reports

  1. Log in to the log service console.
  2. Click Project management in the navigation tree on the left, select the log Project configured during cluster creation, and click the log Project name to go to the log Project page (the default Project name is k8S-log- {cluster-id}).
  3. On the Project details page, the log library page is displayed by default. The nginx-Ingress log library (LogStore) stores all ingress access logs. Click the dashboard in the navigation bar on the left to enter the dashboard list and view the analysis reports of all Ingress.

Set the alarm



  1. Open the Ingress Monitoring center and click Create Alarm on the menu bar in the upper right corner of the Request Success rate chart.
  2. In the alarm dialog box that is displayed, enter the alarm name and select the query range and execution range
  3. Trigger condition is set as:total < 100, that is, the alarm is triggered if the value is not 100%. (In actual scenarios, the value can be smaller.)
  4. Click “Next”
  5. In the notification type 】 【 select 】 【 nailing robot, fill in nailing robot oapi.dingtalk.com/robot/send? WebHook address… (For details about the WebHook address, see Obtaining a Customized robot.) Alarm Settings are complete.

Subscribing to periodic reports

In addition to alarm notification, the log service supports the report subscription function. With this function, you can periodically render reports into pictures and send them by email or group. The following example configates subscriptions for the Ingress Overview to send reports to specified spike groups per minute:

  1. Go to the Ingress Overview report and click the Subscribe button in the upper right corner of the report.
  2. On the configuration page that is displayed, set frequency to CronJob and enter0/5 * * * *(Reports are sent every 5 minutes. In actual scenarios, the time can be set to a fixed time every day.Add a watermarkOptions.
  3. Select from the notification typeNailing robot, fill in the WebHook address of the nail robothttps://oapi.dingtalk.com/robot/send?access_token=a2c6d68cdb67dbe6ad38361b91f24bf669a551ccb6c4ceef9d6a0f8ced044676(See WebHook addressCustom robotGet) to complete the subscription.

Advanced features

Custom analysis of Ingress logs

  1. Log in to the log service console.
  2. Click Project management in the navigation tree on the left, select the log Project configured during cluster creation, and click the log Project name to go to the log Project page (the default Project name is k8S-log- {cluster-id}).
  3. On the Project details page, the log library page is displayed by default. Click the [Query] link on the right of the log library (LogStore) named nginx-ingress to enter the query page.

The log service supports various query and analysis functions. For details, see Log query and analysis. You can customize query and analysis on the query page, for example:

  1. Query requests whose status code is greater than 200:status > 200
  2. Query requests whose host (domain name) is inner.njkj.com and whose response latency is greater than 30ms:Host: inner.njkj.com and request_time > 0.03
  3. Top10 services whose status code is not 200:not status : 200 | SELECT proxy_upstream_name, count(1) as total group by proxy_upstream_name order by total desc limit 10
  4. Statistics of UV distribution in each province:* | select ip_to_province(x_forward_for) as province, approx_distinct(x_forward_for) as total group by province order by total desc limit 1000
  5. The average access latency for the last day with host (domain name) = inner.njkj.com compared to the same day yesterday and the same day last week:host : inner.njkj.com | select d[1] as "Today", d[2] as "Yesterday", d[1] - d[2] / d[2] as "Yesterday Delta", d[3] as "Last Week", d[1] - d[3] / d[3] as "Last Week Delta" from( select compare(val, 86400, 604800 ) as d from(select avg(request_time) as val from log ) )

Ingress reports are based on the analysis function of the log service. You can search for the SQL of the corresponding function in the default Ingress report provided by the system. Click the menu option in the upper right corner of the corresponding chart and choose view Analysis Details to jump to the corresponding query page.







Query deployment operations from audit logs

  1. Log in to the log service console.
  2. Click Project management in the navigation tree on the left, select the log Project configured during cluster creation, and click the log Project name to go to the log Project page (the default Project name is k8S-log- {cluster-id}).
  3. Enter the dashboard list in the left navigation bar and select Kubernetes Resource Operation Details.
  4. Enter Deployments in the upper left corner [Resource Type] filter box to query the Deployment that was previously deployed.
  5. Click the event ID in Resource Creation in the lower left corner to query detailed event information.










Adding Other logs (After the meeting)

You can use the log service Logtail to access service application logs to implement customized query, analysis, visualization, and alarm functions. You can access logs in either of the following ways:

1. Create collection configuration based on log service console:

  • In-container text files
  • Container standard output

2. Create a collection configuration using environment variables:

  • See Kubernetes-crd to configure log collection.

3. Create a collection configuration using CRD:

  • For details, see Using the Log Service to Collect Kubernetes Logs


Author: Yuan Yi

The original link

This article is the original content of the cloud habitat community, shall not be reproduced without permission.