preface

In the early days, we usually used GoAccess or awstat to analyze and statistics access logs such as Nginx. However, with the increasing diversity of statistical analysis and the increasing need for customization such as real-time monitoring of access logs, GoAccess or Awstat can not meet our needs. Therefore, we urgently need more flexible log statistical analysis tools that can assist us in statistics, analysis and monitoring of access logs. At this time, with the wide application of ELK/EFK,nginx and other access logs will also be included in the ELK system, while ELK can also meet our log statistics and analysis, monitoring diverse needs.

The above first









Deployment architecture

The following is a very simple architecture without buffering and aggregation. If you need more logs, you can add Redis or Kafka in the middle.

Why convert ingress or Nginx logs to JSON format? There are two main reasons: 1. It is easy to integrate Elasticseach for real-time monitoring and alarm. For example, direct monitoring of the status field, if there are 20 consecutive 4XX errors in 1 minute will directly alarm, if not converted, you need to further parse access logs, which brings a lot of inconvenience. 2: easy to combine elK to do visual analysis. You can combine different fields and customize different visual reports for different requirements.

Deployment steps

The ingress persistence step

1. Kubernetes ingress persistence

Ingress deployment Reference:www.pvcreate.com/index.php/a…

(1) Ingress Add PVC for ingress log storage

kubectl apply -f ingress-nfs.yaml

apiVersion: v1      
kind: PersistentVolumeClaim      
metadata:      
  name: ingress-nfs      
spec:      
  accessModes:      
  - ReadWriteMany      
  resources:      
    requests:      
      storage: 10Gi      
  storageClassName: nfs-client-local      
Copy the code

(2) Ingress adds mount

kubectl edit deployments.apps ingress-nginx-controller

. volumeMounts: - mountPath: /data/log/ingress/ name: ingress-nfs ...... volumes: - name: ingress-nfs persistentVolumeClaim: claimName: ingress-nfsCopy the code
(3) Ingress changes the log format and storage location

kubectl edit configmaps ingress-nginx-controller

access-log-path: /data/log/ingress/access_$hostname.log error-log-path: /data/log/ingress/error.log log-format-upstream: '{"@timestamp": "$time_iso8601","remote_addr": "$remote_addr","x-forward-for": "$proxy_add_x_forwarded_for","request_id": "$req_id","remote_user": "$remote_user","bytes_sent": $bytes_sent,"request_time": $request_time,"status": $status,"vhost": "$host","request_proto": "$server_protocol","path": "$uri","request_query": "$args","request_length": $request_length,"duration": $request_time,"method": "$request_method","http_referrer": "$http_referer","http_user_agent": "$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstrea m_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$ upstream_response_time,"upstream_status":$upstream_status}'Copy the code
2. Ingress persistence of Aliyun Kubernetes

Since ingress on Aliyun Kubernetes has been deployed by default, it is also recommended to use AliyunLogConfig to automatically access logging services and visualization. We used custom access ingress log stores for customization and other reasons, which means storing the ingress in nas and sending it to ElasticSearch.

(1) Add NAS storage for Ingress through Ali Cloud console





(2) Ingress adds mount

kubectl edit deployments.apps -n kube-system nginx-ingress-controller

. volumeMounts: - mountPath: /data name: nfs-oss ... volumes: - name: nfs-oss persistentVolumeClaim: claimName: ingress-logCopy the code
(3) Ingress changes the log format and storage location

kubectl edit configmaps ingress-nginx-controller Modify the content and the ingress self-built kubernetes above, it is important to note if the log path defined as ingress/data/log/ingress/access. The log, it is important to note that mount directory to exist, that is to say before you modify configmaps make sure / Data /log/ingress is created in advance, either in pod or externally.

Nginx log format changes

In addition to ingress, you will need to change the log format of nginx to JSON if your nginx needs to be pushed to elasticSearch synchronously. Note that ingress and Nginx do not support req_id as nginx does This parameter is not present in x. The following parameters are added to the HTTP global parameters of nginx.conf, adding.vim nginx.conf that is invalid in the server segment

log_format json '{"@timestamp": "$time_iso8601","remote_addr": "$remote_addr","x-forward-for":"$proxy_add_x_forwarded_for","remote_user": "$remote_user","bytes_sent":$bytes_sent,"request_time": $request_time,"status": $status,"vhost": "$host","request_proto":"$server_protocol","path": "$uri","request_query": "$args","request_length": $request_length,"duration":$request_time,"method": "$request_method","http_referrer": "$http_referer","http_user_agent":"$http_user_agent","upstream_addr":"$upstream_addr","upstream_response_length":$upstre am_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":"$upstream_status"}'; access_log /data/log/nginx/access.log json;Copy the code

Filebeat resolves Ingress/Nginx logs

The installation of FileBeat is very simple, so I will not go into details here. I will mainly post the fileBeat configuration file.filebeat.yml

setup.template.name: "local-app" setup.template.pattern: "local-app-*" setup.template.enabled: true setup.ilm.enabled: false filebeat.config: inputs: path: /data/www/apps/filebeat/inputs.d/*.yml reload.enabled: true reload.period: 10s modules: path: /data/www/apps/filebeat/modules.d/*.yml reload.enabled: false output.elasticsearch: protocol: Verification_mode: none hosts: ['192.168.1.100:9200'] username: "elastic" Password: "123456" index: "local-app-%{[fields.appName]}-%{+yyyy.MM.dd}"Copy the code

nginx_ingress.yml

- type: log enable: true tail_files: True # If set to true, Filebeat monitors new files from the end of the file and sends each new line as an event, # instead of resending all the content from the beginning: - /data/log/nginx/access*.log tags: [app, nginx-local] fields: appName: nginx-local json.keys_under_root: Overwrite_keys: true # For keys with the same name, overwrite the original key value json.message_key: Message #message_key is used to merge multiple lines of JSON logs. You also need to configure multiline. Json.add_error_key will be described later: Json.ignore_decoding_error: true # Specifies whether json decoding errors should be logged. If set to true, errors are loggedCopy the code

Note that if multiline is configured, the function of merging multiple JSON logs is enabled. If this function is not needed, comment out the configuration of multiline in this YML. (Since my access json logs are all one line in nginx/ingress, there is no need to configure multiline in nginx logs.)

[multiline.pattern: ^\[multiline.negate: true multiline.match: after] {multiline.pattern: ^\[multiline.negate: true multiline.match: afterCopy the code

Simultaneous configuration:

    processors:      
      - decode_json_fields:      
          fields: ['message']      
          target: json      
Copy the code

Kibana access elasticSearch and visual configuration

The installation and configuration of Kibana is not described here.

Add indexes step by step.



Several examples of typical graphical configurations

(1)PV



(2)UV



(3)Top10(interface visits)





(4)Top10(Proportion of CLIENT IP access)





(5)Top10(slowest interface)





(6) Proportion of back-end upstream





(7) Real-time traffic





(8) Client access ratio





(9) Average concurrency





(10) Statistics of abnormal status codes







(11) traffic



(12) Abnormal interface response code







(13) Interface access time ratio







(14) Average interface access time every 10 seconds





(15) Maximum interface access time every 10 seconds





(16) Status code statistics





(17) Trend chart of traffic volume





(18) The interface that exceeds 30 seconds





(19) The number of times the interface appears for more than 30 seconds





Five, step on the pit guide

Visual Metrics Cannot get the duration field

For example, **Top10(slowest interface)**, obtain the Top10 slowest url and form a table. However, I cannot find the duration field or request_time field in Metrics Maximum value, average value, etc. However, the fields I defined in Ingress are all strings, so they are also strings in ElasticSearch, so I can’t find the Duration field in Kibana Metrics. Now that we’ve found the root of the problem, we’ll start to fix it. Modify the Ingress confimap configuration to add indexes to Kibana. Of course, before adding the index, I delete the original index and add it again. Of course this method is more rough!! There are other ways to solve this problem: (1) If it is logstash, you can use mutate to convert fields

mutate {       
 convert => ["name-of-field", "integer"]       
}       
Copy the code

(2) There is no string conversion value available, but you can create a new index for elasticSearch and import it.

Duration curl -h 'content-type: application/json' -XPUT "http://localhost:9200/ingress_new" curl -H 'Content-Type: application/json' -XPOST "http://localhost:9200/ingress_new/ingress_old/_mapping? pretty" -d ' { "ingress_old": { "properties": { "duration": { "type": "double", "store": "true", "ignore_malformed": -x POST "localhost:9200/_reindex" -h 'content-type: application/json' -d' {"source": { "index": "ingress_old" "size": 5000 }, "dest": { "index": "ingress_new" "routing": "=cat" } }'Copy the code