\

The background,

The services in the K8S cluster need to collect logs. The services are deployed in pod mode in the cluster, that is, collect logs in containers.

Second, the plan

There are two types of container logs

  • Console logs: Console logs are stored in the docker directory of the host computer. The container directory is mounted to FluentD. Fluentd collects logs and sends them to ELK through Kafka.

  • User-generated logging: You need to mount internal container logs to third-party storage and send logs to ELK via Kafka via Fluted.

    Users can perform log retrieval, service monitoring, and alarm configurations based on their own requirements in ELK.

Three, configuration,

Yaml contains the configuration of Fluentd, presented in configMap, and the deployment of Fluentd.

For details about fluentD configuration files, see the official website.

The indicators include: Prometheus Indicators:

@type tail Configuration related to log collection

Added custom Prometheus indicators

Configuration kafka

  • configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config-pro   This name should correspond to that in deployment
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
data:
  fluent.conf: | # # Prometheus exposure index input plugin that exports the metrics < source > @ type Prometheus bind 0.0.0.0 port # 24231 Metrics_path /metrics aggregated_metrics_path/Aggregated_metrics   @type Port for which Prometheus indicators are exposed Monitor_agent bind 0.0.0.0 port 24220   @type forward bind 0.0.0.0 port 24224    
    # input plugin that collects metrics for output plugin
    
    <source>
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    
    # input plugin that collects metrics for in_tail plugin
    <source>
      @type prometheus_tail_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    
    ## Collect file log configuration
    <source>
      @type tail
      path /home/work/logs/hadoop_nm_test/*  The path of logs to be collected can be matched with *
      tag logpath.*                Add a tag and filter it by tag
      rotate_wait 120
      refresh_interval 10
      read_from_head true
      #time_key event_time
      keep_time_key true
      #path_key sourceid
      #limit_recently_modified 86400
      #timekey_zone Asia/Shanghai
      pos_file_compaction_interval 1h
      pos_file /home/work/log-pos/fluentd-log-1.pos  # Fluentd Read the location of the log file. It is best to save it outside the container, otherwise the POD restart will cause the log to be collected from scratch.
      <parse>
       @type none
      </parse>
    </source>

    <filter * * >
     @type record_transformer
     <record>
       nodeip "#{ENV['MY_NODE_IP']}"
     </record>
    </filter>

    <filter * * >
     @type record_transformer
     <record>
       message ${record["nodeip"]} ${record["message"]}
     </record>
    </filter>

    <filter logpath.home.work.face.logs.hadoop_nm_test.**>
      @type prometheus
      # You can use counter type without specifying a key
      # This just increments counter by 1
      <metric>
        name de_input_num_records_total   # Total number of rows read for Prometheus monitoring
        type counter
        desc The total number of input records
        <labels>
           tag ${tag}
           host ${hostname}
           host $.nodeip
        </labels>
      </metric>
    </filter>
    
    # configuration kafka
    <match logpath.home.work.face.logs.hadoop_nm_test.**>
      @type copy
      # for MonitorAgent sample
      <store>
        @type kafka2
        brokers [Kafka node]   # kafka cluster nodes
        <buffer topic>
         @type file
         path /fluentd/buffer/td/api
         flush_interval 3s
        </buffer>
        <format>
         @type single_value
        </format>
        default_topic [kafka topic] # Kafka cluster topic
      </store>
      <store>
        @id caelus_prometheus
        @type prometheus
        <metric>
          name de_output_num_records_total   # Total number of output lines for Prometheus monitoring
          type counter
          desc The total number of outgoing records
          <labels>
            tag ${tag}
            host ${hostname}
            host $.nodeip
          </labels>
        </metric>
      </store>
    </match>
Copy the code
  • deployment
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-pro
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
  annotations:
    configmap.reloader.stakater.com/reload: "fluentd-config-name"  # fluentd-config-name specifies the configmap name
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        env:
        - name: MY_NODE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
# image: fluentd: v1.14.0-1.0
        image: registry.ke.com/docker-virtual/common/fluentd:v1.10.2-1.0
        args:
        #- -v
        # - -c /fluentd/etc/fluent.conf
        #command: [ "sleep", "3600" ]
        securityContext:
          runAsUser: 0
          allowPrivilegeEscalation: false
        imagePullPolicy: IfNotPresent
        ports:
          - name: fluentd
            containerPort: 24231   NodePort is used for index exposure
            hostPort: 24231
        resources:
          limits:
            cpu: 500m
            memory: 1000Mi
          requests:
            cpu: 500m
            memory: 1000Mi
        volumeMounts:
        - name: api-logs
          mountPath: /home/work/face/logs
          readOnly: true
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
        - name: config-source
          mountPath: /fluentd/etc/fluent.conf
          subPath: fluent.conf
        - name: fluted-log-pos
          mountPath: /home/work/log-pos
      terminationGracePeriodSeconds: 30
      volumes:
      - name: api-logs
        hostPath:
          path: /home/work/face/logs  # host log file location
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
      - name: fluted-log-pos
        hostPath:
          path: /home/work/face/log-ops
      - name: config-source
        configMap:
          name: fluentd-config-pro   # corresponds to configmap above
          items:
           - key: fluent.conf
             path: fluent.conf
Copy the code

Hot loading of fluentD configuration files

During FluentD deployment, newly added services often need to collect logs. In this case, you only need to modify the ConfigMap to add the Fluentd configuration. However, if you only modify the ConfigMap, the configuration is not hot loaded in POD.

So use a plug-in Reloader to deploy the plug-in.

kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
Copy the code

And add it in deployment

annotations: configmap.reloader.stakater.com/reload: "fluentd-config-name"
Copy the code

Note: This hot loading will cause pod to restart, so the pos_file in FluentD must be mounted outside the container or the logs will be recollected each time.

Five, the pit

  • There are subdirectories in log files

Fluentd does not collect files under subdirectories if there are subdirectories under the source directory, so you need to add each subdirectory separately. Multiple directories can be configured at the same time.

  • Error collecting docker container console logs

    2018-08-03 06:36:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log unreadable. It is excluded and would  be examined next time. 2018-08-03 06:37:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5bc89ab24.log unreadable. It is excluded and would  be examined next time.Copy the code

    Because the/var/log/containers is soft chain to/var/lib/containers or/home/work/docker/containers below, so the actual log is stored in the soft even the source address.

    You need to mount the original address into the POD as well