Author: Lei Wanjun, Observability r&d engineer at KubeSphere

preface

As an important part of observability, alarm notification helps us discover problems in time, and logs help us locate problems quickly. As an open source container orchestration platform, KubeSphere provides powerful log collection and query capabilities, as well as flexible alarm notification capabilities.

This article will show you how KubeSphere’s logging and notification capabilities are implemented.

The log

KubeSphere collects logs by Fluent Bit. The Fluent Bit collects Pod logs to ElasticSearch cluster. KubeSphere searches ElasticSearch cluster for logs.

Fluent Bit

Fluent Bit is an open source logging processor and repeater that can collect any data, such as metrics and logs, from different sources, process them with filters and send them to multiple destinations. It is the first choice for containerized environments such as Kubernetes.

The Fluent Bit was designed with performance in mind: high throughput, low CPU and memory utilization. Written in C, it has a pluggable architecture and supports more than 70 input, filter, and output extensions.

Logs are sent from the data source to the destination through a data pipeline, usually consisting of Input, Parser, Filter, Buffer, Routing, and Output.

  • Input: Used to extract data from a data source. A data pipeline can contain multiple inputs.
  • Parser: Converts unstructured data extracted from Input into standard structured data. Each Input can have its own Parser.
  • Filter: filters and modifies formatted data. A data pipe can contain multiple filters, which are executed in the same order as in the configuration file.
  • Buffer: Users cache data processed by Filter.
  • Routing: Routes the cached data in the Buffer to different outputs.
  • Outputs: Responsible for sending data to different destinations. A data pipeline can contain multiple outputs.

Fluent Bit supports various types of Input, Parser, Filter, Output plug-ins, which can deal with various scenarios.

Fluent Bit is the preferred log collection tool for container platforms. It is efficient and lightweight, but it is not easily deployed.

  • First, to deploy the Fluent Bit in a traditional way, a DaemonSet needs to be manually created. When the Fluent Bit needs to be modified, the DaemonSet also needs to be manually modified, which is not convenient and error-prone.
  • Secondly, the configuration file of Fluent Bit is not yamL or JSON format commonly used in the cloud native field, so the configuration is cumbersome and error-prone.
  • Again, Fluent Bit does not support dynamic loading of configuration files, and DaemonSet needs to be restarted every time the configuration files are updated.

Based on these issues, the KubeSphere Observability team developed the FluentBit Operator to deploy and manage the FluentBit.

FluentBit Operator

The FluentBit Operator is an open source FluentBit management tool, which can achieve rapid deployment of The FluentBit and dynamic modification and loading of the FluentBit configuration file.

  • Administration: Automatically deploy and destroy Fluent Bit DaemonSet.
  • Custom configuration: Use CRD to define the Fluent Bit configuration.
  • Dynamic loading: Fluent Bit Pod can be updated without restarting the configuration.

The FluentBit Operator creates and manages the FluentBit Pod through CRD and dynamically updates the FluentBit Pod and The FluentBit configuration by monitoring the CHANGES of CRD.

FluentBit Operator CRDS

The CRD defined by the FluentBit Operator includes:

  • FluentBit: Used to create FluentBit DaemonSet.
  • FluentBitConfig: Used to select plug-ins that the FluentBit Operator needs to manage.
  • Input: Used to define the Fluent Bit Input plug-in.
  • Parser: Defines the Fluent Bit Parser plug-in.
  • Filter: Used to define the Fluent Bit Filter plug-in.
  • Output: Used to define the Fluent Bit Output plug-in.

The FluentBit Operator generates the FluentBit configuration file by listening on FluentBitConfig, Input, Parser, Filter, and Output CRD and writes the configuration file to Secret. When the CRD changes, the configuration file is automatically updated.

FluentBit Operator: FluenBit CRD; FluentBit Operator: FluenBit CRD; And mount the configuration file into Fluent Bit DaemonSet. When configuration files change, the configuration files in Fluent Bit Pod are also updated synchronously. How does the FluentBit Operator implement FluentBit to dynamically load configuration files?

Fluent Bit Watcher

Fluent Bit Pod does not run Fluent Bit directly, but runs a process named Fluent-bit-watcher. After starting, Fluent-bit-watcher will start Fluent Bit and listen to configuration files at the same time. When the configuration file changes, Fluent-bit-Watcher restarts The Fluent bit to reload the configuration file.

The logging stored

The Fluent Bit collects logs to the ElasticSearch cluster for persistence. Logs are stored in one fragment per day. KubeSphere supports the configuration of log retention period.

KubeSphere also supports logging to Kafka and Fluentd.

Log retrieval

When collecting logs, Fluent Bit adds container metadata information to the logs, so KubeSphere provides users with a variety of log query methods.

  • Multi-tenant log management implements permission – and domain-based management of logs of different tenants. Each tenant can query logs of only the containers in the Namespace that he/she has permission to access.
  • Multi-level log query Logs are queried by project, workload, container group, container, and keyword to locate problems at different levels.

notice

KubeSphere supports multi-tenant notification. Each tenant can customize its own notification channel to receive notification messages under the Namespace that the tenant has permission to access. In addition, you can set a global notification channel to receive all notification messages, including all tenant notification messages and platform-level notification messages.

KubeSphere’s Notification capabilities are implemented through the Notification Manager.

Notification Manager

Notification Manager is a multi-tenant Notification management system on the Kubernetes platform developed by KubeSphere Observable team. It receives alarm messages from the Alertmanager. In addition, alarms are sent to corresponding notification channels such as DingTalk, Email, Slack, WeCom, Webhook, and SMS platforms such as Alibaba, Tencent, and Huawei based on the tenant labels of alarm messages (such as Namespace).

Notification Manager consists of Operator, which is used to create and manage Deployment, and Deployment, which is used to receive alarm messages, generate Notification messages, In addition, the alarm message is sent to the corresponding notification channel based on the tenant label (such as namespace).

NotificationManager manages and configures the Deployment through CRD, Operator creates and manages the Deployment by listening to NotificationManager CRD, Deployment listens to NotificationManager, Config and Receiver CRD. When CRD changes, it overloads CRD to update configuration information, realizing the dynamic update of notification channel.

The role of CRD defined by the Notification Manager is as follows:

  • NotificationManager: used to configure Webhooks, including mirrors, duplicates, volumes, affinity, stains, and resource quotas. It also defines the configuration required to send notifications, global receiver and default configuration selectors, tenant labels, tenant level receiver selectors, and global configuration of notification channels.
  • Config: used to define the configuration information of the sender of the notification channel, such as the Settings of the mail sending server and the information about the APP that sends messages on the enterprise wechat account.
  • Receiver: Information used to define the Receiver of notification channels, such as email recipients, users or departments in enterprise wechat, slack channels, etc.

Notification Manager supports multi-tenant Notification. How does Notification Manager manage multi-tenant Notification?

The Notification Manager uses a separate mode of sending configuration and receiving configuration. That is, Config defines the sending configuration and Receiver defines the receiving configuration. The Receiver selects the sending configuration required for sending notifications through labels. Config and Receiver are classified into global and tenant types. The following labels are used to distinguish them.

Type: global // global Receiver

Type: default // global Config

Type: tenant // Tenant Receiver or Config

The global Receiver can only use the global Config. The tenant Receiver selects the tenant Config defined by the tenant through the label selector. If the label selector is not defined or the Config is not found, the global Config is used.

In this mode, users can quickly configure complex multi-tenant notification scenarios. For example, you can use a unified sender for Email notification. The administrator can set the global Email Config, and the tenant only needs to configure the recipient Email address to complete Email notification configuration.

The Notification Manager sends notifications to the appropriate Receivers based on the Namespace tag of the Notification message.

  • All notification messages are sent to the global Receiver.
  • If the namespace is empty, it is only sent to the global Receiver.
  • If the namespace is not empty, the Notification Manager searches for the list of tenants to be notified based on the namespace, and then obtains the Receiver to be sent based on the list of tenants. Corresponding to the native Kubernetes cluster, the namespace to be notified is the tenant. For other clusters, such as KubeSphere, the Notification Manager supports getting a list of tenants to be notified through the API. Users can implement the logic of obtaining the list of tenants to be notified by themselves and inject it into Notification Manager Deployment by sidecar for invocation by Notification Manager.

Custom notification messages

Notification Manager supports custom Notification messages that users can customize by writing message templates. The Notification template of the Notification Manager is written using the Go template. Here is a simple message template.

{{ define "__nm_alert_list" }}{{ range . }}Labels:
{{ range .Labels.SortedPairs }}{{ if ne .Name "runbook_url" }}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }}{{ if ne .Name "runbook_url"}}- {{ .Name }} = {{ .Value }}{{ end }}
{{ end }}{{ end }}{{ end }}

{{ define "nm.default.text" }}{{ template "nm.default.subject" . }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__nm_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__nm_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
Copy the code

The output notification message is as follows.

Notification Manager supports multiple levels of template Settings.

  • The Notification Manager sets a default message template for each Notification channel that users can use directly.
  • Users can set global templates for use by all receivers.
  • Users can set a unified template for each notification channel.
  • Users can set up separate templates for each Receiver.

Notification Message Filtering

The Notification Manager supports filtering Notification messages. You can set a filter to filter Notification messages. The Notification Manager supports setting separate filters for each Receiver.

A simple filter to filter out warning alarms.

apiVersion: notification.kubesphere.io/v2beta1
kind: Receiver
metadata:
  labels:
    app: notification-manager
    type: global
  name: global-email-receiver
spec:
  email:
    to:
    - [email protected]
    - [email protected]
    alertSelector:
      matchExpressions:
      - key: severity
        operator: In
        values:
        - error
        - critical
Copy the code

Advantages of KubeSphere notification

  • CRD management supports dynamic update
  • Support rich notification channels
  • Supports notification message filtering
  • Supports custom notification messages

This article is published by OpenWrite!