Fluent Operator: A Swiss Army knife for cloud native log management

Cheng Dehao, Fluent Member, KubeSphere Member

Fluent Operator is introduced

With the rapid development of cloud native technology and continuous iteration of technology, higher requirements are put forward for log collection, processing and forwarding. There is a big difference between the logging architecture design based on physical machines or virtual airport landscape and the logging architecture design based on cloud native architecture. As a graduate project of CNCF, Fluent Bit is undoubtedly one of the preferred solutions for logging problems in cloud environments. However, the installation, deployment and configuration of Fluent Bit in Kubernetes has a certain threshold, which increases the cost of users.

On January 21, 2019, the KubeSphere community developed the Fluentbit Operator to meet the requirement of managing Fluent Bit in a cloud-native manner, and released v0.1.0 on February 17, 2020. Iterations followed, and the Fluentbit Operator was officially donated to the Fluent community on August 4, 2021.

The Fluentbit Operator reduces the threshold of using the Fluent Bit and can process log information efficiently and quickly. However, the Fluent Bit is weak in log processing. We have not integrated log processing tools, such as Fluentd, which has more plug-ins available. Based on the above requirements, the Fluentbit Operator integrates with Fluentd, aiming to integrate Fluentd into an optional log aggregation and forwarding layer, and renamed as the **Fluent Operator (GitHub address: Github.com/fluent/flue… Fluent Operator released v1.0.0 on March 25, 2022, and will continue to iterate on The Fluentd Operator, with v1.1.0 expected in q2 2022, adding more features and highlights.

The Fluent Operator provides flexibility and ease in deploying, configuring, and uninstalling the Fluent Bit and Fluentd. At the same time, the community also provides a large number of plug-ins supporting Fluentd and Fluent Bit, users can customize the configuration according to the actual situation. The official documentation provides detailed examples that are easy to use and reduce the barriers to using the Fluent Bit and Fluentd.

The stages of the logging pipeline

The Fluent Operator can deploy the Fluent Bit or Fluentd separately and is not required to use the Fluent Bit or Fluentd. It also supports multi-tenant log isolation using Fluentd to receive log flows forwarded by Fluent Bit, which greatly increases the flexibility and diversity of deployment. To understand the Fluent Operator more fully, the following figure takes the complete log pipeline as an example and divides the pipeline into three parts: collection and forwarding, filtering, and output.

Collection and forwarding

Both Fluent Bit and Fluentd can collect logs.

When deployed independently, the intPUT plug-in of Fluent Bit, Forward plug-in of Fluentd, and HTTP plug-in can be used to meet log collection requirements. When combined, Fluentd can use forward to accept the log flow of Fluent Bit.

In terms of performance, the Fluent Bit is lighter and consumes less memory (about 650KB) than Fluentd, so the Fluent Bit is mainly responsible for collecting and forwarding logs. The Fluent Bit installed as DaemonSet is used to collect and forward logs on each node.

filter

The data collected by logs is often too messy and redundant, which requires the log processing middleware to provide the ability to filter and process the log information. The Fluent Bit and Fluentd both support the Filter plug-in. Users can integrate and customize log data based on their own requirements.

The output

The Fluent Bit Output or Fluentd Output plug-in outputs the processed log information to multiple destinations, such as third-party components such as Kafka and Elasticsearch.

Introduction of CRD

The Fluent Operator defines two groups for the Fluent Bit and Fluentd: Fluentbit.fluent. IO and fluentD.fluent. IO.

fluentbit.fluent.io

Fluentbit.fluent. IO contains the following 6 CRDs:

• Fluentbit CRD defines the Fluent Bit properties, such as mirroring version, stain, affinity, and other parameters.
• The ClusterFluentbitConfig CRD defines the Fluent Bit configuration file.
• ClusterInput CRD defines the Input plug-in of Fluent Bit, that is, the input plug-in. The plug-in allows you to customize the type of logs to be collected.
• ClusterFilter CRD defines the Filter plug-in of Fluent Bit, which is mainly responsible for filtering and processing the information collected by Fluentbit.
• ClusterParser CRD defines Fluent Bit’s Parser plug-in, which parses log information into other formats.
• ClusterOutput CRD defines the Output plug-in of Fluent Bit, which is mainly responsible for forwarding processed log information to the destination.

fluentd.fluent.io

Fluentd.fluent. IO contains the following seven CRDs:

• Fluentd CRD defines Fluentd attributes, such as mirroring version, stains, and affinity.
• ClusterFluentdConfig CRD defines Fluentd cluster-level configuration files.
• FluentdConfig CRD defines the namespace configuration file of Fluentd.
• ClusterFilter CRD defines the Fluentd cluster-wide Filter plug-in, which filters and processes the information collected by Fluentd. If Fluent Bit is installed, you can further process the log information.
• Filter CRD This CRD defines the Filter plug-in of Fluentd Namespace, which filters and processes the information collected by Fluentd. If Fluent Bit is installed, you can further process the log information.
• ClusterOutput CRD This CRD defines the Cluster-wide Output plug-in of Fluentd, which forwards processed log information to the destination.
• Output CRD This CRD defines the Namespace Output plug-in of Fluentd, which forwards processed log information to the destination.

Instance + Mounted Secret + CRD

Although Fluent Bit and Fluentd both have the ability to collect, process (parse and filter), and output logs, they have different advantages. The Fluent Bit is lighter and more efficient than Fluentd, and Fluentd has more plug-ins.

To accommodate these benefits, the Fluent Operator allows users the flexibility to use the Fluent Bit and Fluentd in a variety of ways:

• Fluent Bit Only mode: If you only need to collect logs and send them to their final destination after simple processing, you only need the Fluent Bit.
• Fluentd only mode: If you need to receive logs over the network in HTTP or Syslog mode, process logs and send them to the final destination, only Fluentd is required.
• Fluent Bit + Fluentd mode: If you need to do some advanced processing on the collected logs or send them to more sinks, you can use Fluent Bit and Fluentd together.

The Fluent Operator allows you to configure the logging pipeline in each of the three modes as required. Fluentd and Fluent Bit have rich plug-ins to meet various customization requirements of users. Because Fluentd and Fluent Bit have similar configuration mount modes, the mount mode of Fluent Bit configuration file is briefly introduced.

Bit in Fluent CRD in each ClusterInput ClusterParser, ClusterFilter, Bit ClusterOutput represents a Fluent configuration section, Selected by the ClusterFluentBitConfig label selector. The Fluent Operator monitors these objects, builds the final configuration, and finally creates a Secret to store the configuration installed into the Fluent Bit DaemonSet. The entire workflow is as follows:

Because the Fluent Bit itself does not reload the interface (see this known issue for details), a wrapper called Fluentbit Watcher was added to enable the Fluent Bit to get and use the latest configuration when the Fluent Bit configuration changes. To restart the Fluent Bit process as soon as a configuration change is detected. This allows you to reload the new configuration without restarting Fluent Bit Pod.

To facilitate user configuration, we extract application and configuration parameters based on CRD’s powerful abstraction capability. You can configure the Fluent Bit and Fluentd using the defined CRD. The Fluent Operator monitors these objects to change the state and configuration of the Fluent Bit and Fluentd. Especially for the definition of the plug-in, in order to make the transition smoother for the user, we basically keep the same naming with the original fields of Fluent Bit to reduce the threshold of use.

How do I implement multi-tenant log isolation

The Fluent Bit can efficiently collect logs, but if complex log information processing is needed, the Fluent Bit is a little powerless, while Fluentd can complete advanced log information processing with the help of its rich plug-ins. Fluent-operator abstracts various plug-ins of Fluentd so that it can process log information to meet user customization requirements.

As can be seen from the above CRD definition, Fluentd config and plug-in CRD are divided into cluster level and namespace level CRD. Multi-tenant isolation can be achieved by defining CRDS into two scopes with the help of Fluentd’s Label Router plug-in.

We added the watchNamespace field in ClusterFluentConfig. Users can select which namespaces to listen to according to their own requirements. If it is empty, it means that all namespaces are monitored. Fluentconfig at the Namesapce level can only listen for CR and global configurations in the namespace where it resides. Therefore, namespace-level logs can be output to both the outputs of the namespace and the outputs of the CLsuter level to achieve multi-tenant isolation.

Fluent Operator vs logging-operator

The difference

• Fluent Bit and Fluentd are automatically deployed for both. Logging-operator requires the Fluent Bit and Fluentd to be deployed simultaneously. The Fluent operator supports pluggable deployment of the Fluent Bit and Fluentd, which is not strongly coupled. Users can deploy Fluentd or Fluent Bit according to their own needs, which is more flexible.
• Logs collected by the Fluent Bit in logging-operator must pass through Fluentd before being output to the final destination, and Fluentd may have a single point of failure if the amount of data is too large. The Fluent Bit in the Fluent Operator can directly send log information to the destination, avoiding the potential single point of failure.
• Logging-operator defines loggings, Outputs, Flows, ClusterFlows, and Fluent Operator defines 13 CRDS. Compared to logging-operator, the Fluent operator has a more diverse CRD definition, allowing users to configure Fluentd and Fluent Bit more flexibly as needed. At the same time, when defining CRD, we choose a name similar to Fluentd and Fluent Bit configuration to make naming clearer and conform to the native component definition.
• Both use the Label Router plug-in of Fluentd for reference to realize multi-tenant log isolation.

Outlook:

• Support HPA automatic scaling;
• Improve Helm Chart, such as collecting metrics information;
• .

Usage,

The Fluent Operator allows for complex log processing, The fluent operator can export logs to ElasticSearch and Kafka by using fluent-operator-walkThrough. To get some hands-on experience with the Fluent Operator, you need a Kind cluster. You also need to set up a Kafka cluster and an Elasticsearch cluster in this type of cluster.

#Create a cluster of KIND and name it Fluent
./create-kind-cluster.sh

#Create a Kafka cluster under Kafka namespace
./deploy-kafka.sh

#Create an Elasticsearch cluster under elastic Namespace
./deploy-es.sh
Copy the code

The Fluent Operator controls the Fluent Bit and Fluentd life cycles. You can start the Fluent Operator in the Fluent namespace using the following script:

./deploy-fluent-operator.sh
Copy the code

Both Fluent Bit and Fluentd have been defined as CRDS in Fluent Operator, You can create FluentBit DaemonSet or Fluentd StatefulSet by declaring FluentBit or Fluentd’s CR.

Fluent Bit Only mode

Fluent Bit Only enables Only the lightweight Fluent Bit to collect, process, and forward logs.

Use Fluent Bit to collect kubelet logs and export them to Elasticsearch

cat <<EOF | kubectl apply -f -
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: - bit: kubesphere/fluent v1.8.11
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/
  resources:
    requests:
      cpu: 10m
      memory: 25Mi
    limits:
      cpu: 500m
      memory: 200Mi
  fluentBitConfigName: fluent-bit-only-config
  tolerations:
    - operator: Exists
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  name: fluent-bit-only-config
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  service:
    parsersFile: parsers.conf
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "fluentbit-only"
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "fluentbit-only"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "fluentbit-only"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: kubelet
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "fluentbit-only"
spec:
  systemd:
    tag: service.kubelet
    path: /var/log/journal
    db: /fluent-bit/tail/kubelet.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=kubelet.service
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
  name: systemd
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "fluentbit-only"
spec:
  match: service.*
  filters:
  - lua:
      script:
        key: systemd.lua
        name: fluent-bit-lua
      call: add_time
      timeAsTable: true
---
apiVersion: v1
data:
  systemd.lua: | function add_time(tag, timestamp, record) new_record = {} timeStr = os.date("! *t", timestamp["sec"]) t = string.format("%4d-%02d-%02dT%02d:%02d:%02d.%sZ", timeStr["year"], timeStr["month"], timeStr["day"], timeStr["hour"], timeStr["min"], timeStr["sec"], timestamp["nsec"]) kubernetes = {} kubernetes["pod_name"] = record["_HOSTNAME"] kubernetes["container_name"] = record["SYSLOG_IDENTIFIER"] kubernetes["namespace_name"] = "kube-system" new_record["time"] = t new_record["log"] = record["MESSAGE"] new_record["kubernetes"] = kubernetes return 1, timestamp, new_record endkind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/name: fluent-bit-lua
  name: fluent-bit-lua
  namespace: fluent
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: es
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "fluentbit-only"
spec:
  matchRegex: (? :kube|service)\.(.*)
  es:
    host: elasticsearch-master.elastic.svc
    port: 9200
    generateID: true
    logstashPrefix: fluent-log-fb-only
    logstashFormat: true
    timeKey: "@timestamp"
EOF
Copy the code

Use Fluent Bit to collect kubernetes application logs and output them to Kafka

cat <<EOF | kubectl apply -f -
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: - bit: kubesphere/fluent v1.8.11
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/
  resources:
    requests:
      cpu: 10m
      memory: 25Mi
    limits:
      cpu: 500m
      memory: 200Mi
  fluentBitConfigName: fluent-bit-config
  tolerations:
    - operator: Exists
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  name: fluent-bit-config
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  service:
    parsersFile: parsers.conf
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "k8s"
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "k8s"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/mode: "k8s"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  tail:
    tag: kube.*
    path: /var/log/containers/*.log
    parser: docker
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: true
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
  name: kubernetes
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  match: kube.*
  filters:
  - kubernetes:
      kubeURL: https://kubernetes.default.svc:443
      kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      labels: false
      annotations: false
  - nest:
      operation: lift
      nestedUnder: kubernetes
      addPrefix: kubernetes_
  - modify:
      rules:
      - remove: stream
      - remove: kubernetes_pod_id
      - remove: kubernetes_host
      - remove: kubernetes_container_hash
  - nest:
      operation: nest
      wildcard:
      - kubernetes_*
      nestUnder: kubernetes
      removePrefix: kubernetes_
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: kafka
  labels:
    fluentbit.fluent.io/enabled: "false"
    fluentbit.fluent.io/mode: "k8s"
spec:
  matchRegex: (? :kube|service)\.(.*)
  kafka:
    brokers: my-cluster-kafka-bootstrap.kafka.svc:9091,my-cluster-kafka-bootstrap.kafka.svc:9092,my-cluster-kafka-bootstrap.kafka.svc : 9093
    topics: fluent-log
EOF
Copy the code

Fluent Bit + Fluentd mode

With Fluentd’s rich plug-ins, Fluentd can act as a log aggregation layer to perform more advanced log processing. You can easily forward logs from the Fluent Bit to Fluentd using the Fluent Operator.

Forward logs from Fluent Bit to Fluentd

To forward logs from Fluent Bit to Fluentd, you need to enable the Forward plug-in of Fluent Bit, as shown below:

cat <<EOF | kubectl apply -f -
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: fluentd
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/component: logging
spec:
  matchRegex: (? :kube|service)\.(.*)
  forward:
    host: fluentd.fluent.svc
    port: 24224
EOF
Copy the code

Deploy Fluentd

The Fluentd Forward Input plugin is enabled by default when Fluentd is deployed, so you only need to deploy yamL to deploy Fluentd:

apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
  name: fluentd
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluentd
spec:
  globalInputs:
  - forward:
      bind: 0.0. 0. 0
      port: 24224
  replicas: 1
  image: Kubesphere/fluentd: v1.14.4
  fluentdCfgSelector:
    matchLabels:
      config.fluentd.fluent.io/enabled: "true"
Copy the code

ClusterFluentdConfig: Fluentd cluster-wide configuration

If you define ClusterFluentdConfig, you can collect logs under any or all namespaces. You can use the watchedNamespaces field to select a namespace to collect logs. The following configuration collects logs from kube-system and the default namespace:

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: cluster-fluentd-config
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces:
  - kube-system
  - default
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/scope: "cluster"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-es
  labels:
    output.fluentd.fluent.io/scope: "cluster"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-cluster-fd
EOF

Copy the code

FluentdConfig: Fluentd Namespace configuration

If you define FluentdConfig, you can only send logs from the same namespace as FluentdConfig to Output. This way you can isolate logs from different namespaces.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: FluentdConfig
metadata:
  name: namespace-fluentd-config
  namespace: fluent
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  outputSelector:
    matchLabels:
      output.fluentd.fluent.io/scope: "namespace"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Output
metadata:
  name: namespace-fluentd-output-es
  namespace: fluent
  labels:
    output.fluentd.fluent.io/scope: "namespace"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-namespace-fd
EOF
Copy the code

Route logs to different Kafka topics based on namespace

Also, you can use Fluentd’s Filter plug-in to distribute logs to different topics based on different namespaces. Here we include recordTransformer plug-in in the Fluentd kernel, which can add, delete, change and check events.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: cluster-fluentd-config-kafka
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces:
  - kube-system
  - default
  clusterFilterSelector:
    matchLabels:
      filter.fluentd.fluent.io/type: "k8s"
      filter.fluentd.fluent.io/enabled: "true"
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/type: "kafka"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFilter
metadata:
  name: cluster-fluentd-filter-k8s
  labels:
    filter.fluentd.fluent.io/type: "k8s"
    filter.fluentd.fluent.io/enabled: "true"
spec:
  filters:
  - recordTransformer:
      enableRuby: true
      records:
      - key: kubernetes_ns
        value: ${record["kubernetes"]["namespace_name"]}
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-kafka
  labels:
    output.fluentd.fluent.io/type: "kafka"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
  - kafka:
      brokers: my-cluster-kafka-bootstrap.default.svc:9091,my-cluster-kafka-bootstrap.default.svc:9092,my-cluster-kafka-bootstrap.defau lt.svc:9093
      useEventTime: true
      topicKey: kubernetes_ns
EOF

Copy the code

Use both cluster-scoped and namespace scoped FluentdConfig

Of course, you can use ClusterFluentdConfig and FluentdConfig together as shown below. FluentdConfig will send the logs under the Fluent namespace to ClusterOutput. ClusterFluentdConfig also sends namespaces (kube-System and default) under watchedNamespaces to ClusterOutput.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: cluster-fluentd-config-hybrid
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces:
  - kube-system
  - default
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/scope: "hybrid"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: FluentdConfig
metadata:
  name: namespace-fluentd-config-hybrid
  namespace: fluent
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/scope: "hybrid"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-es-hybrid
  labels:
    output.fluentd.fluent.io/scope: "hybrid"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-hybrid-fd
EOF

Copy the code

In multi-tenant scenarios, FluentdConfig in both the cluster scope and namespace scope is used

In multi-tenant scenarios, we can use both cluster-scoped and namespace-scoped FluentdConfig to isolate logs.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: FluentdConfig
metadata:
  name: namespace-fluentd-config-user1
  namespace: fluent
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  outputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"
      output.fluentd.fluent.io/user: "user1"
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"
      output.fluentd.fluent.io/user: "user1"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: cluster-fluentd-config-cluster-only
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces:
  - kube-system
  - kubesphere-system
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"
      output.fluentd.fluent.io/scope: "cluster-only"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Output
metadata:
  name: namespace-fluentd-output-user1
  namespace: fluent
  labels:
    output.fluentd.fluent.io/enabled: "true"
    output.fluentd.fluent.io/user: "user1"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-user1-fd
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-user1
  labels:
    output.fluentd.fluent.io/enabled: "true"
    output.fluentd.fluent.io/user: "user1"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-cluster-user1-fd
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-cluster-only
  labels:
    output.fluentd.fluent.io/enabled: "true"
    output.fluentd.fluent.io/scope: "cluster-only"
spec:
  outputs:
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-cluster-only-fd
EOF

Copy the code

Use buffers for Fluentd output

You can add a buffer to cache the output plug-in’s logs.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: cluster-fluentd-config-buffer
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces:
  - kube-system
  - default
  clusterFilterSelector:
    matchLabels:
      filter.fluentd.fluent.io/type: "buffer"
      filter.fluentd.fluent.io/enabled: "true"
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/type: "buffer"
      output.fluentd.fluent.io/enabled: "true"
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFilter
metadata:
  name: cluster-fluentd-filter-buffer
  labels:
    filter.fluentd.fluent.io/type: "buffer"
    filter.fluentd.fluent.io/enabled: "true"
spec:
  filters:
  - recordTransformer:
      enableRuby: true
      records:
      - key: kubernetes_ns
        value: ${record["kubernetes"]["namespace_name"]}
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: cluster-fluentd-output-buffer
  labels:
    output.fluentd.fluent.io/type: "buffer"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
  - stdout: {}
    buffer:
      type: file
      path: /buffers/stdout.log
  - elasticsearch:
      host: elasticsearch-master.elastic.svc
      port: 9200
      logstashFormat: true
      logstashPrefix: fluent-log-buffer-fd
    buffer:
      type: file
      path: /buffers/es.log
EOF
Copy the code

Fluentd Only mode

You can also turn on Fluentd Only mode, which deploys Only Fluentd StatefulSet.

Use Fluentd to receive logs from HTTP and output them to standard output

If you want to enable the Fluentd plugin separately, you can receive logs over HTTP.

cat <<EOF | kubectl apply -f -
apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
  name: fluentd-http
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluentd
spec:
  globalInputs:
    - http:
        bind: 0.0. 0. 0
        port: 9880
  replicas: 1
  image: Kubesphere/fluentd: v1.14.4
  fluentdCfgSelector:
    matchLabels:
      config.fluentd.fluent.io/enabled: "true"

---
apiVersion: fluentd.fluent.io/v1alpha1
kind: FluentdConfig
metadata:
  name: fluentd-only-config
  namespace: fluent
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  filterSelector:
    matchLabels:
      filter.fluentd.fluent.io/mode: "fluentd-only"
      filter.fluentd.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      output.fluentd.fluent.io/mode: "true"
      output.fluentd.fluent.io/enabled: "true"

---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Filter
metadata:
  name: fluentd-only-filter
  namespace: fluent
  labels:
    filter.fluentd.fluent.io/mode: "fluentd-only"
    filter.fluentd.fluent.io/enabled: "true"
spec:
  filters:
    - stdout: {}

---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Output
metadata:
  name: fluentd-only-stdout
  namespace: fluent
  labels:
    output.fluentd.fluent.io/enabled: "true"
    output.fluentd.fluent.io/enabled: "true"
spec:
  outputs:
    - stdout: {}
EOF

Copy the code

This article is published by OpenWrite!