A background

At present, K8S monitoring can be divided into: Resource monitoring, performance monitoring, safety and health, etc., but in K8s, how to express the state of a resource object and some of the resource state conversion, need event monitoring to express, at present Ali has an open source K8s event monitoring project Kube-Eventer, which will be divided into two kinds of events, one is Warning events, Indicates that the state transition that produced this event was between unexpected states; The other is a Normal event, which indicates that the desired state is the same as the current state.

Events of resource objects such as POD/Node/Kubelet can be collected, as well as events of user-defined resource objects can be collected and sent to the receiving end of the configuration well, as shown in the architecture diagram below.

2 Installation and Deployment

Using ali’s open source K8S event monitoring project, you can collect K8S event logs from defining level alarms

2.1 Configuration of nailing robot

Currently, the newly added custom robot needs security configuration. The label can be configured and then defined in sink’s label

Record saving webhook

https://oapi.dingtalk.com/robot/send?access_token=e1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8fe
Copy the code

2.2 Deploying Resources

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    name: kube-eventer
  name: kube-eventer
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eventer
  template:
    metadata:
      labels:
        app: kube-eventer
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: kube-eventer
      containers:
        - image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.1.0-63e7f98-aliyun
          name: kube-eventer
          command:
            - "/kube-eventer"
            - "--source=kubernetes:https://kubernetes.default"
            ## .e.g,dingtalk sink demo
            - --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=exxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx28fe&label=anc hnet-kubesphere&level=Warning&msg_type=markdown
          env:
          # If TZ is assigned, set the TZ value as the time zone
          - name: TZ
            value: America/New_York
          volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
            - name: zoneinfo
              mountPath: /usr/share/zoneinfo
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
            limits:
              cpu: 500m
              memory: 250Mi
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: zoneinfo
          hostPath:
            path: /usr/share/zoneinfo

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-eventer
rules:
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - get
      - list
      - watch
  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
  name: kube-eventer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eventer
subjects:
  - kind: ServiceAccount
    name: kube-eventer
    namespace: kube-system
    
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-eventer
  namespace: kube-system
Copy the code

Access alarm test

3.1 Nail access test

--sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=exxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx28fe&label=anch net-kubesphere&level=Warning&msg_type=markdown
Copy the code

3.2 Enterprise wechat access test

--sink=wechat:? corp_id=wwxxxxxxxxxx1a&corp_secret=gxxxxxxxxxxxxxxxxxxxxxxxxxxx4U&agent_id=10xxxxxx7&to_user=&level=Warning&label=K8S-Al ert-Prod&msg_type=markdown
Copy the code

Refer to the link

  • Github.com/AliyunConta…
  • Github.com/AliyunConta…