Hello everyone, I’m Zhang Jintao.
Previously, I wrote a more Elegant Kubernetes Cluster Event Measurement Scheme, using Jaeger to collect events in Kubernetes cluster and show them. The final effect is as follows:
When writing that article, I set up a flag to introduce the principle of it in detail, pigeon for a long time, now at the end of the year, it should also be issued.
An overview of Eents
Let’s start with a simple example to see what events are in the Kubernetes cluster.
Create a new namespace named Moelove, and then create a Deployment called redis in it. Next, look at all events in this namespace.
(MoeLove) ➜ kubectl create NS MoeLove Namespace/MoeLove Created (MoeLove) ➜ kubectl -n MoeLove create Deployment redis -- image = GHCR. IO/moelove/redis: alpine deployment. The apps/redis created (moelove) ➜ kubectl -n moelove get deploy NAME READY Up-to-date AVAILABLE AGE Redis 1/1 11 11s (MoeLove) ➜ kubectl -n MoeLove get events LAST SEEN TYPE REASON OBJECT MESSAGE 21s Normal Scheduled pod/redis-687967dbc5-27vmr Successfully assigned moelove/redis-687967dbc5-27vmr to kind-worker3 21s Normal Pulling pod/redis-687967dbc5-27vmr Pulling image"ghcr.io/moelove/redis:alpine"
15s Normal Pulled pod/redis-687967dbc5-27vmr Successfully pulled image "ghcr.io/moelove/redis:alpine" in 6.814310968s
14s Normal Created pod/redis-687967dbc5-27vmr Created container redis
14s Normal Started pod/redis-687967dbc5-27vmr Started container redis
22s Normal SuccessfulCreate replicaset/redis-687967dbc5 Created pod: redis-687967dbc5-27vmr
22s Normal ScalingReplicaSet deployment/redis Scaled up replica set redis-687967dbc5 to 1
Copy the codeHowever, by default kubectl get Events is not arranged in the order in which events occur, So we often need to increase – sort – by = ‘{. Metadata. CreationTimestamp}’ parameters to make the output can be arranged according to time.
This is why kubectl alpha events has been added to Kubernetes V1.23. I in the previous article “K8S ecological weekly | Kubernetes v1.23.0 release, new features in” has carried on the detailed introduction, here is not opened.
Sorted by time, you can see the following results:
➜ kubectl -n MoeLove get events --sort-by='{.metadata.creationTimestamp}'
LAST SEEN TYPE REASON OBJECT MESSAGE
2m12s Normal Scheduled pod/redis-687967dbc5-27vmr Successfully assigned moelove/redis-687967dbc5-27vmr to kind-worker3
2m13s Normal SuccessfulCreate replicaset/redis-687967dbc5 Created pod: redis-687967dbc5-27vmr
2m13s Normal ScalingReplicaSet deployment/redis Scaled up replica set redis-687967dbc5 to 1
2m12s Normal Pulling pod/redis-687967dbc5-27vmr Pulling image "ghcr.io/moelove/redis:alpine"
2m6s Normal Pulled pod/redis-687967dbc5-27vmr Successfully pulled image "ghcr.io/moelove/redis:alpine" in 6.814310968s
2m5s Normal Created pod/redis-687967dbc5-27vmr Created container redis
2m5s Normal Started pod/redis-687967dbc5-27vmr Started container redis
Copy the codeThrough the above operations, we can see that Events is actually a resource in the Kubernetes cluster. New events can be generated when the resource state changes in the Kubernetes cluster.
Further Events
Single Event object
Since Events is a resource in the Kubernetes cluster, it would normally include its name in its metadata.name for separate operations. So we can print its name with the following command:
➜ kubectl -n MoeLove get events --sort-by='{.metadata.creationTimestamp}' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'VMR dbc5 redis - 687967-27. 16 c4fb7bde8c69d2 redis - 687967 dbc5. 16 c4fb7bde6b54c4 redis. 16 c4fb7bde1bf769 redis-687967dbc5-27vmr.16c4fb7bf8a0ab35 redis-687967dbc5-27vmr.16c4fb7d8ecaeff8 redis-687967dbc5-27vmr.16c4fb7d99709da9 redis-687967dbc5-27vmr.16c4fb7d9be30c06Copy the codeSelect any one of these event records and output it to YAML format for viewing:
(MoeLove) ➜ kubectl -n MoeLove get events Redis-687967DBc5-27vmr. 16c4fb7bde8c69d2 -o Yaml Action: Binding apiVersion: v1 eventTime:"The 2021-12-28 T19: away. 702987 z"
firstTimestamp: null
involvedObject:
apiVersion: v1
kind: Pod
name: redis-687967dbc5-27vmr
namespace: moelove
resourceVersion: "330230"
uid: 71b97182-5593-47b2-88cc-b3f59618c7aa
kind: Event
lastTimestamp: null
message: Successfully assigned moelove/redis-687967dbc5-27vmr to kind-worker3
metadata:
creationTimestamp: "2021-12-28T19:31:13Z"
name: redis-687967dbc5-27vmr.16c4fb7bde8c69d2
namespace: moelove
resourceVersion: "330235"
uid: e5c03126-33b9-4559-9585-5e82adcd96b0
reason: Scheduled
reportingComponent: default-scheduler
reportingInstance: default-scheduler-kind-control-plane
source: {}
type: Normal
Copy the codeYou can see that there’s a lot of information in there, but WE won’t expand it here. Let’s look at another example.
kubectl describeThe Events in
We can perform describe operations on both the Deployment object and the Pod object separately and get the following results (intermediate output is omitted) :
- The Deployment operation
(MoeLove) ➜ kubectl -n MoeLove describe deploy/ Redis Name: Redis Namespace: MoeLove Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 15m deployment-controller Scaled up replicaset redis-687967dbc5 to 1
Copy the code- The Pod operation
(MoeLove) ➜ kubectl-n MoeLove describe Pods RedIS-687967DBc5-27vMR Name: RedIS-687967dbc5-27vMR Namespace: moelove Priority: 0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 18m default-scheduler Successfully assigned moelove/redis-687967dbc5-27vmr to kind-worker3 Normal Pulling 18m kubelet Pulling image"ghcr.io/moelove/redis:alpine"
Normal Pulled 17m kubelet Successfully pulled image "ghcr.io/moelove/redis:alpine" in 6.814310968s
Normal Created 17m kubelet Created container redis
Normal Started 17m kubelet Started container redis
Copy the codeWe can find that when we describe different resource objects, the events content we can see are directly related to ourselves. Pod-related Events are not available when describing Deployment.
This shows that the Event object contains information about the resource object it describes, and they are directly related.
Combined with the single Event object we saw above, we find that the content in the involvedObject field is the information of the resource object associated with the Event.
Further understanding of Events
Let’s look at the following example, creating a Deployment with a non-existent image:
(MoeLove) ➜ kubectl-n MoeLove create Deployment non-exist --image=ghcr. IO/MoeLove /non-exist deployment.apps/non-exist Created (MoeLove) ➜ kubectl-n MoeLove get Pods NAME READY STATUS RESTARTS AGE non-existent - d9dDBDD84-TNrhd 0/1 ErrImagePull 0 11s redis-687967dbc5-27vmr 1/1 Running 0 26mCopy the codeWe can see that the current Pod is in an ErrImagePull state. View events in the current namespace (I omitted the previous deploy/redis record)
➜ kubectl -n MoeLove get events --sort-by='{.metadata.creationTimestamp}'
LAST SEEN TYPE REASON OBJECT MESSAGE
35s Normal SuccessfulCreate replicaset/non-exist-d9ddbdd84 Created pod: non-exist-d9ddbdd84-tnrhd
35s Normal ScalingReplicaSet deployment/non-exist Scaled up replica set non-exist-d9ddbdd84 to 1
35s Normal Scheduled pod/non-exist-d9ddbdd84-tnrhd Successfully assigned moelove/non-exist-d9ddbdd84-tnrhd to kind-worker3
17s Warning Failed pod/non-exist-d9ddbdd84-tnrhd Error: ErrImagePull
17s Warning Failed pod/non-exist-d9ddbdd84-tnrhd Failed to pull image "ghcr.io/moelove/non-exist": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/moelove/non-exist:latest": failed to resolve reference "ghcr.io/moelove/non-exist:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
18s Normal Pulling pod/non-exist-d9ddbdd84-tnrhd Pulling image "ghcr.io/moelove/non-exist"
4s Warning Failed pod/non-exist-d9ddbdd84-tnrhd Error: ImagePullBackOff
4s Normal BackOff pod/non-exist-d9ddbdd84-tnrhd Back-off pulling image "ghcr.io/moelove/non-exist"
Copy the codePerform the describe operation on this Pod:
(MoeLove) ➜ kubectl-n MoeLove describe Pods Non-existent D9dDBDD84-tnrhd... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m default-scheduler Successfully assigned moelove/non-exist-d9ddbdd84-tnrhd to kind-worker3 Normal Pulling 2m22s (x4 over 3m59s) kubelet Pulling image"ghcr.io/moelove/non-exist"
Warning Failed 2m21s (x4 over 3m59s) kubelet Failed to pull image "ghcr.io/moelove/non-exist": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/moelove/non-exist:latest": failed to resolve reference "ghcr.io/moelove/non-exist:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
Warning Failed 2m21s (x4 over 3m59s) kubelet Error: ErrImagePull
Warning Failed 2m9s (x6 over 3m58s) kubelet Error: ImagePullBackOff
Normal BackOff 115s (x7 over 3m58s) kubelet Back-off pulling image "ghcr.io/moelove/non-exist"
Copy the codeWe can see that the output here is different from that of running Pod correctly before. The main difference is with the Age column. Here we see something like 115s (X7 over 3M58s).
This type of event has occurred seven times in 3M58s, and the latest event occurred 115s ago
However, when we go to kubectl get Events directly, we don’t see events that are repeated 7 times. This means that Kubernetes will automatically merge duplicate events.
Select the last Events (method described above) and print its contents in YAML format:
(MoeLove) ➜ kubectl-n MoeLove get Events non-existent - d9dDBDD84-tnRhd.16c4fce570CFba46 -o Yaml apiVersion: v1 count: 43 eventTime: null firstTimestamp:"2021-12-28T19:57:06Z"
involvedObject:
apiVersion: v1
fieldPath: spec.containers{non-exist}
kind: Pod
name: non-exist-d9ddbdd84-tnrhd
namespace: moelove
resourceVersion: "333366"
uid: 33045163-146e-4282-b559-fec19a189a10
kind: Event
lastTimestamp: "2021-12-28T18:07:14Z"
message: Back-off pulling image "ghcr.io/moelove/non-exist"
metadata:
creationTimestamp: "2021-12-28T19:57:06Z"
name: non-exist-d9ddbdd84-tnrhd.16c4fce570cfba46
namespace: moelove
resourceVersion: "334638"
uid: 60708be0-23b9-481b-a290-dd208fed6d47
reason: BackOff
reportingComponent: ""
reportingInstance: ""
source:
component: kubelet
host: kind-worker3
type: Normal
Copy the codeHere we can see that the field includes a count field, indicating how many times the same event has occurred. And firstTimestamp and lastTimestamp respectively represent the time when the event first appeared last time. This explains the duration of events in the previous output.
Understand Events thoroughly
The following is a random selection from Events, and we can see that it contains some field information:
apiVersion: v1
count: 1
eventTime: null
firstTimestamp: "2021-12-28T19:31:13Z"
involvedObject:
apiVersion: apps/v1
kind: ReplicaSet
name: redis-687967dbc5
namespace: moelove
resourceVersion: "330227"
uid: 11e98a9d-9062-4ccb-92cb-f51cc74d4c1d
kind: Event
lastTimestamp: "2021-12-28T19:31:13Z"
message: 'Created pod: redis-687967dbc5-27vmr'
metadata:
creationTimestamp: "2021-12-28T19:31:13Z"
name: Redis - 687967 dbc5. 16 c4fb7bde6b54c4
namespace: moelove
resourceVersion: "330231"
uid: 8e37ec1e-b3a1-420c-96d4-3b3b2995c300
reason: SuccessfulCreate
reportingComponent: ""
reportingInstance: ""
source:
component: replicaset-controller
type: Normal
Copy the codeThe meanings of the main fields are as follows:
- Count: Indicates how many times the current type of event has occurred (as described earlier)
- InvolvedObject: A resource object (described previously) directly associated with this event, with the following structure:
type ObjectReference struct {
Kind string
Namespace string
Name string
UID types.UID
APIVersion string
ResourceVersion string
FieldPath string
}
Copy the code- Source: Directly associated component, structured as follows:
type EventSource struct {
Component string
Host string
}
Copy the code- Reason: A simple summary (or a fixed code) is more suitable for filtering conditions, mainly to make machine readable. There are currently over 50 such codes;
- Message: Give a detailed explanation that is easier to read
- Type: Indicates the current value
Normal和WarningTwo types, the source code has also written its meaning:
// staging/src/k8s.io/api/core/v1/types.go
const (
// Information only and will not cause any problems
EventTypeNormal string = "Normal"
// These events are to warn that something might go wrong
EventTypeWarning string = "Warning"
)
Copy the codeTherefore, when we collect these Events as the span of tracing, we can classify them according to their source, associate them according to involvedObject, and sort them according to time.
conclusion
In this article, I take a closer look at the actual role of the Events object and the meaning of its individual fields by using two examples, one with correctly deployed Deploy and one with no images.
For Kubernetes, Events contain a lot of useful information, but this information does not affect Kubernetes and is not the actual Kubernetes log. By default, logs in Kubernetes are cleaned up after 1 hour to free up resources on ETCD.
Therefore, in order to better let the cluster administrator know what is happening, in the production environment, we usually collect the Kubernetes cluster events back. My personal recommendation is: github.com/opsgenie/ku…
Of course, you can also follow my previous article “More Elegant Kubernetes Cluster Event Measurement Scheme” to collect events in Kubernetes cluster by Using Jaeger tracing method and show them.
Please feel free to subscribe to my official account [MoeLove]
