Home › Guides › Kubernetes logs triage
Kubernetes logs triage
Direct answer: export the window with kubectl logs (use -p for the crashed instance), then group it. Two Kubernetes specifics make naive grouping fail — generated pod names make identical errors look unique, and klog/controller lines share the file with application lines — and both are handled in the grouper below.
Group this Kubernetes log
⚠ Raw values are visible on this screen only. Saved board data and every export stay redacted — exports never contain raw values.
Time range in file: —
This board was restored from this browser's storage — stored samples are the redacted copies by design. The raw-view toggle has no effect on restored boards.
| Severity | Problem (first sample line) | Count | First seen | Last seen |
|---|
No repeated problems — every entry in this file was seen exactly once. Check the singletons below.
Singletons — entries seen exactly once
Grouping is deterministic normalisation: timestamps, IDs, IPs and numbers are masked, then entries are matched on the masked message plus the top application frame. It groups — it does not diagnose. Check a group's samples before you act on it.
The crash-loop signature
A pod in CrashLoopBackOff produces the most lopsided board you will ever see: one problem with hundreds of occurrences, starting exactly when the rollout landed. When the board looks like that, the grouping has already told you the story — the fix is in whatever changed at first-seen: image, config, secret, or a dependency that stopped answering. Conversely, an even spread of problems across many groups points at cluster-level trouble (DNS, networking, a node misbehaving), not at one bad container.
Pod names: instance identifiers, not content
Kubernetes generates pod name suffixes per replica and per rollout, so raw-text comparison splinters every
cross-replica problem. The grouper masks tokens that behave like instance IDs — they contain a separator
(api-7d9f8b6c5-x2abc) or a digit in the middle (x2abc) — while deliberately leaving
trailing-digit words alone, because sha256 and sha512, or TLSv1 and TLSv2, are
different problems that must not merge. It is the same judgment call every log grouper makes; here it is a
documented rule you can audit, not a black box.
Go panics and klog lines
Controller and kubelet output uses klog format (E1108 09:14:01.123456 1 pod_workers.go:1298] …) —
parsed for severity and stripped to the message before grouping. A Go runtime panic prints a
panic: line, a [signal …] line, a blank line, then the goroutine stack; assembly
keeps the whole block as one FATAL entry with its top frame, instead of shattering it into a dozen
one-line groups. The general method behind all of this is in
how to group repeated errors in a log.
Frequently asked questions
Why do pod names break naive grouping?
Every pod instance has a unique generated name (api-7d9f8b6c5-x2abc). A grouper that compares raw text sees the same ImagePullBackOff error from three pods as three different problems. The fix is masking ID-shaped tokens — names with separators and digits-in-the-middle are instance identifiers, not content — while keeping genuinely distinct words (sha256 vs sha512) unmasked.
What produces the E1108 / I1108 lines in pod logs?
Those are klog lines (the Kubernetes logging library): a severity letter, a date, a time, a PID and a source position, then the message. E is an error, W a warning, I informational, F fatal. kubectl logs mixes them with your application’s own lines, which is why one file contains two grammars — the grouper parses both.
How do I export pod logs for triage?
kubectl logs <pod> --since=1h > pod.log for the incident window; add -p for the previous (crashed) instance — the crashed container’s log is usually the one you want. For a whole deployment, kubectl logs -l app=<name> --max-log-requests=20 aggregates replicas; the pod-name masking described above keeps identical errors from different replicas grouped.