Home › Guides › How to group repeated errors in a log

How to group repeated errors in a log

Step 1 — Assemble entries before counting anything

A log line is not a log entry. A Java exception arrives as a header line, an exception line, a dozen frame lines and possibly a Caused by: chain. Python prints a header, then a traceback block whose final line is the actual exception. If you count lines, one incident becomes forty "problems"; if you count headers without their traces, you lose the throw site — the single most identifying part. Assembly rule of thumb: a new entry starts at a line that looks like a fresh log record (timestamp, level, logger), and everything indented, framed, or chained after it belongs to that entry.

Step 2 — Mask what varies between occurrences

Two occurrences of the same problem differ in their values, not their shape. Replace, in a fixed order: timestamps → <ts>; UUIDs → <uuid>; IPs → <ip>; memory addresses and hex strings → <addr>/<hex>; ID-shaped tokens (pod names, session-ish strings) → <id>; line numbers in file paths → <n>; quoted literals → <str>; emails → <email>; and finally all remaining numbers → <n>. Fixed order matters: a timestamp must be claimed before the generic number rule shreds it into <n>-<n>-<n>. Note what is deliberately not masked: trailing-digit words like sha256 vs sha512 — those are different problems, not different occurrences.

Step 3 — Fingerprint: message + throw site

For entries with an exception, the fingerprint is exception class + top application frame + masked exception line. The frame is what keeps the same NullPointerException thrown from two different methods as two problems — because they are: different code, different fix. For plain messages, the fingerprint is the masked message, plus the logger/channel when the format provides one (the same words from the payment logger and the auth logger are different problems too).

Step 4 — Rank by severity × frequency, keep singletons visible

Sort groups by severity weight multiplied by log-scaled frequency (log scale so 10,000 occurrences don't bury everything else entirely), break ties by recency, and track first/last seen per group. Entries seen exactly once get their own list: a lone FATAL is often the incident, and burying singletons inside "other" is how triage tools lose them.

Doing it by hand on a small file

On a few thousand lines you can approximate this with shell tools: extract a normalised signature per entry (strip timestamps and digits with sed), then sort | uniq -c | sort -rn. It breaks down exactly where you'd expect: multi-line traces, values inside quotes, and files past a few hundred MB — which is when to use the tool above (the Error Grouper), or the streaming approach in log file too big to open. Sharing the result? Redact it first.