Ops & troubleshooting / 2026-05-30
Four journalctl filtering moves: turning three thousand log lines into three
When troubleshooting, the real time-saver isn't reading logs — it's removing the irrelevant lines. Four flag combinations cover most everyday problems.
When a machine goes wrong, journalctl scrolls screen after screen while you hunt for keywords by eye — extremely inefficient. The real trick is to narrow the scope first, then read what's left.
Move one: by time window
# The ten minutes before and after the incident — the most common case
journalctl --since "10 min ago"
# An explicit range
journalctl --since "2026-05-29 22:10" --until "2026-05-29 22:30"
A time window assumes you know when things started going wrong. If you don't, check uptime and the last reboot time, then read forward from there.
Move two: by unit and priority
journalctl -u nginx -u php-fpm --since "1 hour ago"
journalctl -p err -b # every error and above since this boot
-b means "this boot"; paired with -p err it lists every real error since startup in one go. This combination is almost always my first step in any troubleshooting.
Move three: by kernel and hardware
journalctl -k -b | grep -iE "error|warn|ata|nvme|oom"
Disks dropping off the bus, NIC resets, OOM killing processes — all of that lives only in the kernel log. If the application log is clean but the service restarted for no reason, this is where you look nine times out of ten.
Move four: structured output
journalctl -u myapp -o json | jq -r '.MESSAGE' | sort | uniq -c | sort -rn | head
When the same error floods the screen, count first, read second. Usually the top three lines are the root cause and the thousands after them are just repeats of it.
Two settings while you're here
You can cap long-term log usage; otherwise /var/log/journal slowly eats the root partition:
# /etc/systemd/journald.conf
SystemMaxUse=500M
MaxRetentionSec=2week
Then systemctl restart systemd-journald.
One more habit: once you're done troubleshooting, paste the key log excerpts into your notes. Next time the same symptom shows up, searching what you already wrote is much faster than paging through it all again.
Order matters more than flags
The most common mistake is opening with journalctl -xe and then scrolling through dozens of screens. The order that works is: fix the time window, then the unit, then the keyword.
# 1. Time window: only the 5 minutes around the failure
journalctl --since "2026-10-18 09:20" --until "2026-10-18 09:30"
# 2. Unit: pin down which service
journalctl -u nginx --since "-30m" -p warning
# 3. Keyword: pin down the error type
journalctl -u php-fpm8.2 --since "-2h" | grep -iE "oom|segfault|timeout"
Three steps and you usually have the unit within about 3 minutes; only then do you look at the details.
Four common moves
| Purpose | Command |
|---|---|
| Logs since the most recent boot | journalctl -b |
| The previous boot (before the crash) | journalctl -b -1 |
| Filter by priority | journalctl -p err -b |
| Follow a unit in real time | journalctl -u x -f |
-b -1 is the most useful one: after the service restarts, the crash scene is no longer in the current boot's log, so you have to go back to the previous boot.
Disk and retention
/var/log/journal keeps growing. Set a cap:
journalctl --vacuum-size=200M
journalctl --vacuum-time=14d
--vacuum-size means "shrink the total to 200M"; --vacuum-time means "keep only 14 days". Put both in /etc/systemd/journald.conf (SystemMaxUse=, MaxRetentionSec=), otherwise the setting is lost on restart.
One diagnostic trick
When OOM, segfault or connection reset by peer shows up in the log, don't change the config yet — look at frequency and time window first. The same error once every 5 minutes is usually a health check or a scheduled job; a burst concentrated in a single minute is a real failure. The frequency itself is the clue.