Ops & troubleshooting / 2026-08-16
When the clock drifts, certificates "expire" out of nowhere
Certificates and config are fine but the browser says insecure — check the clock first. A few minutes of drift is enough to break TLS validation.
A machine that just came back from sleep, a VM that was suspended for a long time, a laptop that's been offline for weeks — any of them can have a skewed system clock. The symptom: visiting an HTTPS site reports an invalid certificate, while the certificate itself is perfectly fine.
Measure the drift first, then fix it
timedatectl status
chronyc tracking
The offset on the System time line is the key number. Tens of milliseconds is normal, seconds need attention, and past a few minutes problems are basically guaranteed.
Why virtual machines drift so easily
- When the host hibernates/suspends, the guest clock doesn't advance
- Under heavy load, clock interrupts get delayed and drift accumulates
- Some cloud providers' time sources are unstable
In PVE you can install qemu-guest-agent in the VM and let the host sync time into it; containers share the host clock directly and need no extra handling.
A minimal chrony config
# /etc/chrony/chrony.conf
pool ntp.example.org iburst
makestep 1.0 3
rtcsync
makestep 1.0 3 means: during the first three syncs after startup, if the offset exceeds 1 second, step the clock directly instead of slewing it. Very useful for a machine that just resumed from suspend.
Troubleshooting order
timedatectl— check the timezone and whether NTP is enabledchronyc sources -v— check whether any source is usable (^*marks the currently selected one)- If neither works, check whether outbound UDP 123 is blocked
- On a cloud host, confirm the security group isn't blocking NTP
The lowest-effort fallback
Add a cron entry on critical machines to check the time once an hour and notify when the offset is large:
*/60 * * * * chronyc tracking | awk '/System time/ {print $4, $5}' >> /var/log/clock.log
TLS, Kerberos, two-factor authentication, log correlation — all of it depends on the clock. When this one thing breaks, symptoms pop up everywhere, but there is only one root cause.
Wrong time makes certificates "expired"
TLS validation uses the local machine's time. If the system clock is off by two years, the browser says the certificate has expired; if it's before the issuance date, it says the certificate is not yet valid. Both errors look like certificate problems but are actually clock problems.
The troubleshooting order should be: check date first, then the certificate.
date -u
timedatectl status
openssl x509 -in cert.pem -noout -dates
A minimal chrony config
# /etc/chrony/chrony.conf
pool ntp.aliyun.com iburst
pool time.cloudflare.com iburst
makestep 1.0 3
rtcsync
iburst makes it take samples quickly at startup, and makestep 1.0 3 allows the first 3 syncs to step the clock directly (step if the offset exceeds 1 second). Without makestep, a machine with a large offset spends hours "catching up slowly", and certificates keep failing the whole time.
Verify it actually synced
chronyc tracking # check System time offset, Leap status
chronyc sources -v # status of each source, ^* is the currently selected one
Leap status must be Normal. Not synchronised means not a single source connected — usually UDP 123 blocked by a firewall.
A window that's easy to overlook
If the certificate renewal job runs on a machine with an inaccurate clock, ACME's signature timestamp check fails and reports "bad nonce" or "JWS verification error", which has nothing to do with the certificate itself. Sync the clock first, then read the ACME logs — it saves half the troubleshooting time.