Notes
DNS TTLs are a promise, not a suggestion
2026-10-21
Every time I migrate a record I am reminded that a TTL is not a hint to resolvers, it is a contract with every cache between here and the client. Setting it to an hour "just for now" and then changing the target ten minutes later means somebody, somewhere, is still talking to the old address and will keep doing so for the rest of the hour.
The habit that finally stuck: drop the TTL to a few minutes a day before the change, wait for the old value to age out, make the change, and only then put the TTL back. It is unglamorous and it works. The alternative is an afternoon of explaining to yourself why one region behaves differently from the other.
Reading logs when nothing looks broken
2026-09-30
The most useful log line is usually the one from a healthy morning, not the one from the outage. If you only look at logs after something fails, every line looks suspicious and you have no baseline to compare against.
What helped was keeping a week of routine output — a few counters, a couple of timings, nothing elaborate — and glancing at it once a day. After a while the shape becomes familiar, and the day it changes you notice within minutes instead of hours.
The backup you have tested is the only backup
2026-09-12
A file that gets copied somewhere every night is a copy, not a backup. The difference only shows up on the day you need it, which is the worst possible time to discover that the copy has been empty since March.
Restoring one file into a scratch directory once a month is enough. It takes a few minutes, it catches the boring failures — wrong permissions, missing paths, a job that silently stopped — and it means the real restore is a procedure you have already performed rather than one you are improvising.
Clocks drift, and everything downstream believes them
2026-08-27
Two machines disagreeing by ninety seconds is enough to make tokens look expired, certificates look invalid and log timelines look impossible. Nothing crashes outright, which is what makes it annoying: things just fail in ways that do not point at the clock.
Turning on time synchronisation and then actually checking it — rather than assuming the default is doing something — closed a class of bugs I had been blaming on the application for months.