Audit logging best practices: what to log and why
You cannot answer “what happened?” after an incident if nothing was recorded beforehand. Most teams discover their audit log is useless at the worst possible moment: during an investigation, when the log either lacks the events that matter or contains so much noise that the signal is buried. The problem is rarely the volume of logging. It is the design of what gets logged and why.
Start from the questions you must answer
An audit log exists to answer a fixed set of questions: who or what performed the action, when, from where, against what, and with what outcome. Design the schema around those questions rather than around whatever your framework emits by default. Every entry should carry:
- A timestamp with a timezone or UTC offset.
- An actor identifier that distinguishes a human from an automated process.
- An action name from a small, stable vocabulary.
- The target resource or command, without sensitive values.
- The outcome: success, failure, or denial.
- A correlation ID that ties the entry to the parent session.
If a field is not needed to answer one of those questions, it is a candidate for removal.
Log the event, never the secret
The most common audit logging failure is logging the credential itself. A log that records the API key it just resolved is a new leak surface: logs get copied to SIEMs, shipped to object storage, retained for months, and read by people who never needed the key. Log the fact that a credential was resolved, its identifier, and the requesting process — not the value. The same rule applies to tokens, session keys, and environment values that appear in command lines. Apply redaction in the tool that produces the log, not in post-processing; redaction after the fact is fragile and easy to bypass.
Keep the log append-only and protected
An audit log that an attacker can edit is not evidence, it is decoration. Write entries to append-only storage, restrict access to the file itself, and consider hashing or signing entries so tampering is detectable. Rotation matters too: logrotate can truncate a file while a writer still holds it open, silently losing events. A writer that reopens its log on a signal, and an entry format that tolerates rotation, keeps the trail continuous.
Pick the right verbosity
Under-logging hides incidents; over-logging hides them too, by burying meaningful events in routine noise. Log every security-relevant action — access grants, credential resolution, authentication failures, proxy denials — and keep routine operational output out of the audit stream. When in doubt, prefer a structured format such as JSON Lines: it is parseable, greppable, and safe to append to.
What this looks like in practice
A credential broker for AI agents faces exactly this design problem. Every time an agent session resolves a secret, injects it into a subprocess, or refreshes an OAuth token, that is a security-relevant event worth recording. The trustless CLI (https://github.com/ikkun1222/trustless) writes an append-only JSONL audit log with per-event types for injection, denial, and token refresh, reopens the file on SIGHUP so logrotate can rotate it safely, and never writes token values into the entries. That last property is the design decision that matters: the log answers who, what, and when without becoming a secret store itself.
Build the habit before the incident
An audit log is only as good as the discipline around it. Review a sample of entries regularly, test that rotation works, and verify that a simulated incident is reconstructible from the log alone. When the real incident arrives, you will be glad the schema was designed for questions rather than for volume.
Which events in your current logging setup would you actually want to defend in front of an auditor or an investigator?