Scanning logs for secrets: retroactive detection at scale

Prevention is the goal, but most teams discover a leaked credential the same way: someone greps the logs, a bill spikes, or a third party reports it. By then the secret has been sitting in a file for days or weeks. Retroactive detection is the cleanup pass that finds what prevention missed — and log volume changes which techniques work.

Why logs fill up with secrets

Secrets rarely enter logs through malice. A debugging flag dumps request headers. A build script prints DATABASE_URL. An HTTP client logs a full URL including the token in the query string. An AI agent’s transcript captures whatever a tool printed. Each looks reasonable in the moment, and each plants a credential in a file that gets copied to backup and shared with people who never needed it.

The trouble is that you cannot predict every way a secret will leak, which is why retroactive scanning matters: it works on what already happened.

Two layers of detection

Scanning for secrets means looking for two different things:

1. Known values. If you know the exact secret — an API key you can read from your store — you can search for it directly, including base64 and URL-encoded variants. This is fast and rarely produces false positives.
2. Unknown patterns. You cannot know every key, so you also scan for the shapes secrets take: long high-entropy strings near keywords like api_key or token.

The two layers complement each other: known-value scans catch exact leaks, pattern scans catch things you did not know were there. A scanner that does only one of the two will miss most real leaks.

Patterns need a pipeline

Running every rule against every line of every file does not scale. Scanners use a pipeline: a cheap keyword pre-filter discards lines that cannot contain a secret, a regex layer matches candidate shapes, and an entropy check separates real credentials from benign high-entropy strings like UUIDs.

Entropy deserves caution. A threshold too low floods you with false positives; too high, and short secrets slip through. Entropy works best as a second opinion on top of regex matches, not as a standalone detector. And remember encoding: a key in base64 looks nothing like the plaintext key, so scanning must expand the input it checks.

Making the scan safe to run

Retroactive scanning has its own risks. A bug in a redaction tool can corrupt files; an aggressive pattern can rewrite legitimate data. Good hygiene: run in dry-run mode first, review what would change, keep backups, and only then apply. For databases, plain updates leave the old values in pages and indexes — a real erase needs a rebuild of the full-text index and a compaction pass so the bytes do not linger.

This is the design behind trustless dlp scrub-db and scrub-text: known-value and pattern layers, dry-run by default, --apply and --backup for the real pass, and an FTS rebuild plus VACUUM so erased values do not linger. The scans draw on a bundled rule set derived from gitleaks — 40 rules covering common key formats, with attribution in the project’s NOTICE — and you can disable or extend rules per deployment.

Scanning should be routine

Retroactive detection is not a one-time cleanup. New logs, backups, and agent transcripts accumulate daily, so the workflow that scales is one you can run on a schedule: scan, review the diff, apply with backup, and keep an audit trail. If that sounds like more process than you have, start small — scan one directory, learn your false-positive rate, then widen the net.

trustless is an open-source credential broker that includes this retroactive scrubbing alongside process-level injection and a DLP proxy; the commands docs cover the scrub commands in detail.

The question that matters before you run your first scan: do you already know which of your log files contain a live credential?