Shannon entropy in secret detection: the math behind the threshold

Secret scanners have a false-positive problem. A regex will happily match the placeholder sk-xxxxxxxxxxxxxxxx in your documentation — and it will match a real key just as happily. Regex sees shape, not randomness. Real keys are random strings; placeholders and prose are not. That difference is measurable, and the measurement is Shannon entropy.

What entropy measures

Shannon entropy, from Claude Shannon’s 1948 paper, quantifies how unpredictable a string is. For a string of characters, the per-character entropy is:

H = −Σ p(c) log₂ p(c)

where p(c) is the probability of character c. The result is in bits per character. A string drawn uniformly from 26 lowercase letters carries about 4.7 bits per character; add uppercase and digits (62 characters) and you get roughly 5.9. That’s the neighborhood real API keys live in — random-looking, high entropy.

Human-written text is different. English prose and code typically measure in the low-to-mid 3–4 bits per character range, because letters and words are predictable. Configuration placeholders are lower still: your-api-key-here, <redacted>, example123 are highly predictable, so their entropy is low.

Why 3.5 is the default threshold

The insight is a population split. Most real secrets are random strings — high entropy — while the strings that trip scanners by accident (documentation placeholders, sample values, “changeme” tokens) are predictable — low entropy. A threshold around 3.5 bits per character separates the two populations reasonably well: placeholders fall below it, real keys clear it.

That’s why the DLP layer of trustless uses entropy as the final stage of its detection pipeline — keyword pre-filter → RE2 regex → Shannon entropy, with a default threshold of 3.5. The regex stage says “this looks like an AWS key.” The entropy stage says “and it’s random enough to actually be one.” Both must pass, and the expensive entropy calculation only runs on lines that already survived the cheap keyword and regex stages.

Per-rule override exists because formats differ. A long, dense token has different entropy characteristics than a short legacy key, so each rule can raise or lower the threshold instead of forcing one number on every pattern.

Entropy is a filter, not a detector

Entropy cannot tell a real key from a random UUID, and it misses low-entropy secrets such as short tokens or dictionary words. Its job is to cut false positives from the pattern stage, not to find everything.

And like any threshold, it’s a trade. Set it too low, and placeholders and base64 blobs get redacted from legitimate traffic. Set it too high, and short or unusually structured keys slip through. That’s why a complete setup pairs entropy with two other layers: known-value matching for secrets you already hold, and format patterns for the ones you don’t.

Where this runs in practice

A DLP proxy applies the whole pipeline per request, in order: the keyword pre-filter rejects most traffic instantly, the regex stage checks surviving lines against key formats, and entropy makes the final call. Rules can be disabled per installation when one proves noisy, replaced wholesale via a rules file, and reloaded without a restart — so the threshold tuning is a config change, not a redeployment.

trustless is a credential broker CLI for AI agents whose trustless serve proxy implements exactly this pipeline: 40 gitleaks-derived rules, keyword → RE2 → entropy, default threshold 3.5 with per-rule overrides, and log/mask modes. The DLP design is documented in the command reference.

Have you ever tuned an entropy threshold and watched a rule that used to fire on every placeholder go quiet — or, worse, found a real key that slipped through because it was too short for the math to believe?