Data loss prevention 101: what DLP actually detects

Data loss prevention sounds like a filter that inspects every byte leaving your network and stops the bad ones. In practice, DLP is a pattern-matching pipeline with specific strengths and specific blind spots, and teams that don’t understand the mechanics tend to over-trust it. A DLP product that misses a leak isn’t necessarily broken — it never knew the value was sensitive. Understanding what DLP actually detects is the difference between a tool you rely on and a tool you audit.

What a detection pipeline looks like

Modern DLP engines detect secrets in three layers, each with different error characteristics:

Keyword pre-filtering. Before anything expensive runs, the engine looks for cheap signals: api_key, password, token, BEGIN PRIVATE KEY, AKIA (the AWS access key prefix). A keyword filter alone has no idea whether password = "sunshine" is a real credential or a joke in a config comment.

Regular expression rules. The engine matches known formats: AWS access keys, GitHub tokens, Slack webhooks, private key headers. Good rule sets are derived from real-world scans and tuned for both recall (catching real keys) and precision (not flagging every 20-character hex string in a codebase).

Entropy analysis. For secrets with no fixed format, the engine computes Shannon entropy — how unpredictable the string is. Long random strings score high and get flagged; readable text passes. This catches the API key with no recognizable prefix.

The layers compose: keyword filter narrows the input, regex identifies formats, entropy catches the formatless. A detection is only as good as the weakest layer, and every layer has a false-positive problem.

What DLP misses

The honest list is long:

• Obfuscated values. Base64, URL-encoding, and hex encoding defeat plain regex matching unless the engine decodes first — and production pipelines handle only the common encodings.
• Keys with no format and low entropy. A short, human-chosen “key” looks like a word, not a secret.
• Formats the rule set has never seen. No rule collection covers every vendor’s token format.
• Context-dependent secrets. A database password that happens to be sunshine is undetectable by pattern — only by knowing the value.

This last point is why value-based detection exists as a complement: if you know the actual secret values, you can match them exactly, with zero false positives. “Is this exact string one of our known secrets?” is a much stronger question than “does this look like a secret?” The trade-off: the engine must know the values, so it needs access to the store.

Building a sane DLP posture

• Run both layers. Use pattern detection (regex plus entropy) for breadth and known-value matching for precision.
• Fail safe on configuration. A misconfigured rule set should not silently stop scanning — keep the last good configuration.
• Choose redact or block deliberately. Decide per rule whether a match is logged, masked, or blocked. Masking (showing sk-****) preserves context for debugging while hiding the value.
• Watch the false-positive rate. A DLP that flags everything gets ignored. Treat the pattern set as a living artifact.

The pattern pipeline in trustless follows this design: 40 gitleaks-derived rules (MIT, attribution in the NOTICE file) run behind a keyword pre-filter, then RE2 regex, then Shannon entropy with a default threshold of 3.5. Its serve command is a DLP reverse proxy for OpenAI-compatible endpoints (HTTPS MITM, per-host header injection) with per-rule log/mask modes, a rules_file for custom patterns, SIGHUP hot reload, and an atomic swap that keeps the previous rule set on a broken config. The same rules power its retroactive dlp scrub commands, so what you detect in flight is what you can erase later. The command surface is documented at docs/commands/.

What was the last secret your DLP missed — and did you find out from the tool, or from somewhere else?