LLM gateway security: routing, keys, and audit
An LLM gateway is the chokepoint every prompt, completion, and API key passes through. That makes it the most useful security control you own — and the most dangerous component if it leaks. Three problems dominate the design space: how requests get routed, how keys are stored, and what gets recorded.
Routing: where keys are attached
Most OpenAI-compatible gateways attach credentials at the transport layer. The gateway holds a key per upstream provider and injects it as a header when it forwards a request. This is the right idea: clients never see the upstream key, and you can rotate provider keys without touching every agent.
The details matter. Per-host injection means you can point different agent workloads at different providers or models while keeping a single entry point. If the gateway only supports one global key, a compromised agent effectively has the whole fleet’s budget. And if the gateway logs request bodies before injection, keys leak into your observability stack as a side effect.
Keys: storage and handling
A gateway config file with plaintext provider keys is a credential database with no access control. Anyone who can read the config — a teammate, a CI runner, a backup — gets every key. Treat gateway config like a vault: keys belong in a password store, and the gateway should resolve them at startup or on reload rather than persisting them in a repository.
Check how keys travel, too. Environment variables are preferable to command-line arguments, which are visible in process listings and shell history. If the gateway uses a session token, that token should arrive through an environment variable, and the gateway should fail closed — refuse requests when the value is invalid — rather than degrade silently.
Audit: what the gateway records
The gateway sees everything: who called which model, with what prompt, and when. An audit log that records events without token values gives you accountability without turning the log itself into a secret store. Useful events include every injection (which key was attached to which request), every denial (why a request was blocked), and every redaction (which pattern fired). Append-only JSONL with restrictive permissions survives log rotation and gives compliance reviewers something to query.
DLP at the gateway
A gateway is also the last place to catch secrets before they leave your network. Pattern-based redaction — keyword filters, regular expressions, and entropy scoring for values that look random — can mask or block credentials embedded in prompts and responses. Two-layer redaction (known values plus patterns) catches both the exact token you already know about and the ones you do not. Hot reload for rule changes, with a fail-safe that keeps the previous config on error, lets you tune detection without restarts.
A concrete shape
You can get all three properties from a local reverse proxy in front of any OpenAI-compatible endpoint: per-host header and query injection, HTTPS MITM so the gateway can actually inspect traffic, and an audit log that never contains token values. That is exactly what trustless serve provides, and the audit events it emits — proxy.inject, proxy.deny, dlp.redact — map one-to-one onto the categories above. The bundled DLP rules derive from gitleaks, so you do not have to hand-write the first hundred patterns. The source is on GitHub at https://github.com/ikkun1222/trustless, and the docs at https://trustless-security.com/docs/ cover the command reference.
None of the pieces are exotic: per-host routing, keys resolved from a store rather than a config file, and an audit log that records decisions instead of secrets. The question is whether your gateway has all three. When did you last check what your gateway logs — and who can read it?