Token expiry handling in automation: the silent breaker

Most automation failures are loud — a crash, a non-zero exit, a stack trace. Token expiry is not one of them. It fails quietly, at the worst possible moment, with a 401 that looks like an outage, a permissions problem, or a network hiccup. By the time anyone notices, the retry loop has been hammering the API for hours. Expiry is the silent breaker of unattended systems.

Why expiry fails silently

An access token is a promise with a timestamp. The failure modes follow from that:

Timezones and clock skew. The token says it expires at a UTC instant; your server reads local time, and the issuing machine’s clock drifts. A token with a minute of life is already rejected.

Caches go stale at different rates. Tokens get copied into config files, caches, and process environments. Each copy expires at the same instant, but each consumer discovers it at a different time — so one outage surfaces as unrelated errors across services.

Retries hide the cause. The first 401 triggers a retry, which triggers another, each with backoff. The log fills with transport errors while the real message — “this credential is dead” — appears only if someone reads the response body.

Reauth needs a human. Refresh tokens get revoked by the user, by the provider, or by a security policy. From then on, every refresh fails with invalid_grant, and retrying changes nothing. The automation now needs a person to re-approve — a fact it communicates by failing forever.

The retry storm

The expensive version is the retry storm. A client that treats a 401 as transient and retries — or re-fetches a new token on every retry — hammers the authorization server while making no progress. If refresh tokens are not rotated on use, retries also replay consumed tokens. The outage has two victims: your job and the auth endpoint’s rate limits.

Handling expiry on purpose

Refresh before expiry, with leeway. Do not wait for a 401 to discover the token is dead. Refresh while the token still has minutes of life, using a leeway that absorbs clock skew. Expiry then becomes a scheduled, testable event instead of a runtime failure.

One refresh loop, not many. The worst architecture has every caller refreshing independently — N callers, N token caches, N points of staleness. One component should own the refresh cycle and hand out fresh access tokens at use time, so callers never see a refresh token, never cache one, and never reason about expiry.

Treat invalid_grant as terminal. A revoked refresh token is not a transient error; it means reauthorization. Stop retrying, mark the credential reauth_required, and surface it to a human. Automation that distinguishes “expired, refreshable” from “revoked, reauth needed” pages a person instead of burning API quota.

Alert on the boundary, not the noise. The signal to monitor is not “401 happened” but “refresh failed” and “reauth required” — the events that predict a long outage.

What this looks like in a credential broker

Credential brokers use this model: tokens live in the credential store, refresh happens at the credential layer, and consumers get a fresh access token at use time — never cached, never in agent context. Status reporting distinguishes valid, expired, and reauth_required, so automation can treat the three cases differently. The invalid_grant case is never retried; it is reported once and left for a human.

The test of your expiry handling is not whether it works on a sunny Tuesday, but whether the 3am failure tells you what happened — and whether the fix is a button or a deploy. trustless implements this refresh-and-status model for OAuth entries, and it is a good baseline for whatever you run today.

When was the last time an expired token took down one of your jobs — and how long did it take to find the cause?