API key rotation goes wrong in a predictable way. A new key is issued, the old key is revoked, and somewhere between those two events a client makes a request with the key that used to work. If the two events aren't separated by a real window of time, that request fails, and it fails in production, for a customer who did nothing wrong. The fix isn't a smarter client. It's a server-side contract: for a defined interval, two keys are valid for the same principal at once.

This article works through that contract in detail: the four phases of a safe rotation, how to handle asynchronous consumers and edge caches that don't know a rotation is happening, what to watch to know when it's safe to revoke, and what to do when it isn't.

The rotation contract

Every safe rotation rests on one guarantee: a principal always has at least one valid key. Naive rotation violates this by construction, swapping keys in a single write. The alternative is a dual-key acceptance window: the authentication layer accepts requests signed with either the outgoing key or the incoming key for the duration of the rotation. Neither key is privileged. Both are checked against the same scopes and rate limits. The only difference tracked internally is a rotation state on each key: active, expiring, or revoked.

This sounds obvious once written down, but it has a real cost: your key-verification path now does up to two lookups instead of one, and your key storage needs to represent "two active keys, one principal" as a normal, not exceptional, state. A single api_key column on the account row usually needs to become a small keys table keyed by principal before rotation can be done safely at all.

Four phases: issue, verify, converge, revoke

Issue. A new key is generated and returned to the account owner, typically through a dashboard action or an API call they make themselves. The old key is not touched. Both keys are now active. Nothing in production changes yet; this phase only creates the option to rotate.

Verify. The account owner starts using the new key. This is the phase most teams skip when they are in a hurry, and it is the one that catches the real failures: a key baked into a container image that is hard to redeploy, a key hardcoded in a partner's integration, a background job that reads the key once at startup and never again. Verification is best driven by traffic, not by a timer: track requests per key per principal, and don't move on until the new key is carrying meaningful volume.

Converge. The old key is marked expiring. It still works, but every response carries a deprecation signal so remaining callers get a nudge before the key stops working entirely. This phase waits for old-key traffic to approach zero, not for a fixed amount of elapsed time. A key with steady, real traffic that hasn't dropped to near-zero after a week is telling you there's a caller you haven't accounted for.

Revoke. Once old-key traffic has been at zero, or a deliberately accepted noise floor, for a defined observation period, the old key moves to revoked and stops authenticating. It is a hard cutover, but by the time it happens the cutover has no observable effect, because nothing is depending on the key anymore.

Async and queued workloads

The phase model above assumes each request checks the key at the moment it is made. Two categories of traffic break that assumption.

Queued and delayed jobs. A message enqueued with an old key's credentials embedded in its payload might not be processed until hours after the key entered expiring, and in slower systems, after revoked. If your queue consumers re-authenticate against a live key store at processing time, this is a non-issue: the phase model applies at the point of use, not the point of enqueue. If credentials are baked into the message body itself, a common shortcut in webhook-delivery and batch systems, you need either a converge window sized to your maximum queue depth, or a migration to re-fetch credentials at dequeue time before you can rotate safely at all.

Edge and CDN auth caches. Any layer that caches an authentication decision, an edge worker validating a key locally, a CDN's edge-auth feature, a reverse proxy with a local key cache, will keep accepting or rejecting based on a stale copy of key state until its cache entry expires. Two failure modes follow: a revoked key can still work at the edge past your intended revocation time, and a brand-new key can be rejected at the edge before its issuance has propagated. Keep edge cache TTLs shorter than your minimum converge window, and treat edge propagation lag as part of the rotation timeline, not a rounding error.

What "safe to revoke" looks like

Convergence is a traffic question, not a clock question, so the signal that matters is old-key request volume, broken out by principal, over time. A useful dashboard needs three things: request count on the old key per principal per hour, a trend line so a flat-but-nonzero rate is visually distinct from a decaying one, and a per-principal alert when old-key traffic hasn't moved in the expected direction after a defined period. Teams that revoke on a fixed timer regardless of traffic eventually revoke a key still in active use, usually the one behind a system nobody remembers is running.

Comparing rotation strategies

The dual-key window is the right default for long-lived API keys used by third-party integrations, where you don't control the client's deploy cadence. Continuous rotation with short-lived tokens is stronger where you own both ends of the connection and can afford the added client complexity.

Rendered from this article's Markdown source, the same diagram block used to draft it.

Rollback

Rollback in this model is simple because nothing destructive has happened yet by the time you would want to roll back. If verification surfaces a caller that can't move to the new key on schedule, the fix is to hold at converge and extend the observation window, not to revoke on the original timeline anyway. If a key was revoked prematurely, before old-key traffic actually reached zero, re-issuing under the same principal is safe as long as the previous key ID isn't reused: generate a new key, and treat the failure as a signal that the converge dashboard's alert threshold needs tightening.

The only rollback that is genuinely hard is one where a key was revoked and the calling system doesn't have a redeploy path to pick up a new one quickly. That is an argument for pairing rotation policy with an incident-response runbook for the account owner, not an argument against rotation itself.

Common mistakes

A few patterns account for most rotation incidents: revoking on a fixed timer instead of an observed-traffic threshold; treating queued or cached authentication as instantaneous when it isn't; rotating every principal on the platform in a single batch instead of staggering rollout so a systemic bug affects a handful of accounts, not all of them; and shipping the dual-key acceptance logic without ever exercising it in staging with two genuinely different keys, so the first real test happens in production.

None of these require sophisticated tooling to avoid. They require treating rotation as a monitored process with an explicit state machine, not a one-line change to swap a credential.