Database credential rotation done carelessly is one of the most common self-inflicted outages — because it's tempting to just change the password on the existing database user and update the config, which breaks every connected service the instant the password changes, before they're updated.
The naive (and risky) approach
Changing a database user's password in place, then updating each dependent service's configuration one by one, guarantees an outage window — every service still using the old password fails to connect the moment the password changes, until each one is individually updated.
The safer dual-credential approach
- Create a new database user (or new credential set) alongside the existing one — don't modify the existing user yet.
- Deploy the new credential to every consuming service.
- Confirm each service is connecting successfully with the new credential.
- Revoke the old credential only once every consumer is confirmed migrated.
This avoids any window where a still-dependent service is left with invalid credentials.
Mapping every consumer first
Before rotating, identify every service, script, and scheduled job that connects to the database — not just the obvious application servers. Forgotten consumers (an analytics pipeline, a monitoring script, an old admin tool) are the most common cause of a rotation causing unexpected breakage days later.
Dynamic secrets remove most of this complexity
Tools like Vault's dynamic database credentials generate short-lived, unique credentials per consumer automatically, with built-in expiration. This shifts rotation from a manual, coordinated event to a continuous background process — each credential simply expires and gets replaced on its own schedule, without a coordinated cutover.
Routine rotation, not just incident-driven
Waiting to rotate database credentials only after a suspected compromise means credentials are valid — and increasingly likely to have leaked somewhere — for a long time by default. A routine rotation schedule (quarterly or shorter, depending on sensitivity) limits how long any single credential remains a viable target even if it's never confirmed leaked.
After rotation
Confirm the old credential is genuinely revoked, not just unused — an old database user left active with a known-old password is still a viable target if that password was ever exposed anywhere.