Secret management secures sensitive credentials (API keys, database passwords, TLS certificates, encryption keys) throughout their lifecycle: generation, storage, distribution, rotation, and revocation. Ad-hoc secret management (hardcoded secrets, .env files in repos) creates security vulnerabilities and operational toil. A dedicated secret management system centralizes control and enables automated rotation.
HashiCorp Vault Architecture
Vault is the dominant open-source secret management system. Core components: storage backend (persists encrypted data — Consul, DynamoDB, or Raft integrated storage), encryption layer (encrypts all data at rest using a master key), seal/unseal mechanism (master key is split using Shamir's Secret Sharing; requires N of M key shares to unseal on startup), and secret engines (plugins for different secret types).
Dynamic Secrets
Static secrets (same password shared by all consumers) are hard to rotate and audit. Vault generates dynamic secrets on demand: when a service requests a database credential, Vault creates a new database user with a short TTL (e.g., 1 hour), returns the credentials, and automatically revokes the user when the TTL expires. Each service instance gets unique credentials; compromise of one set doesn't affect others. The database secret engine supports PostgreSQL, MySQL, MongoDB, and others.
Authentication Methods
Services authenticate to Vault using platform identity rather than static credentials: Kubernetes auth (Vault validates the pod's service account token via the Kubernetes API), AWS IAM auth (Vault validates an AWS STS token; IAM role determines what secrets the service can access), GCP/Azure equivalents. This eliminates the bootstrap credential problem: the only secret the service needs is its platform identity, which is issued by the platform itself.
Secret Injection Patterns
Three injection patterns for containerized services: Vault Agent sidecar (runs alongside the application container, authenticates to Vault, fetches secrets, writes them to a shared volume as files or environment variables, renews leases automatically), Vault CSI driver (mounts secrets as volume files via Kubernetes CSI), or native SDK integration (application code calls Vault API directly — most control, most coupling). The sidecar pattern is most common as it requires no application code changes.
Envelope Encryption
Envelope encryption uses two keys: a data encryption key (DEK) that encrypts the data, and a key encryption key (KEK, stored in a KMS like AWS KMS or GCP Cloud KMS) that encrypts the DEK. The encrypted DEK is stored alongside the data. To decrypt: fetch the encrypted DEK, call KMS to decrypt it using the KEK, use the DEK to decrypt the data. The plaintext DEK is never stored; KMS access controls limit who can decrypt the DEK, providing an additional access control layer.
Secret Rotation
Automated rotation eliminates long-lived static secrets. For database passwords: Vault generates a new password, updates the database, updates all consumers (by short-TTL dynamic credentials or by Vault Agent refreshing the secret file), then invalidates the old password. For API keys: rotate by creating a new key, distributing it, confirming consumption (observing API calls using the new key), then revoking the old key. Blue-green rotation maintains two valid keys simultaneously during the transition window.
Audit Logging
Every Vault operation is recorded in an audit log: who requested a secret, which secret, when, from which IP, and whether it was granted or denied. Audit logs are append-only and written to multiple sinks (file, syslog) for tamper resistance. SIEM (Security Information and Event Management) ingests Vault audit logs to detect anomalies: service S accessing a secret it has never accessed before, high-frequency secret fetches, access from unexpected IP ranges.