Why Secret Management Matters
Secrets (database passwords, API keys, TLS certificates, SSH keys) are the crown jewels of any system. Hardcoded secrets in source code, stored in environment variables without rotation, or shared via email are the most common source of security breaches. A secret management system centralizes storage, controls access, enables rotation, and provides audit trails for all secret access. HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager are the primary solutions.
Core Secret Management Requirements
- Centralized storage: secrets stored encrypted at rest with strong keys (AES-256-GCM)
- Access control: fine-grained policies — service A can read only its own database password, not service B’s
- Audit logging: every secret read, write, and rotation is logged with who accessed what and when
- Rotation: secrets are periodically refreshed; old versions are gracefully retired
- Dynamic credentials: credentials created on demand, valid for a short TTL, automatically revoked
HashiCorp Vault Architecture
Vault is a cluster of servers with a storage backend (Consul, DynamoDB, or S3). The data is encrypted with a root key. The root key is split using Shamir’s Secret Sharing (N-of-M: 3-of-5 key holders must provide their share to unseal Vault after a restart). In production, Vault uses auto-unseal via AWS KMS or Azure Key Vault — on startup, Vault fetches the root key from KMS rather than requiring manual key shares.
Authentication Methods
Vault supports multiple auth methods for different contexts:
- AWS IAM auth: EC2 instances and Lambda functions authenticate using their IAM role identity. Vault verifies with AWS APIs. No long-lived token needed — the IAM role credential is the authenticator.
- Kubernetes auth: pods authenticate using their ServiceAccount JWT. Vault verifies the JWT with the Kubernetes API server. Each pod gets a short-lived Vault token scoped to its service’s policies.
- AppRole: for CI/CD pipelines — a role-id (like a username) combined with a secret-id (like a password) generates a Vault token.
Secrets Engines
# KV (Key-Value) secrets engine: simple storage
vault kv put secret/myapp/db password="s3cr3t"
vault kv get secret/myapp/db
# Dynamic database credentials (the killer feature):
vault secrets enable database
vault write database/config/mypostgres
plugin_name=postgresql-database-plugin
connection_url="postgresql://{{username}}:{{password}}@db:5432/mydb"
allowed_roles="readonly"
vault write database/roles/readonly
db_name=mypostgres
creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'
VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";"
default_ttl="1h"
max_ttl="24h"
# Application calls:
vault read database/creds/readonly
# Returns: username=v-appname-abc123, password=, lease_duration=1h
# Vault CREATES a real PostgreSQL user for this 1-hour window
# After 1h, Vault revokes it: DROP ROLE "v-appname-abc123"
Dynamic database credentials are more secure than static passwords: each application instance gets its own credential valid for 1 hour. If a credential leaks, it expires automatically. The database has no permanent privileged accounts. All credential creation/revocation is logged in Vault.
AWS Secrets Manager
AWS Secrets Manager is a managed service — no cluster to operate. Stores secrets as JSON blobs, encrypted with KMS. Key features:
- Automatic rotation: built-in rotation for RDS passwords via Lambda function. Configure rotation schedule (e.g., every 30 days). The Lambda rotates the password in RDS and updates the secret — applications using the Secrets Manager SDK transparently get the new password.
- Cross-account access: share secrets across AWS accounts via resource-based policies.
- Native AWS integrations: ECS task definitions, Lambda, and CodeBuild can reference secrets by ARN — the runtime injects them as environment variables without the application making an SDK call.
# ECS task definition: inject secret as environment variable
{
"environment": [],
"secrets": [
{
"name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456:secret:prod/db/password"
}
]
}
# ECS agent fetches the secret from Secrets Manager at task startup
# The application reads DB_PASSWORD as a normal environment variable
PKI Automation: Certificate Lifecycle Management
TLS certificates expire (typically 90 days to 1 year). Manual certificate management at scale (thousands of services) leads to expired certs causing outages. Vault PKI secrets engine acts as an internal CA:
# Vault issues short-lived TLS certs on demand:
vault secrets enable pki
vault write pki/root/generate/internal
common_name="example.com" ttl=87600h # 10-year root CA
vault write pki/roles/server
allowed_domains="example.com"
allow_subdomains=true
max_ttl=72h # certs valid for 72 hours only!
# Service requests a certificate:
vault write pki/issue/server
common_name="api.example.com"
# Returns: certificate, private_key, issuing_ca
# Valid for 72 hours; service must renew before expiry
Short-lived certificates (72 hours) are preferable to long-lived ones (1 year): if a private key is compromised, it expires in days rather than a year. Cert-manager (Kubernetes) integrates with Vault to automatically provision and renew TLS certificates for all services — zero manual certificate management.
Zero-Trust Secret Access
Zero-trust principles applied to secrets: never trust the network, always verify identity. Implementation:
- Every service authenticates to Vault using its workload identity (Kubernetes ServiceAccount, AWS IAM role)
- Vault issues a short-lived token (TTL: 15 minutes to 1 hour) with policies scoped to that service
- The service reads its secrets using the token
- Token expires; the service re-authenticates for the next secret read cycle
- No long-lived static tokens or passwords stored on disk
Secret injection via init containers (Vault Agent): the Vault Agent Injector runs an init container alongside every app pod. It authenticates with Vault, fetches secrets, writes them to a shared volume. The app reads secrets from files, not environment variables (env vars are visible in process listings; files can have restricted permissions). The agent also handles token renewal and secret refresh without the app needing any Vault SDK.
Key Interview Points
- Dynamic credentials > static passwords: Vault creates database users on demand with short TTLs, auto-revoked on expiry
- Kubernetes/IAM auth: workload identity eliminates bootstrapping secrets (no secret to store the secret-getter)
- AWS Secrets Manager: simpler than Vault, native AWS integrations, automatic RDS rotation — best for AWS-native shops
- PKI: short-lived TLS certs (72h) via Vault + cert-manager eliminate manual certificate management
- Vault Agent Injector: secrets as files in init containers — no Vault SDK in application code, no secrets in env vars
- Every secret access logged: who, what, when — enables breach investigation and compliance audits
Frequently Asked Questions
How does HashiCorp Vault manage dynamic database credentials?
Vault's database secrets engine creates real database users on demand with a short TTL, rather than distributing long-lived passwords. Configuration: you register Vault with your database using a privileged account (vault write database/config/mypostgres), then define a role specifying the SQL CREATE ROLE statement and a TTL (e.g., default_ttl="1h"). When an application requests credentials (vault read database/creds/readonly), Vault executes the CREATE ROLE statement against the live database, returning a unique username (e.g., v-appname-abc123) and random password valid for 1 hour. When the TTL expires, Vault executes DROP ROLE automatically — the credential is revoked in the database. The application never holds a long-lived password. If the application is compromised and the credential is stolen, it expires within 1 hour. This is fundamentally more secure than rotating a shared password, because each application instance gets its own unique credential that expires independently. AWS RDS, MySQL, PostgreSQL, MongoDB, and Cassandra are all supported as backends.
How does Vault auto-unseal work and why is it preferred over manual Shamir unsealing?
Vault encrypts all data with a root key. The root key is protected using Shamir's Secret Sharing: it is split into N shares, distributed to key holders, and reconstructing it requires M shares (e.g., 3-of-5). On startup, Vault is "sealed" — it cannot decrypt anything until the root key is reconstructed by M key holders each entering their share. This is secure but operationally difficult: every Vault restart (deployment, crash, maintenance) requires M humans to manually enter their keys, which can take minutes and blocks all secret access. Auto-unseal delegates this to a trusted external service: on startup, Vault calls AWS KMS (or Azure Key Vault, GCP Cloud KMS) to decrypt the root key. No human intervention needed. The security model shifts: instead of requiring M humans, you require IAM access to the KMS key. IAM access is tightly controlled via AWS policies — only the Vault server's IAM role can call kms:Decrypt on that specific key. For production deployments, auto-unseal enables automatic recovery from crashes, rolling restarts, and node replacement without operational toil. Manual unsealing is still available as a fallback if KMS is unavailable.
How does Vault Agent Injector enable zero-trust secret injection in Kubernetes?
Vault Agent Injector is a Kubernetes admission webhook that automatically injects a Vault Agent sidecar container into pods annotated with vault.hashicorp.com/agent-inject: "true". The workflow: (1) Pod is scheduled; the admission webhook intercepts the creation request and adds an init container running Vault Agent. (2) The init container authenticates to Vault using the pod's Kubernetes ServiceAccount JWT — Vault verifies the JWT with the Kubernetes API and issues a short-lived Vault token scoped to the service's policies. (3) Vault Agent reads the configured secrets and writes them to a shared tmpfs volume (in-memory filesystem, not on disk). (4) The main application container reads secrets from /vault/secrets/ as files — no environment variables, no hardcoded values, no secrets in Kubernetes Secrets (which are base64-encoded but not encrypted at rest by default). (5) A sidecar Vault Agent keeps running, renewing the lease and refreshing secrets before they expire. This achieves zero-trust: the application never hardcodes secrets, secrets are never stored in etcd (Kubernetes Secrets), and each pod authenticates independently using its ServiceAccount identity.
{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “How does HashiCorp Vault manage dynamic database credentials?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Vault’s database secrets engine creates real database users on demand with a short TTL, rather than distributing long-lived passwords. Configuration: you register Vault with your database using a privileged account (vault write database/config/mypostgres), then define a role specifying the SQL CREATE ROLE statement and a TTL (e.g., default_ttl=”1h”). When an application requests credentials (vault read database/creds/readonly), Vault executes the CREATE ROLE statement against the live database, returning a unique username (e.g., v-appname-abc123) and random password valid for 1 hour. When the TTL expires, Vault executes DROP ROLE automatically — the credential is revoked in the database. The application never holds a long-lived password. If the application is compromised and the credential is stolen, it expires within 1 hour. This is fundamentally more secure than rotating a shared password, because each application instance gets its own unique credential that expires independently. AWS RDS, MySQL, PostgreSQL, MongoDB, and Cassandra are all supported as backends.”
}
},
{
“@type”: “Question”,
“name”: “How does Vault auto-unseal work and why is it preferred over manual Shamir unsealing?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Vault encrypts all data with a root key. The root key is protected using Shamir’s Secret Sharing: it is split into N shares, distributed to key holders, and reconstructing it requires M shares (e.g., 3-of-5). On startup, Vault is “sealed” — it cannot decrypt anything until the root key is reconstructed by M key holders each entering their share. This is secure but operationally difficult: every Vault restart (deployment, crash, maintenance) requires M humans to manually enter their keys, which can take minutes and blocks all secret access. Auto-unseal delegates this to a trusted external service: on startup, Vault calls AWS KMS (or Azure Key Vault, GCP Cloud KMS) to decrypt the root key. No human intervention needed. The security model shifts: instead of requiring M humans, you require IAM access to the KMS key. IAM access is tightly controlled via AWS policies — only the Vault server’s IAM role can call kms:Decrypt on that specific key. For production deployments, auto-unseal enables automatic recovery from crashes, rolling restarts, and node replacement without operational toil. Manual unsealing is still available as a fallback if KMS is unavailable.”
}
},
{
“@type”: “Question”,
“name”: “How does Vault Agent Injector enable zero-trust secret injection in Kubernetes?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Vault Agent Injector is a Kubernetes admission webhook that automatically injects a Vault Agent sidecar container into pods annotated with vault.hashicorp.com/agent-inject: “true”. The workflow: (1) Pod is scheduled; the admission webhook intercepts the creation request and adds an init container running Vault Agent. (2) The init container authenticates to Vault using the pod’s Kubernetes ServiceAccount JWT — Vault verifies the JWT with the Kubernetes API and issues a short-lived Vault token scoped to the service’s policies. (3) Vault Agent reads the configured secrets and writes them to a shared tmpfs volume (in-memory filesystem, not on disk). (4) The main application container reads secrets from /vault/secrets/ as files — no environment variables, no hardcoded values, no secrets in Kubernetes Secrets (which are base64-encoded but not encrypted at rest by default). (5) A sidecar Vault Agent keeps running, renewing the lease and refreshing secrets before they expire. This achieves zero-trust: the application never hardcodes secrets, secrets are never stored in etcd (Kubernetes Secrets), and each pod authenticates independently using its ServiceAccount identity.”
}
}
]
}