Question 1

What is envelope encryption and why does a secret manager use it?

Accepted Answer

Envelope encryption protects secrets at rest without exposing the master key. Process: (1) Generate a unique Data Encryption Key (DEK) for each secret. (2) Encrypt the secret value with the DEK using AES-256-GCM. (3) Encrypt the DEK with a Key Encryption Key (KEK) stored in a Hardware Security Module (HSM) or KMS like AWS KMS. (4) Store only the encrypted DEK (wrapped DEK) and the ciphertext in the database. To decrypt: call KMS to unwrap the DEK (KMS decrypts using the KEK, which never leaves the HSM), then use the DEK to decrypt the secret locally. Benefits: (1) Even if the database is compromised, secrets are unreadable without the KEK. (2) DEKs are small — cheap to re-encrypt with a new KEK (key rotation). (3) The KEK is centrally managed and audited in KMS. The secret manager never sees the KEK in plaintext.

Question 2

How does zero-downtime secret rotation work?

Accepted Answer

The challenge: rotating a DB password requires updating both the DB and every service that uses it, without causing authentication failures. Zero-downtime protocol: (1) Generate a new password. (2) Update the DB to accept BOTH the old and new password (create new DB user, or use DB-specific dual password support). (3) Create a new SecretVersion with status=CURRENT; set old version to status=PREVIOUS. (4) Services polling with TTL=60s will fetch the new version within 1 minute. (5) After grace period (e.g., 2x the TTL = 2 minutes), revoke the old password from the DB and mark old SecretVersion as DEPRECATED. Services that refresh on authentication failure will also handle the transition: if auth fails with cached secret, re-fetch and retry once. This handles the edge case where a service just fetched the old version before the rotation completed.

Question 3

How does a service authenticate to a secret manager without a bootstrap credential?

Accepted Answer

The bootstrap credential problem: to get secrets from a secret manager, a service needs credentials to authenticate — but those credentials are themselves secrets that need to be stored securely. Solutions: (1) Cloud workload identity (preferred): AWS IAM role attached to an EC2 instance or ECS task; GCP Service Account attached to a GKE pod. The cloud provider validates the identity using the instance metadata service (IMDS). No credentials to store — the VM/container proves its identity through a signed token from the cloud provider. (2) Kubernetes service accounts with IRSA (IAM Roles for Service Accounts): pods have a JWT automatically injected and exchanged for cloud credentials. (3) Vault AppRole: a role_id (not secret, can be in config) + a short-lived secret_id injected by CI/CD at deploy time. Cloud workload identity is cleanest: zero additional secrets, enforced by the cloud provider.

Question 4

What should be in the audit log for a secret manager?

Accepted Answer

Every secret access must be logged for compliance (SOC2, HIPAA, PCI-DSS require access audit trails). Each audit record should include: secret_path (what was accessed), version_id (which version), principal_id (who accessed it — service account, user), action (READ, WRITE, ROTATE, DELETE), timestamp, ip_address, success (true/false), and reason_denied (if failed — authorization failure, secret not found, version deprecated). Store audit logs immutably: write to append-only storage (Kafka → S3 with object lock, or write to a separate tamper-evident DB). Retention: SOC2 requires 1 year, HIPAA requires 6 years, PCI-DSS requires 1 year. Alerts: alert on unusual patterns — high read volume from a single IP, reads of secrets outside the principal's normal set, failed auth attempts from a service.

Question 5

How does secret caching in the client SDK work?

Accepted Answer

Client SDK caching prevents every secret access from hitting the secret manager (which adds latency and creates a hard dependency). Cache behavior: on first access, fetch from secret manager, store in memory with TTL=60s. On subsequent accesses within TTL: serve from cache — O(1), no network call. On TTL expiry: background refresh (fetch new value, update cache) or lazy refresh (fetch on next access). On secret manager unavailability: serve stale cached value — never return an error. On authentication failure with cached credentials: refresh immediately (don't wait for TTL). For rotation: services with a 60s cache TTL will see the new secret within 60 seconds — acceptable for most use cases. For zero-downtime rotation, the grace period must be at least 2x the TTL. Monitor staleness: if cache age > 10 minutes, alert — the secret manager may be unreachable.

Secret Manager System Low-Level Design

What is a Secret Manager?

Requirements

Data Model

Encryption Architecture (Envelope Encryption)

Secret Rotation

Service Authentication (Workload Identity)

Local Caching and High Availability

Key Design Decisions