# Secrets Management System Low-Level Design: Vault Storage, Dynamic Secrets, Rotation, and Audit

Source: https://www.techinterview.org/post/3233468937/lld-secrets-management/
Updated: 2026-06-30 · techinterview.org

A secrets management system stores, distributes, and rotates sensitive credentials — database passwords, API keys, TLS certificates, and cloud credentials — so that no application ever hard-codes a secret. This guide covers the low-level design: encrypted storage, dynamic secret generation, lease management, automatic rotation, access control, and tamper-evident audit logging.

## 1. Secret Storage and Envelope Encryption

Every secret is encrypted at rest using **AES-256-GCM**. To avoid re-encrypting all secrets when a master key rotates, the system uses **envelope encryption**:

  - A **Data Encryption Key (DEK)** is generated per secret (or per secret version).

  - The DEK is encrypted by a **Key Encryption Key (KEK)** stored in an HSM or cloud KMS (AWS KMS, Google Cloud KMS).

  - The database stores only the encrypted DEK and the ciphertext of the secret value.

  - On read: fetch encrypted DEK, call KMS to decrypt it, use DEK to decrypt secret, wipe DEK from memory immediately after use.

KEK rotation requires only re-wrapping the DEK for each secret — a cheap operation — not re-encrypting secret values.


```
-- Secret storage schema
CREATE TABLE Secret (
    id             BIGSERIAL PRIMARY KEY,
    path           TEXT NOT NULL UNIQUE,          -- e.g. "prod/db/postgres"
    encrypted_value BYTEA NOT NULL,               -- AES-256-GCM ciphertext
    encrypted_dek  BYTEA NOT NULL,                -- DEK wrapped by KEK
    kek_version    TEXT NOT NULL,                 -- KMS key alias/version
    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at     TIMESTAMPTZ,
    version        INT NOT NULL DEFAULT 1
);

CREATE TABLE SecretLease (
    id           BIGSERIAL PRIMARY KEY,
    secret_id    BIGINT NOT NULL REFERENCES Secret(id),
    client_id    TEXT NOT NULL,
    lease_token  TEXT NOT NULL UNIQUE,
    expires_at   TIMESTAMPTZ NOT NULL,
    renewed_at   TIMESTAMPTZ,
    revoked      BOOLEAN NOT NULL DEFAULT FALSE
);

CREATE TABLE SecretRotationSchedule (
    secret_id                BIGINT PRIMARY KEY REFERENCES Secret(id),
    rotation_interval_seconds INT NOT NULL,
    last_rotated_at          TIMESTAMPTZ,
    next_rotation_at         TIMESTAMPTZ NOT NULL
);

CREATE TABLE SecretAudit (
    id         BIGSERIAL PRIMARY KEY,
    secret_id  BIGINT NOT NULL,
    action     TEXT NOT NULL,   -- READ, WRITE, ROTATE, REVOKE
    client_id  TEXT NOT NULL,
    timestamp  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```


## 2. Dynamic Secrets

Static secrets are long-lived and shared; **dynamic secrets** are generated on demand and scoped to a single caller for a limited TTL. Two canonical examples:

  - **Database credentials:** on request, the secrets engine creates a new Postgres user with limited GRANT and returns the username/password. On lease expiry, the user is dropped.

  - **AWS temporary credentials:** the engine calls `sts:AssumeRole` and returns an access key, secret key, and session token. TTL matches the STS session duration.

Dynamic secrets eliminate credential sharing and reduce blast radius: a leaked credential is short-lived and maps to exactly one client.

## 3. Lease Management

Every secret access creates a **lease** with a TTL. The client receives a `lease_token` and must renew it before expiry to continue using the secret. On expiry:

  - Static secrets: the lease is revoked; the client must re-authenticate and re-fetch.

  - Dynamic secrets: the associated credential is deleted (DB user dropped, STS session expires).

Renewal is a lightweight heartbeat — no new credential is issued; the expiry timestamp is extended up to a configurable max TTL. If a process crashes, the lease simply expires and credentials are cleaned up automatically.

## 4. Automatic Rotation

The rotation scheduler runs a background job that queries `SecretRotationSchedule WHERE next_rotation_at <= NOW()`. For each due secret:

  - Generate or retrieve a new credential value from the upstream system.

  - Write the new value as a new version of the secret (version N+1).

  - Keep version N alive for a **grace period** (e.g., 60 seconds) so in-flight requests using the old credential can complete.

  - After the grace period, mark version N as expired; revoke all leases pointing to it.

  - Update `last_rotated_at` and compute `next_rotation_at = NOW() + rotation_interval_seconds`.

This rolling window ensures zero-downtime rotation for well-behaved clients that re-fetch on lease renewal.

## 5. Access Control

Policies bind **service identities** (Kubernetes service account, AWS IAM role, mTLS cert CN) to allowed operations on secret path prefixes. Example policy (HCL-like):


```
path "prod/db/*" {
  capabilities = ["read", "renew"]
}
path "prod/db/admin" {
  capabilities = []   -- deny even if wildcard matches
}
```

Policy evaluation is **deny-by-default**: a request is allowed only if an explicit allow policy matches and no deny policy overrides it. Service identities are authenticated via a platform-provided token (Kubernetes projected service account token, AWS IAM, etc.) before any policy is checked.

## 6. Audit Logging

Every READ, WRITE, ROTATE, and REVOKE is appended to `SecretAudit`. The table is insert-only: a DB trigger raises an exception on any UPDATE or DELETE attempt. Audit records feed a SIEM for anomaly detection (e.g., a single client reading 1000 secrets in one minute).

## 7. Python Reference Implementation


```
import hashlib, os, time
from dataclasses import dataclass
from typing import Optional

@dataclass
class Lease:
    lease_token: str
    secret_value: str
    expires_at: float

def get_secret(path: str, client_id: str) -> Lease:
    """Fetch a secret, create a lease, append audit record."""
    secret = db.query_one(
        "SELECT * FROM Secret WHERE path = %s AND (expires_at IS NULL OR expires_at > NOW())",
        [path]
    )
    if not secret:
        raise KeyError(f"Secret not found: {path}")

    policy_check(client_id, path, "read")          # raises if denied

    dek = kms_decrypt(secret.encrypted_dek, secret.kek_version)
    plaintext = aes_gcm_decrypt(secret.encrypted_value, dek)
    dek = None                                     # wipe immediately

    token = os.urandom(32).hex()
    expires_at = time.time() + DEFAULT_LEASE_TTL
    db.execute(
        "INSERT INTO SecretLease (secret_id, client_id, lease_token, expires_at) VALUES (%s,%s,%s,%s)",
        [secret.id, client_id, token, expires_at]
    )
    db.execute(
        "INSERT INTO SecretAudit (secret_id, action, client_id) VALUES (%s,%s,%s)",
        [secret.id, "READ", client_id]
    )
    return Lease(lease_token=token, secret_value=plaintext, expires_at=expires_at)

def create_dynamic_secret(type: str, params: dict) -> Lease:
    """Generate a short-lived credential on demand."""
    if type == "postgres":
        username = f"dyn_{os.urandom(6).hex()}"
        password = os.urandom(24).hex()
        db.execute(f"CREATE USER {username} WITH PASSWORD %s", [password])
        db.execute(f"GRANT SELECT ON ALL TABLES IN SCHEMA public TO {username}")
        value = f"{username}:{password}"
    elif type == "aws_sts":
        creds = boto3.client("sts").assume_role(
            RoleArn=params["role_arn"],
            RoleSessionName=f"dyn-{os.urandom(4).hex()}",
            DurationSeconds=params.get("ttl", 3600)
        )["Credentials"]
        value = json.dumps({
            "AccessKeyId": creds["AccessKeyId"],
            "SecretAccessKey": creds["SecretAccessKey"],
            "SessionToken": creds["SessionToken"]
        })
    else:
        raise ValueError(f"Unknown dynamic secret type: {type}")

    token = os.urandom(32).hex()
    return Lease(lease_token=token, secret_value=value,
                 expires_at=time.time() + params.get("ttl", 3600))

def rotate_secret(secret_id: int) -> None:
    """Rotate a secret in place with a grace period for old version."""
    secret = db.query_one("SELECT * FROM Secret WHERE id = %s", [secret_id])
    new_value = fetch_new_credential(secret.path)
    dek = os.urandom(32)
    ciphertext = aes_gcm_encrypt(new_value, dek)
    encrypted_dek = kms_encrypt(dek, current_kek_version())
    dek = None

    db.execute(
        "UPDATE Secret SET encrypted_value=%s, encrypted_dek=%s, version=version+1 WHERE id=%s",
        [ciphertext, encrypted_dek, secret_id]
    )
    db.execute(
        "UPDATE SecretRotationSchedule SET last_rotated_at=NOW(), "
        "next_rotation_at=NOW() + (rotation_interval_seconds || ' seconds')::INTERVAL WHERE secret_id=%s",
        [secret_id]
    )
    db.execute(
        "INSERT INTO SecretAudit (secret_id, action, client_id) VALUES (%s,'ROTATE','scheduler')",
        [secret_id]
    )

def revoke_lease(lease_token: str) -> None:
    """Immediately revoke a lease; dynamic secrets are cleaned up."""
    lease = db.query_one("SELECT * FROM SecretLease WHERE lease_token=%s", [lease_token])
    if not lease:
        return
    db.execute(
        "UPDATE SecretLease SET revoked=TRUE WHERE lease_token=%s", [lease_token]
    )
    db.execute(
        "INSERT INTO SecretAudit (secret_id, action, client_id) VALUES (%s,'REVOKE',%s)",
        [lease.secret_id, lease.client_id]
    )
```


## 8. Scalability and Reliability

  - **HA storage backend:** secrets stored in Raft-replicated storage (Consul, etcd, or Vault integrated storage) with quorum writes.

  - **Caching:** in-process cache with TTL < lease TTL to reduce KMS decrypt calls; cache is encrypted in memory.

  - **KMS latency:** batch DEK decryptions; use a DEK cache keyed by (kek_version, encrypted_dek) with a short TTL.

  - **Rotation worker:** distributed lock (Redis SETNX or Postgres advisory lock) prevents double-rotation when multiple scheduler instances run.

See also: [Netflix Interview Guide 2026: Streaming Architecture, Recommendation Systems, and Engineering Excellence](https://www.techinterview.org/post/3233460274/netflix-interview-guide-2026-streaming-architecture-recommendation-systems-and-engineering-excellence/)

See also: [Meta Interview Guide 2026: Facebook, Instagram, WhatsApp Engineering](https://www.techinterview.org/post/3233460272/meta-interview-guide-2026-facebook-instagram-whatsapp-engineering/)

See also: [Stripe Interview Guide 2026: Process, Bug Bash Round, and Payment Systems](https://www.techinterview.org/post/3233460268/stripe-interview-guide-2026-process-bug-bash-round-and-payment-systems/)