# Bulkhead Pattern Low-Level Design: Thread Pool Isolation, Semaphore Limits, and Shed Load

Source: https://www.techinterview.org/post/3233470065/lld-bulkhead-service/
Updated: 2026-04-17 · techinterview.org

The bulkhead pattern partitions resources so that a slow or failing dependency cannot consume all threads and starve the rest of the application. This design covers thread pool isolation, semaphore-based concurrency limits, and load shedding when queues overflow.

## Requirements

### Functional

- Assign each downstream dependency its own resource pool (threads or semaphore permits).

- Reject calls that cannot acquire a resource within a timeout rather than queuing indefinitely.

- Shed excess load with a configurable rejection policy (fail-fast or fallback).

- Support dynamic pool resizing without restart.

- Expose per-bulkhead utilization metrics.

### Non-Functional

- Acquisition overhead under 2 ms.

- Pool exhaustion must not propagate latency to other bulkheads.

- Configuration hot-reload via a control plane API.

## Two Isolation Strategies

### Thread Pool Isolation

Each dependency gets a dedicated `ThreadPoolExecutor` with a fixed core size, bounded queue, and a caller-runs or abort rejection handler. The calling thread submits work and blocks on `Future.get(timeout)`. This provides strong isolation: a slow dependency cannot borrow threads from another pool. The cost is context-switch overhead and stack memory per thread.

### Semaphore Isolation

A `Semaphore(permits, fair=false)` guards each dependency. The calling thread acquires a permit before executing the call inline and releases it in a finally block. No thread switching occurs, so latency is lower, but the calling thread is still tied up during a slow call. Semaphore isolation is preferred for lightweight, CPU-bound calls or when you already have a separate async executor.

## Data Model

- **BulkheadConfig** — `dependencyKey`, `isolationType` (THREAD_POOL or SEMAPHORE), `maxConcurrent`, `maxQueueSize`, `acquireTimeoutMs`, `fallbackFn`.

- **BulkheadState** — `activeCalls` (AtomicInteger), `queuedCalls` (AtomicInteger), `rejectedTotal` (LongAdder), `successTotal` (LongAdder), `failureTotal` (LongAdder).

- **BulkheadRegistry** — concurrent map from `dependencyKey` to `Bulkhead` instance. Supports `update(config)` to hot-swap pool parameters.

## Core Algorithms

### Concurrency Limit Enforcement

For semaphore mode: call `semaphore.tryAcquire(acquireTimeoutMs, MILLISECONDS)`. If it returns false, increment `rejectedTotal` and invoke `fallbackFn`. On success, run the dependency call, then release in a finally block. Track `activeCalls` with an AtomicInteger for metrics without touching the semaphore internals.

For thread pool mode: submit the task to the executor. If the bounded queue is full, the `RejectedExecutionHandler` fires. Use a custom handler that increments `rejectedTotal` and returns a completed future containing the fallback value rather than throwing, so callers get a clean result type.

### Dynamic Resizing

To resize a semaphore bulkhead, compare old and new `maxConcurrent`. If increasing, call `semaphore.release(delta)`. If decreasing, drain excess permits by calling `semaphore.tryAcquire(delta)` in a background thread — this blocks until in-flight calls complete naturally. For thread pool bulkheads, call `executor.setCorePoolSize` and `setMaximumPoolSize` atomically with a size-change lock.

### Load Shedding Priority

Assign each request a priority tier (CRITICAL, NORMAL, BEST_EFFORT). When the bulkhead is saturated, reject BEST_EFFORT requests first. Implement this by maintaining three sub-queues inside the thread pool and having a priority-aware dispatcher feed the executor. Critical requests bypass the bulkhead queue up to a reserved headroom of `criticalReservePercent * maxConcurrent` permits.

## API Design

- `Bulkhead.execute(Supplier<T> call, Supplier<T> fallback, Priority priority): T` — main entry point.

- `BulkheadRegistry.register(BulkheadConfig config): Bulkhead` — register or update a bulkhead.

- `Bulkhead.getSnapshot(): BulkheadSnapshot` — returns active, queued, rejected, success, failure counts for health endpoints.

- `BulkheadRegistry.resize(String key, int newMax): void` — control-plane hot-resize call.

## Scalability and Observability

Expose a Prometheus gauge `bulkhead_active_calls{dependency}` and counter `bulkhead_rejected_total{dependency,reason}`. A rejection rate above 1% on a critical bulkhead should page. Plot utilization as a heatmap per dependency to identify which services need pool expansion versus which upstream callers need rate limiting.

- Combine bulkheads with circuit breakers: the circuit breaker trips first on error rate; the bulkhead prevents thread exhaustion during the Open-to-Half-Open transition.

- Keep bulkhead timeouts shorter than the upstream caller timeout to give the fallback time to respond.

- Test rejection behavior with chaos tooling: inject artificial latency into a dependency and verify that sibling dependencies remain unaffected.

- For multi-tenant systems, namespace bulkheads by tenant ID to prevent noisy-neighbor resource contention.

See also: [Netflix Interview Guide 2026: Streaming Architecture, Recommendation Systems, and Engineering Excellence](https://www.techinterview.org/post/3233460274/netflix-interview-guide-2026-streaming-architecture-recommendation-systems-and-engineering-excellence/)

See also: [Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering](https://www.techinterview.org/post/3233460275/scale-ai-interview-guide-2026-data-infrastructure-rlhf-pipelines-and-ml-engineering/)

See also: [Anthropic Interview Guide 2026: Process, Questions, and AI Safety](https://www.techinterview.org/?p=3233460282)