Bulkhead Pattern: Isolating Failures in Distributed Systems

The Titanic Analogy

The RMS Titanic sank in part because its watertight compartments were not truly watertight — water could flow over the top of the bulkheads between compartments. The bulkhead pattern in software takes the lesson seriously: if one compartment (dependency, service call, resource pool) floods, the failure must not propagate to adjacent compartments.

In distributed systems, the cascade failure scenario plays out like this:

API Gateway → Service A → Database (slow) → thread pool exhausted
                       → Service B (blocked waiting for threads)
                       → Service C (blocked waiting for threads)
                       → API Gateway (all threads blocked)
                       → Total outage

A single slow database causes a total outage because all services share the same thread pool. Bulkheads partition the pool so that database slowness can only exhaust the threads allocated for database calls — not threads serving Service B or Service C.

Types of Bulkheads

Thread Pool Isolation

The most common form: each dependency gets its own fixed-size thread pool. When the pool is exhausted, calls fail immediately with a rejection rather than queuing and blocking other work:

Main thread pool (200 threads)
├── payment-service pool  (20 threads) → payment calls
├── inventory-service pool (15 threads) → inventory calls
├── email-service pool   (5 threads)  → email calls
└── database pool        (30 threads) → DB queries

If the payment service hangs, only its 20 threads block. The other 195 threads continue serving inventory, email, and database work normally.

Semaphore-Based Isolation

Semaphores control the maximum number of concurrent calls without dedicating threads. This is lighter-weight than thread pool isolation and works naturally with async frameworks where threads are cheap:

import asyncio
from functools import wraps
from typing import Any, Callable, TypeVar

T = TypeVar('T')

def bulkhead(max_concurrent: int) -> Callable:
    """Decorator that limits concurrent calls to a function."""
    semaphore = asyncio.Semaphore(max_concurrent)

    def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
        @wraps(func)
        async def wrapper(*args: Any, **kwargs: Any) -> Any:
            if semaphore.locked() and semaphore._value == 0:
                raise RuntimeError(
                    f'Bulkhead full: {func.__name__} at capacity ({max_concurrent})'
                )
            async with semaphore:
                return await func(*args, **kwargs)
        return wrapper
    return decorator

@bulkhead(max_concurrent=10)
async def call_payment_service(order_id: str) -> dict:
    async with httpx.AsyncClient() as client:
        resp = await client.post('/payments', json={'order_id': order_id}, timeout=5.0)
        return resp.json()

Connection Pool Partitioning

Database connection pools are a natural bulkhead boundary. Instead of one shared pool for all operations, maintain separate pools for different workloads:

# Django multi-database routing with separate pools
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'myapp',
        'CONN_MAX_AGE': 60,
        'OPTIONS': {'pool': {'min_size': 5, 'max_size': 20}},  # OLTP writes
    },
    'analytics': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'myapp_replica',
        'CONN_MAX_AGE': 60,
        'OPTIONS': {'pool': {'min_size': 2, 'max_size': 5}},   # Reporting reads
    },
}

Slow analytics queries can exhaust the 5-connection analytics pool without touching the 20-connection OLTP pool serving user-facing requests.

Implementation: Resilience4j

Resilience4j is the go-to bulkhead library for JVM services:

// Thread pool bulkhead (async, dedicated executor)
BulkheadConfig threadPoolConfig = BulkheadConfig.custom()
    .maxThreadPoolSize(10)        // Max threads
    .coreThreadPoolSize(5)        // Core threads
    .queueCapacity(0)             // No queue — fail fast
    .keepAliveDuration(Duration.ofMillis(100))
    .build();

// Semaphore bulkhead (same thread, non-blocking check)
BulkheadConfig semaphoreConfig = BulkheadConfig.custom()
    .maxConcurrentCalls(10)       // Max concurrent calls
    .maxWaitDuration(Duration.ZERO)  // Fail immediately if full
    .build();

Bulkhead paymentBulkhead = BulkheadRegistry.of(semaphoreConfig)
    .bulkhead('payment-service');

// Decorate the remote call
Supplier<PaymentResult> decorated = Bulkhead
    .decorateSupplier(paymentBulkhead, () -> paymentClient.charge(orderId));

Try.ofSupplier(decorated)
    .recover(BulkheadFullException.class, ex -> PaymentResult.rejected('CAPACITY'))
    .get();

Combining Bulkhead with Circuit Breaker

Bulkhead and circuit breaker solve different problems and work best together:

Incoming request
       ↓
[Bulkhead] — Is there capacity? If not → immediate rejection (503)
       ↓
[Circuit Breaker] — Is the service healthy? If open → immediate rejection (503)
       ↓
[Retry with backoff] — Transient failure? Retry 2–3 times
       ↓
[Timeout] — Cancel if no response within N seconds
       ↓
Remote service call

The bulkhead acts first, protecting local resources. The circuit breaker acts second, protecting the downstream service from overload during recovery. Together they prevent both local resource exhaustion and thundering herd:

import asyncio
from contextlib import asynccontextmanager

class ResilienceLayer:
    def __init__(self, max_concurrent: int, failure_threshold: float):
        self._semaphore = asyncio.Semaphore(max_concurrent)
        self._failure_count = 0
        self._circuit_open = False
        self._failure_threshold = failure_threshold

    @asynccontextmanager
    async def protect(self):
        if self._circuit_open:
            raise RuntimeError('Circuit open — fast fail')
        if not self._semaphore._value:
            raise RuntimeError('Bulkhead full — fast fail')
        async with self._semaphore:
            try:
                yield
                self._failure_count = 0
            except Exception:
                self._failure_count += 1
                if self._failure_count >= self._failure_threshold:
                    self._circuit_open = True
                raise

Monitoring and Tuning

Bulkheads that are never triggered provide no value. Bulkheads that trigger constantly are sized too small. Key metrics to track:

Start with pool sizes 2–3× your expected peak concurrency for that dependency. If p99 latency for the dependency is 200ms and you receive 50 requests/second to that path, you need 50 × 0.2 = 10 concurrent calls at steady state — so a pool of 20–30 provides comfortable headroom.

Reduce the pool size if you see the downstream service overwhelmed during bursts. Increase it if you see legitimate requests being rejected during normal operation (starvation). The goal is to shed load gracefully under pressure, not to routinely reject requests.