Handling Thread Exhaustion & Backpressure

In high-throughput microservices, threads are a limited resource. Improper management can lead to thread exhaustion, increased latency, or system crashes. Backpressure prevents the system from being overwhelmed, ensuring stability and predictable performance.

What Is Thread Exhaustion?

Thread exhaustion happens when all threads in a pool are busy, and new tasks cannot be processed.

Common in I/O-heavy microservices, especially with synchronous calls.

Symptoms:

• Slow response times
• Growing request queues
• Potential OutOfMemoryError if tasks pile up

Causes:

• Blocking I/O operations (DB, REST, file systems)
• Small or fixed-size thread pools
• Traffic spikes
• Unbounded task queues

Traditional Threads: Each consumes ~1MB of memory, limiting scalability.

Virtual Threads (Java 21+): Extremely lightweight (~1KB per thread), perfect for high-concurrency, I/O-bound workloads. However, always monitor external resources like database or API connections to avoid bottlenecks.

Backpressure

Backpressure helps prevent thread exhaustion by slowing down or rejecting requests when a service is overloaded. This ensures system stability and predictable performance.

Common Strategies:

• Return HTTP 429 (Too Many Requests) for excess REST calls
• Queue requests up to a defined limit
• Apply rate limiting at clients or API gateways (e.g., Nginx)
• Shed low-priority tasks during traffic spikes

Thread Pool Management

Proper thread pool configuration is essential for managing asynchronous and concurrent tasks in microservices.

a) Bounded Thread Pools

• Prevents unlimited task accumulation
• Provides natural backpressure when the pool reaches its limit

ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(20);
executor.setMaxPoolSize(50);
executor.setQueueCapacity(100); // Prevent unbounded queue growth
executor.setThreadNamePrefix("AppThread-");
executor.initialize();

b) RejectedExecutionHandler

Handles tasks that cannot be executed because the pool and queue are full:

executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());

Common Policies:

• AbortPolicy: Throws exception for rejected tasks
• CallerRunsPolicy: Runs task in the calling thread
• DiscardPolicy: Silently discards the task
• DiscardOldestPolicy: Discards the oldest queued task to make room

Prevent Blocking in Thread Pools

Blocking operations can exhaust threads and increase latency. To prevent this:

• Avoid blocking I/O in fixed thread pools
• Use non-blocking clients:
  • WebClient (Spring WebFlux) instead of RestTemplate
  • R2DBC for reactive database access instead of JDBC
• Virtual Threads (Java 21+) allow blocking operations without exhausting OS threads:

ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();
CompletableFuture.supplyAsync(() -> serviceCall(), executor);

Queue Management

Proper queue handling prevents memory pressure and ensures predictable latency:

• Avoid unbounded queues
• Use bounded queues with defined overflow strategies:
  • Reject new tasks
  • Drop oldest tasks
  • Rate-limit incoming requests

This approach maintains stable throughput and prevents thread exhaustion under high load.

Load Shedding & Rate Limiting

To protect microservices during traffic spikes, control incoming requests using:

• Token Bucket / Leaky Bucket algorithms
• API Gateway rate limiting (e.g., Nginx, Spring Cloud Gateway)
• Circuit breakers with libraries like Resilience4j or Hystrix

Monitoring & Metrics

Keep track of system health and thread usage to avoid thread exhaustion:

Key Metrics:

• Active threads
• Queue size
• Rejected tasks
• CPU and memory usage

Monitoring Tools:

• Micrometer + Prometheus + Grafana for dashboards
• JVM ThreadMXBean for thread insights
• Spring Boot Actuator metrics for live monitoring

Best Practices

• Always configure bounded thread pools
• Avoid blocking I/O in fixed pools
• Use Virtual Threads for scalable I/O-bound tasks
• Apply backpressure at gateway or service layer
• Monitor metrics and set alerts
• Use circuit breakers for downstream failures
• Combine async programming with thread pool sizing for predictable throughput

Practical Example: Payment Service with Thread Pool & Backpressure

// 1. Configure a bounded thread pool for async tasks
@Configuration
@EnableAsync
public class AsyncConfig {

    @Bean(name = "paymentExecutor")
    public Executor paymentExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(20);
        executor.setMaxPoolSize(50);
        executor.setQueueCapacity(50); // Bounded queue
        executor.setThreadNamePrefix("Payment-");
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        executor.initialize();
        return executor;
    }
}

// 2. PaymentService with async processing
@Service
public class PaymentService {

    @Async("paymentExecutor")
    public CompletableFuture<String> processPayment(String orderId) {
        try {
            // Simulate I/O-bound payment processing
            Thread.sleep(2000); // e.g., call to external payment API
            return CompletableFuture.completedFuture("Payment processed for order " + orderId);
        } catch (InterruptedException e) {
            return CompletableFuture.completedFuture("Payment failed for order " + orderId);
        }
    }
}

// 3. Controller with backpressure handling
@RestController
@RequestMapping("/payments")
public class PaymentController {

    private final PaymentService paymentService;
    private final Semaphore semaphore = new Semaphore(50); // Backpressure control

    public PaymentController(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @PostMapping("/{orderId}")
    public ResponseEntity<String> pay(@PathVariable String orderId) {

        // Apply backpressure: reject requests if max concurrent tasks reached
        if (!semaphore.tryAcquire()) {
            return ResponseEntity.status(429).body("Too many requests, try later");
        }

        CompletableFuture<String> result = paymentService.processPayment(orderId)
            .whenComplete((res, ex) -> semaphore.release());

        return ResponseEntity.ok("Payment request accepted for order " + orderId);
    }
}

Real-World Example

Scenario: High-concurrency payment service under heavy load

Implementation:

• API Gateway rate limiting – e.g., 100 requests/sec per client
• Thread pool executor – ThreadPoolTaskExecutor with bounded queue (size 50)
• Rejected tasks – Extra requests return HTTP 429 (Too Many Requests)
• Non-blocking I/O – Downstream calls use WebClient instead of blocking RestTemplate
• Virtual threads – Handle massive concurrent I/O efficiently
• Monitoring dashboards – Track queue length, active threads, and rejected requests

Benefits:

• Prevents thread exhaustion
• Maintains controlled, predictable latency
• Ensures stable resource usage under spikes
• Supports high-concurrency safely and efficiently

Conclusion

Thread exhaustion can severely impact microservice performance. Proper thread pool sizing and backpressure mechanisms prevent overload. Using non-blocking I/O or Virtual Threads allows handling thousands of concurrent requests efficiently. Monitoring and alerting ensures visibility into queues, active threads, and resource usage. Combining these strategies makes microservices resilient, scalable, and responsive, even under high traffic.