Resilience4j: Circuit Breaker, Retry, Rate Limiter and Bulkhead

In a distributed system, one slow service can bring down all others because threads pile up waiting for it. Resilience4j gives you small, composable patterns to stop this cascade: circuit breaker, retry, time limiter, bulkhead and rate limiter.

The cascade failure problem

Service A ─► Service B ─► Service C  (C is slow: 30 s per call)
A's 200 threads wait for B, B's threads wait for C ... after a minute ALL threads are blocked.
A, B and even unrelated endpoints are now unavailable. One slow dependency killed the system.

With a circuit breaker:  C fails often ─► breaker OPENS ─► calls fail FAST (fallback, 1 ms) ─► threads stay free
                         later the breaker tests C again (HALF-OPEN) and closes when C recovers.

Circuit breaker states

            failure rate >= threshold (e.g. 50% of the last 10 calls)
   CLOSED ─────────────────────────────────────────────► OPEN
   (all calls pass,                                       (calls are rejected immediately
    results are counted)                                   CallNotPermittedException for waitDuration, e.g. 10 s)
      ▲                                                       │ after waitDuration
      │  test calls succeed                                   ▼
      └───────────────────────────── HALF_OPEN ◄──────────────┘
                                     (allows a few trial calls;
                                      if they fail ─► back to OPEN)

Setup

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<!-- Boot 4: use the Resilience4j version that supports Spring Boot 4 (check its release notes). Spring Framework 7 also adds core @Retryable and @ConcurrencyLimit support. -->

resilience4j:
  circuitbreaker:
    instances:
      paymentApi:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 10
        minimum-number-of-calls: 5
        failure-rate-threshold: 50
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 80
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 3
        automatic-transition-from-open-to-half-open-enabled: true
        register-health-indicator: true
  retry:
    instances:
      paymentApi:
        max-attempts: 3
        wait-duration: 500ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.ResourceAccessException
        ignore-exceptions:
          - com.example.BusinessException
  bulkhead:
    instances:
      paymentApi:
        max-concurrent-calls: 20
        max-wait-duration: 0
  ratelimiter:
    instances:
      paymentApi:
        limit-for-period: 50
        limit-refresh-period: 1s
        timeout-duration: 0

Use it with annotations

@Service
public class PaymentClient {

    private static final Logger log = LoggerFactory.getLogger(PaymentClient.class);
    private final RestClient client;

    public PaymentClient(RestClient.Builder builder) {
        this.client = builder.baseUrl("http://payment-service").build();
    }

    @CircuitBreaker(name = "paymentApi", fallbackMethod = "chargeFallback")
    @Retry(name = "paymentApi")
    @Bulkhead(name = "paymentApi")
    public PaymentResult charge(String orderId, BigDecimal amount) {
        return client.post().uri("/payments")
                .body(Map.of("orderId", orderId, "amount", amount))
                .retrieve().body(PaymentResult.class);
    }

    // same parameters + the exception as the last parameter, same return type
    private PaymentResult chargeFallback(String orderId, BigDecimal amount, Throwable ex) {
        log.warn("Payment service unavailable for order {}: {}", orderId, ex.toString());
        return PaymentResult.pending(orderId);            // degrade gracefully: accept the order, pay later
    }
}

Order in which the annotations wrap the call

Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( your method ) ) ) ) )
Each retry attempt passes through the circuit breaker and is counted by it.
If the breaker is open, CallNotPermittedException is thrown at once (and the fallback is called).

Programmatic use and events

@Service
public class Programmatic {
    private final CircuitBreakerRegistry registry;
    public Programmatic(CircuitBreakerRegistry registry) { this.registry = registry; }

    public String call(Supplier<String> remote) {
        CircuitBreaker cb = registry.circuitBreaker("paymentApi");
        cb.getEventPublisher().onStateTransition(e -> System.out.println("Breaker: " + e.getStateTransition()));
        return cb.executeSupplier(remote);
    }
}

curl localhost:8080/actuator/health                          # breaker state shown when register-health-indicator is true
curl localhost:8080/actuator/circuitbreakers
curl localhost:8080/actuator/metrics/resilience4j.circuitbreaker.calls

Design rules

• Always set timeouts on HTTP clients. A circuit breaker cannot help if calls hang forever.
• Retry only idempotent operations, or send an idempotency key. Retrying a payment without one can charge twice.
• Retry with exponential backoff and jitter to avoid a retry storm that overloads a recovering service.
• Prefer a useful fallback (cached value, default, queued work) to a plain error.
• Do not count business errors (validation, 404) as failures. Use ignore-exceptions.
• Monitor breaker states and alert when a breaker stays open.

Common mistake: Retrying non-idempotent calls, and retrying at every layer. Three layers with 3 retries each turn one user request into 27 calls and can overload the dependency. Retry at one carefully chosen layer.

Interview questions

What is the difference between Retry and Circuit Breaker?

Retry repeats one call to survive a short glitch. The Circuit Breaker watches many calls and stops sending traffic to a dependency that is failing, to protect both sides.

Why must the fallback method have the same signature?

The framework looks for a method with the same parameters plus a Throwable (or a specific exception type) and the same return type, and calls it by reflection.

Does the circuit breaker work across several instances?

No. Each application instance keeps its own state. That is fine because each instance protects its own threads.