Resilience (Rust)

Resilience Patterns

ai-lib-rust includes production-grade reliability patterns out of the box.

Circuit Breaker

Prevents cascading failures by stopping requests to failing providers:

States:

• Closed — Normal operation, requests flow through
• Open — Too many failures, requests immediately rejected
• Half-Open — After cooldown, allows a test request

Configuration:

export AI_LIB_BREAKER_FAILURE_THRESHOLD=5
export AI_LIB_BREAKER_COOLDOWN_SECS=30

The circuit opens after FAILURE_THRESHOLD consecutive failures and stays open for COOLDOWN_SECS before testing.

Rate Limiter

Token bucket algorithm prevents exceeding provider rate limits:

export AI_LIB_RPS=10    # Max requests per second
export AI_LIB_RPM=600   # Max requests per minute

Requests beyond the limit are queued rather than rejected, providing smooth throughput.

Backpressure

Limits concurrent in-flight requests with a semaphore:

export AI_LIB_MAX_INFLIGHT=50

When the limit is reached, new requests wait until a slot opens.

Retry

Exponential backoff retry driven by the protocol manifest’s retry policy:

# In the provider manifest
retry_policy:
  strategy: "exponential_backoff"
  max_retries: 3
  initial_delay_ms: 1000
  max_delay_ms: 30000
  retryable_errors:
    - "rate_limited"
    - "overloaded"
    - "server_error"

Only errors classified as retryable trigger retries. Authentication errors, for example, fail immediately.

Combining Patterns

All resilience patterns work together. A typical request flow:

1. Backpressure — Wait for a slot if at max inflight
2. Circuit Breaker — Reject immediately if circuit is open
3. Rate Limiter — Wait for a token if rate limited
4. Execute — Send the request
5. Retry — If retryable error, wait and retry
6. Update — Record success/failure for circuit breaker

Observability

Monitor resilience state at runtime:

// Check circuit breaker state
let state = client.circuit_state();
println!("Circuit: {:?}", state); // Closed, Open, HalfOpen

// Check current inflight count
let inflight = client.current_inflight();

Next Steps

• Advanced Features — Embeddings, cache, plugins
• AiClient API — Client usage