← Back to Glossary

Network Reliability

Network Reliability is a critical concern in microservices architectures, where communication happens over a potentially unreliable network. Unlike monolithic applications where components communicate through direct method calls, microservices must handle network failures, latency, and partial outages gracefully.

The Challenge of Distributed Communication

In microservices, what were once in-memory function calls become network requests across potentially unreliable connections. This introduces several failure modes that don't exist in monolithic applications.

Common Network Issues

  • • Network partitions and connectivity loss
  • • High latency and timeouts
  • • Service unavailability
  • • Partial failures
  • • Message loss or duplication

Impact on System

  • • Cascading failures
  • • Resource exhaustion
  • • Degraded user experience
  • • Data inconsistency
  • • System-wide outages
Real-world Example: During AWS's 2017 S3 outage, many applications relying on S3 for various services experienced cascading failures because they weren't designed to handle prolonged service unavailability gracefully.

Circuit Breakers

The Circuit Breaker pattern prevents an application from repeatedly trying to execute an operation that is likely to fail. It allows the system to fail fast and prevent cascading failures, while periodically checking if the underlying problem has been resolved.

How Circuit Breakers Work

Circuit breakers monitor the success and failure rates of operations. Based on configurable thresholds, they transition between three states to protect the system from overload.

🟢
CLOSED

Normal operation

  • • Requests flow normally
  • • Monitor failure rate
  • • Count consecutive failures

→ OPEN if failures exceed threshold

🔴
OPEN

Failing fast

  • • Immediately reject requests
  • • Return cached response or error
  • • Wait for timeout period

→ HALF-OPEN after timeout

🟡
HALF-OPEN

Testing recovery

  • • Allow limited requests through
  • • Test if service recovered
  • • Monitor success rate

→ CLOSED if successful

→ OPEN if still failing

Circuit Breaker Configuration

Key Parameters
  • • Failure Threshold: Number/percentage of failures to trigger OPEN
  • • Recovery Timeout: Time to wait before trying HALF-OPEN
  • • Success Threshold: Successes needed to return to CLOSED
  • • Request Volume Threshold: Minimum requests before evaluation
Fallback Strategies
  • • Return cached/default response
  • • Degrade functionality gracefully
  • • Route to alternative service
  • • Queue requests for later processing
Example: Netflix uses circuit breakers extensively in their microservices. When their recommendation service becomes unavailable, circuit breakers prevent the homepage from hanging and instead show a default set of popular content.

Exponential Backoff

Exponential Backoff is an error handling strategy where you periodically retry a failed request with progressively longer wait times between retries. This prevents overwhelming a struggling service while still attempting to recover from transient failures.

How Exponential Backoff Works

Instead of immediately retrying or using fixed intervals, exponential backoff increases the delay between retries exponentially, often with some randomization (jitter) to prevent thundering herd problems.

Retry Progression Example

Without Jitter:

  • • Attempt 1: Immediate (0s)
  • • Attempt 2: Wait 1s
  • • Attempt 3: Wait 2s
  • • Attempt 4: Wait 4s
  • • Attempt 5: Wait 8s
  • • Max wait: 30s (cap)

With Jitter (Recommended):

  • • Attempt 1: Immediate (0s)
  • • Attempt 2: Wait 0.5-1.5s
  • • Attempt 3: Wait 1-3s
  • • Attempt 4: Wait 2-6s
  • • Attempt 5: Wait 4-12s
  • • Random spread prevents spikes

Types of Backoff Strategies

Linear Backoff

Fixed increase: 1s, 2s, 3s, 4s...

  • • Simple to implement
  • • Predictable behavior
  • • May overwhelm during recovery
Exponential Backoff

Exponential increase: 1s, 2s, 4s, 8s...

  • • Backs off aggressively
  • • Better for overloaded services
  • • May delay recovery unnecessarily
Exponential with Jitter

Random variation: 0.5-1.5s, 1-3s, 2-6s...

  • • Prevents thundering herd
  • • Spreads load during recovery
  • • Recommended approach
Full Jitter

Random within range: 0-1s, 0-2s, 0-4s...

  • • Maximum randomization
  • • Best for high-traffic systems
  • • Used by AWS SDK

Implementation Considerations

Best Practices
  • • Set maximum retry limits
  • • Use timeout caps (e.g., 30s max)
  • • Add jitter to prevent synchronized retries
  • • Consider request idempotency
  • • Monitor retry patterns
Common Pitfalls
  • • Infinite retry loops
  • • Thundering herd on recovery
  • • Not differentiating error types
  • • Retry on non-transient errors
  • • Ignoring upstream rate limits
Example: AWS services use exponential backoff with jitter in their SDKs. When S3 returns a 503 Service Unavailable error, the SDK automatically retries with increasing delays, preventing clients from overwhelming the service during recovery.

Combining Reliability Patterns

For maximum resilience, circuit breakers and exponential backoff are often used together along with other patterns to create a comprehensive network reliability strategy.

Reliability Pattern Stack

  • • Timeouts: Prevent hanging requests
  • • Retries with Backoff: Handle transient failures
  • • Circuit Breakers: Fail fast during outages
  • • Bulkhead Pattern: Isolate critical resources
  • • Rate Limiting: Protect against overload

Monitoring & Observability

  • • Track circuit breaker state changes
  • • Monitor retry rates and patterns
  • • Alert on excessive failure rates
  • • Measure recovery times
  • • Analyze failure correlation
Example: Spotify combines all these patterns in their music streaming services. During peak hours, they use circuit breakers to protect recommendation services, exponential backoff for playlist synchronization, and bulkheads to ensure music playback remains available even when social features fail.

Tools and Frameworks

Hystrix (Netflix)

Circuit breaker library with dashboard and real-time monitoring

Resilience4j

Lightweight fault tolerance library for Java functional programming

Polly (.NET)

Resilience and transient-fault-handling library for .NET

Istio Service Mesh

Built-in circuit breaking, retries, and traffic management

AWS SDK

Built-in exponential backoff with jitter for all AWS services

Consul Connect

Service mesh with automatic retry and circuit breaker policies