Hedged requests are a tail-latency optimization. The idea is simple:
Instead of waiting for one server that might be slow, ask multiple identical servers the same question and use whichever responds first.
This intentionally spends extra compute to achieve lower latency.
Why is this needed?
Imagine you have three replicas of the same service.
Replica A
Replica B
Replica C
Each replica has the same data and can answer the request.
Normally, your load balancer chooses one.
Client
|
+----> Replica A
Suppose the response times are:
| Request | Replica A |
|---|---|
| 1 | 12 ms |
| 2 | 15 ms |
| 3 | 14 ms |
| 4 | 180 ms |
Most requests are fast.
Occasionally a replica experiences:
- CPU scheduling delays
- GC pauses
- disk I/O
- network congestion
- noisy neighbors
- container throttling
That one slow request becomes your tail latency.
Normal request
Client
|
+------> Replica A
|
|
180 ms
User waits 180 ms.
Hedged request
Instead:
Client
+---------> Replica A
+---------> Replica B
Both start processing immediately.
Suppose
Replica A = 180 ms
Replica B = 42 ms
Timeline
0 ms
|
|------Send to A
|------Send to B
42 ms
|
|------Replica B replies
43 ms
|
|------Cancel request to A
180 ms
|
|------A finishes but result discarded
User experiences
42 ms
instead of
180 ms
Why does this work?
Most latency distributions look like this.
Response Times
10ms ███████████████████
15ms ███████████
20ms ███████
30ms ████
40ms ██
150ms █
300ms █
Most requests are fast.
A tiny percentage are extremely slow.
Those few slow ones dominate your:
- P95
- P99
- P99.9
Hedged requests eliminate many of those slow outliers.
Example
Without hedging
Request 1 18 ms
Request 2 20 ms
Request 3 19 ms
Request 4 240 ms
Request 5 16 ms
P99 becomes terrible because of Request 4.
Now hedge.
Request 4
Replica A = 240 ms
Replica B = 25 ms
Winner = 25 ms
Now
18
20
19
25
16
Tail disappears.
Why cancel the slower request?
Suppose both continue running.
Replica A
180 ms of CPU
Replica B
42 ms of CPU
You wasted
180 + 42 = 222 ms
Instead
42 ms
cancel
A stops processing
Less wasted CPU.
This requires deadline propagation and context cancellation, which is why many RPC frameworks (such as gRPC) propagate cancellation signals to the server.
Where does the 2× load come from?
Normally
1000 requests/sec
↓
1000 upstream requests/sec
With immediate hedging
1000 user requests
↓
2000 upstream requests
Each client request becomes two backend requests.
Hence
~2× network
~2× CPU
~2× connections
Why is Netflix okay with that?
Netflix optimizes for viewer experience.
Imagine opening Netflix.
If recommendations take
80 ms
instead of
350 ms
the UI feels instant.
Reducing latency can improve engagement and user satisfaction enough to justify the additional infrastructure cost.
Is it always exactly 2×?
No.
A common optimization is delayed hedging.
Instead of sending both immediately:
0 ms
Send to A
Wait 20 ms
If A hasn't responded
↓
Send to B
Timeline
0 ms
A starts
18 ms
A responds
Done
Never send B
Only one request was needed.
For slow requests
0 ms
A starts
20 ms
Still waiting
↓
Send B
40 ms
B responds
Cancel A
Many production systems use this approach because it greatly reduces the extra load while still cutting tail latency.
When should you use hedged requests?
They work well when:
- The operation is idempotent (safe to execute multiple times).
- Multiple replicas can process the same request independently.
- Low latency is more valuable than the extra resource usage.
- The request path is latency-sensitive (e.g., user-facing APIs with strict P99 SLOs).
Avoid or use caution when:
- The operation has side effects (e.g., charging a credit card or creating an order), unless protected by idempotency keys.
- The backend is already heavily overloaded, since duplicate requests can make overload worse.
- Replicas share the same bottleneck (for example, all waiting on the same slow database), in which case hedging may provide little benefit.
Relation to fan-out
Hedging is a tail-latency tactic layered on top of 04 — Fan-Out / Fan-In.
Suppose Netflix’s homepage needs data from five services:
Home API
|
--------------------------
| | | | |
User Ads ML Video Ratings
If any one service is slow, the entire page is delayed.
Netflix may hedge requests to each dependency:
Ratings Service
Replica A <-----+
+---- Client
Replica B <-----+
This reduces the chance that a single slow replica determines the overall page latency, which is especially valuable in fan-out architectures where the slowest branch often dictates the total response time.
Local graph
Linked from 7 notes
Q2 Answer — Hedging Trade-off
Worked answer to Fan-Out/Fan-In Practice Q2: the load-vs-latency math of hedged requests, when to enable them, and what to instrument first to justify the decision.
Q9 Answer — Validating Hedging and Deadline Propagation
Worked answer to Fan-Out/Fan-In Practice Q9: fault-injection, load testing, and canary comparison for validating a deadline-propagation and hedging rewrite before it reaches production.
08 — Retry with Exponential Backoff and Jitter
Retry transient failures with exponentially increasing wait times and randomised jitter to prevent thundering-herd recovery storms. The foundational pattern for resilient RPC.
04 — Fan-Out / Fan-In
Decompose a request into parallel sub-tasks (fan-out), execute concurrently, then merge results (fan-in). The foundational pattern for latency-bound aggregation.
11 — Reliability Patterns (Microservice Building Blocks)
Timeout, Rate Limiter, Fallback, and Adaptive Concurrency — the remaining resilience building blocks beyond Circuit Breaker, Retry, Bulkhead, and Hedged Requests, which already have their own chapters in this book.
Patterns
A book-shaped table of contents for reusable engineering patterns spanning object-oriented design, enterprise architecture, distributed systems, messaging, APIs, cloud infrastructure, observability, security, concurrency, AI/agentic systems, and organizational design — grounded in production experience at scale.
System Design
Principal/Staff-level system design reference collection for MAANG interview preparation — observability pipelines, distributed systems, reliability engineering, and beyond.
Related notes
Q1 Answer — Search Fan-Out Design
Worked answer to Fan-Out/Fan-In Practice Q1: partitioning, deadline propagation, partial-result policy, and instrumentation priority for a 200-shard search API at 150ms P99.
Q2 Answer — Hedging Trade-off
Worked answer to Fan-Out/Fan-In Practice Q2: the load-vs-latency math of hedged requests, when to enable them, and what to instrument first to justify the decision.
Q3 Answer — Context Cancellation Leak
Worked answer to Fan-Out/Fan-In Practice Q3: diagnosing a ghost-request leak where client-visible errors look healthy but infra cost and downstream CPU are elevated.
Q4 Answer — Aggregator Bottleneck
Worked answer to Fan-Out/Fan-In Practice Q4: min-heap merge strategy for a 500-shard top-K aggregation, its complexity, and how to keep aggregator latency from contaminating per-shard dashboards.