Direct answer
A timeout only identifies the deadline that expired. Measure each phase to avoid masking connection failures or slow generation with one oversized global timeout.
What this guide helps you solve
Fix API connect, read, first-byte, generation, proxy, and job timeouts without unsafe retries.
Symptoms
The client reports a timeout before connection, before first byte, during generation, or while reading a stream.
Short requests may work while long context, large output budgets, concurrency, or proxy hops exceed a shorter hidden deadline.
Likely causes
- Slow DNS, TCP connection, TLS negotiation, or an unreachable network path.
- Model queueing or generation exceeds the client first-byte or read timeout.
- The SDK, reverse proxy, load balancer, and job runner use inconsistent deadlines.
- Connection-pool starvation delays a request before it reaches the network.
Diagnostic procedure
- 01Establish the boundaryMeasure DNS, TCP, TLS, first-byte, and total duration separately.
- 02Create a baselineUse one short non-streaming request to establish a baseline.
- 03Compare one variableCompare short and long input, output budgets, streaming, and controlled concurrency.
- 04Record decisive evidenceList every timeout in the SDK, proxy, gateway, and outer job; find the shortest one.
curl -sS -o /tmp/timeout-body.json --connect-timeout 10 --max-time 60 -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first=%{time_starttransfer} total=%{time_total} status=%{http_code}
' 'https://<your-api-host>/v1/models' -H "Authorization: Bearer ${API_KEY:?set API_KEY first}"
Remediation
- 01Correct the failing layerSet separate connect, read, and total deadlines based on observed service behavior.
- 02Restore required behaviorReduce context, output budget, and queue pressure before simply increasing timeouts.
- 03Remove temporary workaroundsAlign proxy and job deadlines so outer layers do not terminate healthy longer requests.
connect_timeout = 10s
read_or_idle_timeout = 90s
overall_job_deadline = 120s
retry_attempts = 2
# These values are examples. Calibrate them against actual model latency, proxy limits, and the service SLO.Verification
- Connection failures exit quickly under a clear connect deadline.
- The read and total budget covers expected model latency without being unbounded.
- Connection pools remain available under controlled concurrency.
- Requests with side effects are not blindly replayed after an ambiguous timeout.
Sensitive diagnostic data
- Do not disable every timeout; stalled connections must remain bounded.
- Never include authorization data in timing traces.
- Use idempotency and task state before retrying tool or write operations.
- Escalate with phase timings, final host, model, payload size, and sanitized request IDs.
Official sources and verification scope
This guide is grounded in protocol specifications and official client documentation. Error text, retry headers, and configuration fields may change by service or client version; verify the sources and redact logs and request samples before sharing.
View documentation scope