Docs/Reliability and Error Handling

Reliability and Error Handling

Failover, retries, and error semantics

Token Hub fronts multiple independent upstream routes for every model. When one route fails, the gateway fails over automatically — most transient upstream problems never reach your code.

Automatic failover

Property	Behavior
Retry scope	Retries happen before the first byte of a response — never mid-stream, so you never see duplicated partial answers.
Attempts	Up to two tries per upstream route, bounded by the gateway's global attempt budget.
Retryable statuses	408, 409, 429, 500, 502, 503, 504, 529 (capacity).
Different-model fallback	If all routes for a model are unhealthy the request fails fast with 502 — the gateway never silently substitutes a different model.

Error envelope

Every non-2xx response uses one envelope, so client handling stays uniform:

json
{
  "error": {
    "code": "rate_limited",
    "message": "rate limit exceeded",
    "details": { "retry_after": 12 }
  },
  "request_id": "req_01j9..."
}

Always log request_id — it correlates directly with rows in your console request log and with operator traces.

Status codes

Status	Code	Meaning	Client action
401	invalid_api_key / key_revoked / key_expired	Authentication problem	Fix the key; do not retry
402	budget_exceeded	Org or key spend cap reached	Raise budget in console, then retry
403	model_not_allowed / insufficient_scope	Key lacks access	Use an allowed model or reissue key
404	model_not_found	Unknown model id	Check the catalog; do not retry
429	rate_limited	Key or org rate limit hit	Back off; honor details.retry_after
500	internal	Gateway fault	Retry with backoff
502	no_upstream / upstream_error	All upstream routes unhealthy	Retry after a short backoff
504	upstream_timeout	Upstream exceeded the time budget	Retry with backoff; consider a smaller max_tokens

Client retry strategy

python
import time, random

def with_retry(fn, tries=4):
    for i in range(tries):
        try:
            return fn()
        except Exception as e:
            if i == tries - 1:
                raise
            time.sleep((2 ** i) + random.random() * 0.5)  # 1, 2, 4s + jitter

TipThe gateway already absorbs upstream blips for you — client retries are only for the residual cases (429s from your own rate limits, or full upstream outages). Prefer a few well-jittered retries over aggressive hammering.