Skip to main content

Routing

Routing configures how the Gateway picks a model when more than one is eligible, how long a failing model is excluded from the pool, what fallback models to try after failures, and how aggressively the Gateway retries before giving up.

All values on this page are global — they apply to every team and every request unless overridden at the team level. Click Save Configuration to persist.

Routing Strategy​

Five strategies are supported:

  • Simple Shuffle (default) — Round-robin across eligible models. Fairest, simplest, and the right default when models are interchangeable.
  • Least Busy — Picks the model with the fewest in-flight requests. Best when models have asymmetric concurrency limits.
  • Latency Based — Picks the model with the lowest recent P95 latency. Useful when providers have regional performance differences that move around.
  • Cost Based — Picks the cheapest eligible model per request. Good for batch or low-priority workloads.
  • Usage Based — Prefers the model that is furthest from its TPM/RPM ceiling. Smooths traffic across an uneven fleet.

The strategy only disambiguates between models that are both eligible (team's allowlist) and healthy (not in cooldown). See Load Balancing for how the strategies interact with health checks.

Cooldown Time​

When a model fails (provider 5xx, timeout, rate limit from the provider side), the Gateway excludes it from routing for the configured Cooldown Time (seconds). The default is 300 seconds. Shorter cooldowns recover faster but may thrash against a partially degraded provider; longer cooldowns stabilize routing but keep capacity offline during real incidents.

Fallback Models​

A comma-separated list of model Display Names the Gateway tries when every eligible model in the team's allowlist is in cooldown. The fallback list is global — it fires regardless of which team made the request, so keep the entries broadly useful (e.g. gpt-4o, claude-3-opus).

Retries​

  • Number of Retries — How many times the Gateway retries a single request before surfacing the error to the caller. Default is 3.
  • Retry After (seconds) — Wait between retries. Default is 60 seconds.

Retries are counted across the fallback chain — a request that fails on Model A, retries and fails on Model B, then succeeds on Model C consumes two retries.

Best Practices​

  • Start on Simple Shuffle. It is the safest strategy and the one where any regression will be most obvious.
  • Keep Cooldown Time at 300 seconds unless you have evidence of frequent flaps. Shorter values make incident post-mortems harder.
  • Use Fallback Models as an emergency escape hatch, not as load-shedding policy. If fallbacks fire routinely, scale the primary pool instead.
  • Never set Number of Retries above 5. Agent-side timeouts are not designed for long retry chains, and the extra retries usually surface as slow UX rather than improved success rates.