Skip to main content

Load Balancing

Load balancing in Beak AI Gateway is the execution of the Routing configuration at request time. The Gateway picks a model from the team's allowlist, honors the configured routing strategy, respects model health, and falls back or retries according to policy — all before the request leaves the Gateway.

Selection Order​

For each incoming request, the Gateway resolves a model in this order:

  1. Team allowlist — Only models the team is permitted to call are considered.
  2. Health check — Models currently in cooldown (after a recent failure) are excluded.
  3. Type match — For /chat/completions requests, only Chat models qualify; for /embeddings, only Embedding models, etc.
  4. Routing strategy — Applies the strategy (Simple Shuffle, Least Busy, Latency Based, Cost Based, Usage Based) to pick one model from the filtered pool.

If the pool is empty — every eligible model is in cooldown — the Gateway tries the global Fallback Models list. If that fails too, the retry policy applies.

Health Gating​

Every model tracks a small health window. A failure (5xx from the provider, timeout, provider-side rate limit) excludes the model from routing for Cooldown Time seconds. During the cooldown, the Gateway re-probes with a trickle of traffic so a recovering provider can come back online without waiting the full window.

Failover Chain​

On a failure mid-request, the Gateway:

  1. Counts it against the request's retry budget.
  2. Waits Retry After seconds.
  3. Re-enters the selection order with the failed model excluded.
  4. Picks the next-best model and retries.

If the team's allowlist is exhausted, the Gateway pulls from the Fallback Models list. If that is also exhausted, the request fails with the last provider error surfaced to the caller.

Interaction With Team Rate Limits​

Team TPM and RPM are enforced at the Gateway edge, before routing. A request that would exceed the team's budget or rate limit is rejected immediately with a 429 — it never touches the routing layer. This keeps a noisy team from consuming cooldown slots on behalf of quieter teams.

Best Practices​

  • Keep the routing strategy consistent with team workload. Cost Based is wrong for a user-facing chat agent; Latency Based is wrong for a batch job.
  • Watch Cooldown Time × failure rate. If half the fleet is in cooldown half the time, the real problem is provider instability — no routing strategy rescues that.
  • Size Fallback Models to cover catastrophic failure of the primary providers, not ordinary blips. The fallback chain is a safety net, not a capacity plan.