Skip to main content

Route models and control access

A route maps a requested model name to the provider that serves it. In AI Gateway v2, the route table is the models list on each entry in providers. Change the provider behind a model by moving the model name to another provider, without updating clients.

Route a specific model​

List each model under the provider that serves it:

values.yaml
ai-gateway-v2:
  providers:
    - kind: openai
      baseUrl: https://api.openai.com
      models: ['gpt-5', 'gpt-5-mini']
      credentialSecretRef:
        name: ai-gateway-v2-openai
        key: api-key
    - kind: anthropic
      baseUrl: https://api.anthropic.com
      models: ['claude-sonnet-5']
      credentialSecretRef:
        name: ai-gateway-v2-anthropic
        key: api-key

Write each model name exactly as the provider expects it. The gateway sends the name upstream unchanged, and the pricing catalog uses the same name to price the request. claude-sonnet-4-5 and claude-sonnet-4-5-20250929 are two different routes and two different prices.

How the gateway picks a provider​

The gateway looks up the model named in the request within the client API the request arrived on. A model listed under an anthropic provider is served on the Anthropic Messages API, and a model listed under an openai provider is served on Chat Completions and the Responses API. The gateway does not translate between APIs, so a client must call a model through an API its provider speaks.

Within one client API, a model name maps to exactly one provider. Listing the same model under two providers that share an API, such as anthropic and bedrock-anthropic, is a configuration error, and the new pods fail to start. Use distinct model names for the two, such as claude-sonnet-5 and us.anthropic.claude-sonnet-5.

There is no default route. A request for a model that no provider lists returns 404 with no provider is configured for model <MODEL>, so only the models you list are reachable. When no provider serves the Responses API or Gemini streaming, requests to those endpoints return 501 instead.

Control which models each caller can use​

Model access limits each caller to the models their directory grants cover. With model access on, the gateway refuses a request for a model the caller has no grant for with 403, before it calls any provider. The model discovery endpoints list only the models the caller can use.

Grants come from model sets, named collections of models that you manage on the Models page in the console's administration experience and grant to users or groups. The All Users set is held by every active user. It starts out granting every model, so turning model access on changes nothing until you narrow it.

Turn model access on in the chart values:

values.yaml
ai-gateway-v2:
  modelAccess:
    enabled: true

Model access uses the same receiver, token, and certificate authority as budget enforcement, and the directory connection used for virtual API keys. The chart refuses to render when one of these is missing. See Configure AI Gateway v2 for the platform values it depends on.

Model access fails closed for both requests and discovery calls. If the gateway can't get a decision for a request, it refuses the request. If it can't get a decision for a discovery call, it returns 503 with model access unavailable instead of the full model list.

Discover models​

Clients list the models they can call through the discovery endpoints. Each endpoint returns the response shape its client family expects:

EndpointResponse shapeLists
GET /v1/modelsOpenAIModels served on Chat Completions and Responses
GET /anthropic/v1/modelsAnthropicModels served on Anthropic Messages
GET /v1beta/modelsGeminiModels served on the Gemini API, with their actions

A GET /v1/models request that carries an anthropic-version header returns the Anthropic shape and models.

Where the gateway knows a model's context window, it includes it in the response: context_window in the OpenAI shape, max_input_tokens in the Anthropic shape, and inputTokenLimit in the Gemini shape. The gateway ships a catalog of context windows generated from public model metadata. A model missing from the catalog is listed without a context window, so clients fall back to their own default.

Retries and timeouts​

AI Gateway v2 applies one fixed resilience policy to every provider:

  • Retries. The gateway retries a request up to two times when the connection to the provider fails, or when the provider answers 502, 503, or 504. It retries only before any response bytes reach the client, and always against the same provider.
  • Response headers. The gateway waits up to 5 minutes for the provider to start responding.
  • Stalled streams. The gateway ends a stream that sends nothing for 5 minutes.
  • Stream lifetime. The gateway ends any stream after lifecycle.maxStreamDuration, which defaults to 15m. The pod termination grace period and the rollout drain are sized from this value, so raise it only when your clients hold longer streams.

Budget admission runs once per request, before the provider call, and the request is charged once for the usage the provider reports. Retries don't add charges.

Limits worth knowing​

  • Each model is served by one provider. AI Gateway v2 has no weighted routing across providers and no failover to a second provider.
  • AI Gateway v2 has no catch-all route. Every model a client can call is listed in providers.
  • The route table is independent of the Envoy-based AI Gateway's routes. If you run both during a migration, add a model to both.

Next steps​