Gateways

A gateway is a provider that fans out to many upstream LLMs behind a single API key. AgentBreeder ships with two built-in gateway presets — LiteLLM (self-hosted) and OpenRouter (SaaS) — surfaced alongside direct providers in the /models → Gateways tab.

This page covers when to use which, how to configure them, and how 3-segment model refs work.

Direct provider vs. gateway — at a glance

Both kinds use the same OpenAICompatibleProvider under the hood — the only difference is parsing.

When to pick which

LiteLLM — choose when you want to self-host, enforce per-agent virtual keys + budgets, run caching locally, or unify cost tracking across your own pool of provider keys. AgentBreeder's docker-compose stack ships a LiteLLM proxy on :4000. See Model Gateway for the full LiteLLM setup.

OpenRouter — choose when you want zero infrastructure, per-token pay-as-you-go pricing across 100+ models, or rapid model evaluation. One key, every model. The catalog ships default_headers for HTTP-Referer and X-Title so OpenRouter attributes calls to your AgentBreeder workspace.

Direct providers — choose for lowest latency and simplest billing when you only need one upstream. Skip the gateway hop entirely.

You can mix all three in the same workspace — gateways and direct providers coexist in the catalog.

3-segment model refs

Once a gateway is configured, reference its models in agent.yaml using a 3-segment ref:

# OpenRouter — pick any model from openrouter.ai/models
model:
  primary: openrouter/moonshotai/kimi-k2

# LiteLLM — model_name is whatever you defined in litellm_config.yaml
model:
  primary: litellm/anthropic/claude-sonnet-4
  fallback: litellm/openai/gpt-4o

The parser splits on / exactly twice:

• segment 1 → gateway name (must be a type: gateway catalog entry)
• segment 2 → upstream provider (forwarded to the gateway as the <upstream>/ prefix)
• segment 3 → model id (rest of the string, may contain extra /)

The wire-level model field sent to the gateway is <segment-2>/<segment-3>, which is the canonical form both LiteLLM and OpenRouter expect.

Configuring a gateway from the dashboard

1. Open /models and switch to the Gateways tab.
2. Find litellm or openrouter and click Configure.
3. Paste the API key. It is written to your workspace secrets backend (keychain locally, AWS Secrets Manager / Vault when self-hosted) under the deterministic name <gateway>/api-key — never to the database.
4. The row flips to a green Configured badge.

Configure flow is identical to direct providers — the only visible difference on a gateway row is the small gateway badge.

Configuring a gateway from agent.yaml

Catalog defaults are usually correct. To override (e.g. point LiteLLM at a remote proxy), use the optional gateways: block:

name: my-agent
version: 1.0.0
team: platform
owner: alice@example.com

model:
  primary: litellm/anthropic/claude-sonnet-4

gateways:
  litellm:
    url: https://litellm.platform.example.com
    api_key_env: PLATFORM_LITELLM_KEY
    fallback_policy: fastest          # advisory; not enforced yet
  openrouter:
    api_key_env: TEAM_OPENROUTER_KEY  # override the catalog default

deploy:
  cloud: aws

Each field is optional and falls back to the catalog default. fallback_policy is advisory — it'll be enforced once Track A's workspace primitive lands (#146).

Environment variables

Each gateway entry declares an env var for its api key in engine/providers/catalog.yaml:

Setting <NAME>_BASE_URL (uppercase) overrides the catalog base_url at runtime — handy for swapping a local LiteLLM proxy for a managed one without editing the catalog.

How it routes

agent.yaml model.primary: openrouter/moonshotai/kimi-k2
        │
        ▼
engine/providers/catalog.parse_gateway_ref
  → GatewayModelRef(gateway="openrouter", upstream="moonshotai", model="kimi-k2")
        │
        ▼
engine/providers/openai_compatible.from_catalog
  → OpenAICompatibleProvider(
        base_url=https://openrouter.ai/api/v1,
        api_key=$OPENROUTER_API_KEY,
        default_headers={HTTP-Referer, X-Title},
        default_model="moonshotai/kimi-k2",
    )
        │
        ▼
POST https://openrouter.ai/api/v1/chat/completions
  body: {"model": "moonshotai/kimi-k2", "messages": [...]}

LiteLLM follows the same path — it just terminates at your proxy on :4000 instead of OpenRouter's edge.

Adding a custom gateway

Drop a ~/.agentbreeder/providers.local.yaml entry:

version: 1
providers:
  myproxy:
    type: gateway
    base_url: https://gw.internal.example.com/v1
    api_key_env: MYPROXY_KEY
    default_headers:
      X-Tenant: agentbreeder

Restart the API server. myproxy now appears on the Gateways tab and accepts 3-segment refs (myproxy/anthropic/claude-sonnet-4).

Reference

• Catalog source: engine/providers/catalog.yaml
• 3-segment parser: engine.providers.catalog.parse_gateway_ref
• JSON Schema: engine/schema/agent.schema.json → gateways
• Issue: #164 — feat(gateways): LiteLLM + OpenRouter as first-class
• Companion: Model Gateway — deep dive on the LiteLLM proxy specifically
• Companion: Providers — direct OpenAI-compatible providers