Gateways (LiteLLM & OpenRouter)

A gateway is a provider that fans out to many upstream LLMs behind a single API key. AgentBreeder ships with two built-in gateway presets — LiteLLM (self-hosted) and OpenRouter (SaaS) — surfaced alongside direct providers in the /models → Gateways tab.

This page covers when to use which, how to configure them, how 3-segment model refs work, and the full LiteLLM proxy deep-dive (virtual keys, budgets, caching, guardrails, observability).

Direct provider vs. gateway — at a glance

Both kinds use the same OpenAICompatibleProvider under the hood — the only difference is parsing.

When to pick which

LiteLLM — choose when you want to self-host, enforce per-agent virtual keys + budgets, run caching locally, or unify cost tracking across your own pool of provider keys. AgentBreeder's docker-compose stack ships a LiteLLM proxy on :4000. See Self-hosted LiteLLM below for the full setup.

OpenRouter — choose when you want zero infrastructure, per-token pay-as-you-go pricing across 100+ models, or rapid model evaluation. One key, every model. The catalog ships default_headers for HTTP-Referer and X-Title so OpenRouter attributes calls to your AgentBreeder workspace.

Direct providers — choose for lowest latency and simplest billing when you only need one upstream. Skip the gateway hop entirely.

You can mix all three in the same workspace — gateways and direct providers coexist in the catalog.

3-segment model refs

Once a gateway is configured, reference its models in agent.yaml using a 3-segment ref:

# OpenRouter — pick any model from openrouter.ai/models
model:
  primary: openrouter/moonshotai/kimi-k2

# LiteLLM — model_name is whatever you defined in litellm_config.yaml
model:
  primary: litellm/anthropic/claude-sonnet-4
  fallback: litellm/openai/gpt-4o

The parser splits on / exactly twice:

• segment 1 → gateway name (must be a type: gateway catalog entry)
• segment 2 → upstream provider (forwarded to the gateway as the <upstream>/ prefix)
• segment 3 → model id (rest of the string, may contain extra /)

The wire-level model field sent to the gateway is <segment-2>/<segment-3>, which is the canonical form both LiteLLM and OpenRouter expect.

Configuring a gateway from Studio

1. Open /models and switch to the Gateways tab.
2. Find litellm or openrouter and click Configure.
3. Paste the API key. It is written to your workspace secrets backend (keychain locally, AWS Secrets Manager / Vault when self-hosted) under the deterministic name <gateway>/api-key — never to the database.
4. The row flips to a green Configured badge.

Configure flow is identical to direct providers — the only visible difference on a gateway row is the small gateway badge.

Configuring a gateway from agent.yaml

Catalog defaults are usually correct. To override (e.g. point LiteLLM at a remote proxy), use the optional gateways: block:

name: my-agent
version: 1.0.0
team: platform
owner: alice@example.com

model:
  primary: litellm/anthropic/claude-sonnet-4

gateways:
  litellm:
    url: https://litellm.platform.example.com
    api_key_env: PLATFORM_LITELLM_KEY
    fallback_policy: fastest          # advisory; not enforced yet
  openrouter:
    api_key_env: TEAM_OPENROUTER_KEY  # override the catalog default

deploy:
  cloud: aws

Each field is optional and falls back to the catalog default. fallback_policy is advisory — it'll be enforced once Track A's workspace primitive lands (#146).

Environment variables

Each gateway entry declares an env var for its api key in engine/providers/catalog.yaml:

Setting <NAME>_BASE_URL (uppercase) overrides the catalog base_url at runtime — handy for swapping a local LiteLLM proxy for a managed one without editing the catalog.

How it routes

agent.yaml model.primary: openrouter/moonshotai/kimi-k2
        │
        ▼
engine/providers/catalog.parse_gateway_ref
  → GatewayModelRef(gateway="openrouter", upstream="moonshotai", model="kimi-k2")
        │
        ▼
engine/providers/openai_compatible.from_catalog
  → OpenAICompatibleProvider(
        base_url=https://openrouter.ai/api/v1,
        api_key=$OPENROUTER_API_KEY,
        default_headers={HTTP-Referer, X-Title},
        default_model="moonshotai/kimi-k2",
    )
        │
        ▼
POST https://openrouter.ai/api/v1/chat/completions
  body: {"model": "moonshotai/kimi-k2", "messages": [...]}

LiteLLM follows the same path — it just terminates at your proxy on :4000 instead of OpenRouter's edge.

Adding a custom gateway

Drop a ~/.agentbreeder/providers.local.yaml entry:

version: 1
providers:
  myproxy:
    type: gateway
    base_url: https://gw.internal.example.com/v1
    api_key_env: MYPROXY_KEY
    default_headers:
      X-Tenant: agentbreeder

Restart the API server. myproxy now appears on the Gateways tab and accepts 3-segment refs (myproxy/anthropic/claude-sonnet-4).

Self-hosted LiteLLM

The LiteLLM preset routes every inference call through a self-hosted proxy that sits between your agents and every LLM provider. When you set model.gateway: litellm (or use a litellm/... 3-segment ref) in your agent.yaml, all inference calls route through the proxy instead of calling providers directly.

model:
  primary: claude-sonnet-4
  fallback: gpt-4o
  gateway: litellm        # route through the proxy
  temperature: 0.7

Why route through LiteLLM

Without the gateway, each deployed agent holds its own API keys and calls providers directly. With the gateway:

How it works

agent.yaml (model.gateway: litellm)
        │
        ▼
AgentBreeder engine
  ├── RBAC check
  ├── Mints per-agent virtual key (sk-agent-<name>)
  └── Injects LITELLM_API_KEY + LITELLM_BASE_URL into container
        │
        ▼
LiteLLM proxy (:4000)
  ├── Validates virtual key
  ├── Enforces team budget
  ├── Runs PII guardrail
  ├── Checks Redis cache
  ├── Routes to provider (with retries + fallback)
  └── Logs OTEL span → AgentBreeder tracing
        │
        ▼
Provider (Anthropic / OpenAI / Google / Ollama / ...)

AgentBreeder owns governance (RBAC, audit, team budgets). LiteLLM handles routing (fallbacks, retries, caching, provider translation). Neither owns the other's domain.

Supported providers

The gateway can route to any LiteLLM-supported provider. Out of the box, the quickstart config includes:

Add more models by editing deploy/litellm_config.yaml:

model_list:
  - model_name: my-custom-alias
    litellm_params:
      model: anthropic/claude-opus-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

Virtual keys

Every agent gets a scoped virtual key (sk-agent-<name>) automatically minted at deploy time. The key:

• Is injected into the deployed container as LITELLM_API_KEY
• Is scoped to the agent's allowed models (from agent.yaml)
• Is attributed to the agent's team for cost tracking
• Can be revoked from Studio without redeploying the agent

View and manage keys from Studio under Settings → API Keys, or via the CLI:

agentbreeder describe my-agent --keys

Cost tracking

Every LLM call through the gateway is tracked and attributed to the calling agent and its team. View spend in Studio under Costs, or via the API:

# Spend for a specific team
GET /api/v1/costs?team=engineering

# Spend by model
GET /api/v1/costs?group_by=model

Set team budgets when creating a team:

agentbreeder team create engineering --budget 500 --budget-period 30d

The gateway sends alerts at 85% and 95% of the budget before enforcing the hard limit.

Guardrails

Two guardrails are enabled by default when the gateway is active:

Presidio PII detection — scans every request for PII (names, emails, credit cards, SSNs) before the call reaches the LLM. Redacts or blocks based on configuration.

Lakera prompt injection — detects attempts to hijack agent behavior through crafted user inputs.

Configure guardrail behavior in deploy/litellm_config.yaml:

guardrails:
  - guardrail_name: presidio-pii
    litellm_params:
      guardrail: presidio
      mode: pre_call          # scan before the LLM call
      output_parse_pii: true  # also scan LLM output

To disable guardrails for a specific agent (not recommended for production):

# agent.yaml
guardrails: []

Caching

The gateway caches LLM responses in Redis. Identical prompts return the cached response without making a provider call — reducing latency and cost.

# deploy/litellm_config.yaml
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379
    ttl: 600   # 10 minutes

Per-request cache control (pass in the request body from your agent):

# Force fresh call, don't use cache
response = client.chat.completions.create(
    model="claude-sonnet-4",
    messages=[...],
    extra_body={"cache": {"no-cache": True}}
)

Fallbacks and retries

Set a fallback model in agent.yaml and the gateway handles automatic failover:

model:
  primary: claude-sonnet-4
  fallback: gpt-4o
  gateway: litellm

If the primary model returns a rate limit error or is unavailable, the gateway automatically retries with the fallback. No code change in your agent — the same LITELLM_BASE_URL endpoint works regardless of which provider responds.

For more complex routing (load balancing across multiple deployments of the same model), configure routing strategies directly in litellm_config.yaml:

router_settings:
  routing_strategy: latency-based-routing  # route to fastest deployment

Running the proxy locally

The proxy is included in the default Docker Compose stack:

docker compose -f deploy/docker-compose.yml up -d   # starts postgres, redis, API, dashboard, and litellm

The LiteLLM admin UI is available at http://localhost:4000/ui. The default master key is sk-agentbreeder-quickstart (set LITELLM_MASTER_KEY in .env for production).

To verify the gateway is healthy:

curl http://localhost:4000/health

Observability

Every gateway call emits an OpenTelemetry span with:

• Model used, provider, latency
• Token counts (input / output / total)
• Cost in USD
• Virtual key alias (agent attribution)
• x-litellm-call-id — correlated to AgentBreeder audit log entries

View traces in Studio under Tracing, or connect your own OTEL collector by updating the endpoint in litellm_config.yaml.

The Gateway view in Studio

The Gateway view in AgentBreeder Studio shows:

• Live status of each configured provider
• Model catalog with pricing
• Request log with latency, token counts, and cost per call
• Cost comparison table across providers

Skipping the gateway

If you need to bypass the gateway for a specific agent (e.g., a local dev agent that calls Ollama directly):

model:
  primary: ollama/llama3.2
  # no gateway field — calls Ollama directly

Direct calls skip virtual key minting, budget enforcement, guardrails, and caching. Use only for local development.

Reference

• Catalog source: engine/providers/catalog.yaml
• 3-segment parser: engine.providers.catalog.parse_gateway_ref
• JSON Schema: engine/schema/agent.schema.json → gateways
• Issue: #164 — feat(gateways): LiteLLM + OpenRouter as first-class
• Companion: Providers — direct OpenAI-compatible providers