Model Gateway

The Chalk Model Gateway is a managed, OpenAI-compatible router for LLM traffic, hosted alongside your Chalk deployment. Point any OpenAI-compatible client at it and it forwards your request to the right provider (OpenAI, Anthropic, or Google Gemini) behind a single API. Because the Model Gateway sits in front of every provider, it is also where you centralize the things you do not want scattered across application code: API keys and budgets, rate limits, automatic fallback between models, and provider credentials.

The same endpoint also serves the Anthropic Messages API, so Anthropic SDKs and Claude Code route through the Model Gateway without a translation layer.

The Model Gateway runs as a service in your Chalk environment, alongside the engine, and the Chalk API routes each request to it. To deploy it, open the Cloud Resource Configuration pane in the Chalk dashboard, add Model Gateway to a resource group from the Add Config menu, where it is listed under Advanced, set its resource request, and choose Save and Apply Service. See Resource Configuration for how services and resource groups work. On a dedicated or self-hosted deployment, running there means your LLM traffic stays inside your own cloud.

​

Endpoint

The Model Gateway exposes the standard OpenAI-compatible paths under this base URL:

The same base URL also serves the Anthropic Messages API. Anthropic SDKs append their own /v1 segment to the base URL they are given, so those paths read with the extra segment:

Use whichever format your client already speaks: the Model Gateway resolves the requested model against your configured providers either way, and a key’s restrictions are enforced on both. See Use with Claude Code for a client configured this way.

If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own API server host. You can find it in the API Server row of chalk config, or as apiServer.value in chalk config --format json. Requests are routed to the calling environment’s Model Gateway, selected by the X-Chalk-Env-Id header described below.

​

Authentication

Every request carries an issued Model Gateway API key as a bearer token, plus the environment to route to:

Model Gateway API keys are issued and revoked from the dashboard or the CLI. See API keys below. A key is shown only once when it is issued, so copy it immediately.

​

Using the Model Gateway

The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header.

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-chalk-api-host>/v1/router",
    api_key="<YOUR_ROUTER_API_KEY>",
    default_headers={"X-Chalk-Env-Id": "<env-id>"},
)

resp = client.chat.completions.create(
    model="openai/gpt-5.6-luna",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Model IDs are always prefixed with their provider, as in openai/gpt-5.6-luna or anthropic/claude-sonnet-5. Run chalk router model list to see every model your configured providers expose.

The same request with curl:

curl https://<your-chalk-api-host>/v1/router/chat/completions \
  -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \
  -H "X-Chalk-Env-Id: <env-id>" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-5.6-luna", "messages": [{"role": "user", "content": "Hello"}]}'

The model you request is resolved against the providers you have configured. A key’s provider restriction, model allow-list, usage pool, and daily token budget are all enforced on every request.

An Anthropic client is configured the same way (the base URL, the key, and the environment header) and calls the Messages API:

from anthropic import Anthropic

client = Anthropic(
    base_url="https://<your-chalk-api-host>/v1/router",
    api_key="<YOUR_ROUTER_API_KEY>",
    default_headers={"X-Chalk-Env-Id": "<env-id>"},
)

resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.content[0].text)

If a request does not work, two CLI commands report what the Model Gateway sees. chalk router status shows whether the Model Gateway is reachable and your harness configuration, and chalk router doctor diagnoses the Model Gateway, the model you asked for, and the harness together.

​

Use with Claude Code

Claude Code speaks the Anthropic Messages API, so the Model Gateway works as its LLM gateway: model requests route through Chalk, where the provider credential, model allow-list, daily budget, and rate limits already live, and each developer holds a revocable Model Gateway API key instead of a provider key.

The CLI configures this for you, using the credentials you are already logged in with:

chalk router claude on

Pass --model <model-id> to pin a model. To start a single session without changing Claude Code’s configuration at all, use chalk router claude launch instead: it writes the Model Gateway settings to a temporary file, hands them to Claude Code with --settings, and removes the file when the session ends, so ~/.claude/settings.json is never read or rewritten. chalk router claude off restores whatever the CLI changed, and chalk router claude status shows the current state.

The rest of this section covers the same setup by hand, which is worth reading if you are scripting it or debugging a connection.

Set the base URL and the key in your shell:

export ANTHROPIC_BASE_URL=https://<your-chalk-api-host>/v1/router
export ANTHROPIC_AUTH_TOKEN=<YOUR_ROUTER_API_KEY>
export ANTHROPIC_CUSTOM_HEADERS="X-Chalk-Env-Id: <env-id>"

Use ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY: the Model Gateway reads the key from the Authorization header, and ANTHROPIC_API_KEY sends it in x-api-key. The Model Gateway accepts either, but ANTHROPIC_API_KEY also needs a one-time approval prompt in Claude Code before it takes effect.

Confirm the Model Gateway answers before starting Claude Code, so a failure points at the Model Gateway rather than at your configuration:

curl -X POST "$ANTHROPIC_BASE_URL/v1/messages" \
  -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \
  -H "X-Chalk-Env-Id: <env-id>" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-4-6", "max_tokens": 1, "messages": [{"role": "user", "content": "."}]}'

A body starting with {"id":"msg_ means the URL, key, and environment header all work. An unknown-model error also confirms them, since the Model Gateway authenticated the request before rejecting the model.

To make the configuration apply everywhere Claude Code runs, including background agents, move the values into the env block of ~/.claude/settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://<your-chalk-api-host>/v1/router",
    "ANTHROPIC_AUTH_TOKEN": "<YOUR_ROUTER_API_KEY>",
    "ANTHROPIC_CUSTOM_HEADERS": "X-Chalk-Env-Id: <env-id>"
  }
}

Run /status in Claude Code to confirm: the Status tab shows an Anthropic base URL line with your Chalk host and an Auth token line naming the variable you set.

Two things to know before rolling this out:

• The Model Gateway API key replaces a claude.ai login. While ANTHROPIC_AUTH_TOKEN is set, a saved claude.ai subscription is unused and its limits do not apply; usage bills against the provider credential the Model Gateway forwards. Features that need a claude.ai identity, such as Remote Control and voice dictation, are unavailable while it is set.
• Serve the Model Gateway’s model names. If a model route exposes a name that is not one of Claude Code’s built-in models, set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 so the /model picker is populated from the Model Gateway’s /v1/models response.

​

Calling the Model Gateway from Chalk

Because the Model Gateway is OpenAI-compatible, you can also call it from inside feature computation with chalk.functions. F.openai_complete issues a chat completion while a query runs. Route it through the Model Gateway rather than OpenAI in one of two ways:

• Set api_server to your Chalk API host.
• Set the OPENAI_BASE_URL environment variable on the execution host and omit api_server.

import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame

df = DataFrame({"questions": ["Recommend some movies like High and Low by Akira Kurosawa"]})
(
    df.with_columns(
        F.openai_complete(
            prompt=_.questions,
            model="o4-mini",
            service_tier="flex",
            api_server="https://api.chalk.ai",  # route through the Model Gateway
        ).alias("result")
    )
    .run()
    .to_arrow()
)

F.openai_complete takes the following arguments:

It returns a struct with the completion text plus prompt_tokens, completion_tokens, total_tokens, model, finish_reason, and the upstream ratelimit_remaining_tokens / ratelimit_remaining_requests headers.

​

Throttling with rate limits

LLM calls are blocking and metered, so throttle them with the policy modifiers chained onto the expression. with_rate_limit caps how often the call may run across every expression that shares its key:

import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame

df = DataFrame({"questions": ["Recommend some movies like Buzzard by Joel Potrykus"]})
(
    df.with_columns(
        F.openai_complete(prompt=_.questions, model="o4-mini", service_tier="flex")
        .with_rate_limit(rate=3, key="openai", per="minute")
        .alias("result")
    )
    .run()
    .to_arrow()
)

The same modifier works on a feature defined in a @features class: chain .with_rate_limit(...) before selecting the .completion field.

with_rate_limit takes:

These policy modifiers compose: chain with_concurrency, with_rate_limit, and with_retry on the same expression to bound in-flight calls, cap the call rate, and retry transient failures with backoff:

(
    F.openai_complete(prompt=_.questions, model="o4-mini", service_tier="flex")
    .with_concurrency(max_concurrent=4, key="my_api")
    .with_rate_limit(rate=100, key="my_api")
    .with_retry(max_retries=3, key="my_api")
)

Reusing one key ties the policies to the same logical resource, so every expression that calls "my_api" shares a single budget: here, at most 4 concurrent calls and 100 calls per second (per defaults to "second"), with each failed call retried up to 3 times.

​

API keys

Issue and revoke Model Gateway API keys from Model Gateway → Access → API keys in the Chalk dashboard, or from the CLI with chalk router api-key (create, list, and revoke). Each key can be scoped at issue time so that a leaked or over-eager client cannot do more than you intend:

Each key tracks its total token usage, and revoking a key takes effect immediately.

​

Usage pools

A usage pool groups API keys so you can manage and limit access by pool rather than key by key. Create a pool, then assign keys to it at issue time. Pools are also managed from the CLI with chalk router usage-pool (create, list, and delete). Rate limits can be scoped to a pool so every key in it shares one budget. Deleting a pool leaves its keys working and removes their pool association.

​

Usage budgets

A usage budget caps how many tokens a key or a pool may consume in a fixed window. Create budgets under Model Gateway → Access → Usage budgets.

Each budget sets three things:

• Scope: an individual API key, or a usage pool.
• Token limit: the tokens allowed in one window.
• Period: daily, weekly, or monthly.

You can disable a budget rather than delete it. Each one reports the tokens counted so far in the current window and the time of the next reset. That counter decides admission rather than serving as a usage ledger, so read it as how close a scope is to being cut off, and use Access → Usage for historical accounting.

Scope a usage budget to the key rather than setting the key’s Daily token budget. Both cap tokens, and a usage budget also covers weekly and monthly windows and reports its own state. The older field stays for keys that already set it.

​

Rate limits

Rate limit policies cap throughput and protect against runaway spend. Each policy has a limit type, an optional target, and a scope:

• Limit type: tokens per minute, requests per minute, or concurrent requests.
• Target (optional): narrow the policy to a specific provider or model. Leave it unset to apply across all traffic.
• Scope: apply the limit per individual key (token) or shared across a usage pool.

Policies can be enabled or disabled without deleting them.

​

Fallback policies

A fallback policy keeps requests succeeding when a primary model is unavailable. For a primary model you define an ordered list of fallbacks; if a request against the primary fails, the Model Gateway retries the fallbacks in order. For example:

openai/gpt-5.6-luna → [openai/gpt-4o-mini, anthropic/claude-3-5-sonnet]

If openai/gpt-5.6-luna is unavailable, the Model Gateway tries openai/gpt-4o-mini, then anthropic/claude-3-5-sonnet, before giving up. Edit fallback rules under Model Gateway → Routing → Fallback policy.

​

Judge policy

A judge policy lets a scoring model decide when a request can be served by a cheaper one. The judge model scores each request, and the score selects a rung on a ladder of models. Configure the policy under Model Gateway → Routing → Judge policy, and override it for a single route from the Judge tab on that route’s detail page.

A ladder is the model the caller requested plus the cheaper targets you list. Boundaries divide the 0 to 1 score range into one band per rung. Leave the boundaries empty to split the range into equal bands. Otherwise supply exactly one fewer value than there are rungs, each strictly between 0 and 1 and each larger than the one before it. A target that repeats the requested model is ignored.

Two settings control rollout:

• Shadow mode: score requests and record what the policy would have done, without changing which model serves them.
• Shadow sample rate: the share of requests to score while shadow mode is on.

Scope the policy per key when you issue it with chalk router api-key create:

• --downgradable: allow this key’s requests to be downgraded.
• --model-floor: the cheapest model this key may be downgraded to.

Keys that set neither flag fall back to the policy’s own defaults for both.

A judge policy takes effect only once the judge is enabled for your deployment. After that, an edit reaches every replica within about a minute. A judge call that times out, or that arrives while the judge is at its concurrency limit, leaves the request on the model the caller asked for, so a slow or unavailable judge never fails a request.

​

Provider connections

A connection holds the credential for one provider along with the settings that decide which of its models the Model Gateway offers. Create and edit connections under Model Gateway → Routing → Connections.

You can create several connections for the same provider, each with its own credential, base URL, and model prefix. Scope an API key to a whole provider kind with --provider, or to one connection with --connection-id.

The Model Gateway supports these provider kinds:

Each connection also carries:

• Prefix: the segment the Model Gateway puts in front of a model ID, so /v1/models lists the connection’s models as <prefix>/<model>.
• Exposure: whether the connection’s models appear in /v1/models. Dynamic lists every model the provider reports. Unlisted keeps those models reachable but hidden, so a caller has to know the ID already. Explicit serves nothing from the provider’s catalog, so callers reach models only through a model route. Unspecified means no policy is stored, and the connection falls back to the default.
• State: whether the credential is configured, readable, and healthy.

Credentials apply at runtime, so changing one needs no redeploy.

​

Model routes

A model route maps the model ID a caller asks for to a model on a specific connection. Manage routes under Model Gateway → Routing → Routes.

Each route sets:

• Public model ID: the ID callers use.
• Connection: the provider connection that serves the request.
• Upstream model ID: the ID that connection’s provider expects.

Give the upstream ID in its bare form, without the connection’s prefix. /v1/models presents a connection’s models as <prefix>/<model>, and the Model Gateway strips that one leading segment before dispatching, so a route holding the prefixed form reaches the provider as an unknown model. An upstream ID may contain slashes of its own, and those segments stay.

Publish a route to make it callable. Each route reports a diagnostic alongside its status: ready, unconfigured, disabled, unreadable, or missing its connection.

​

Playground

Model Gateway → Playground is an in-dashboard tester for the Model Gateway. It authenticates with your Chalk session (no separate API key needed) and lets you exercise the Model Gateway across three tabs:

• Chat: send streaming chat completions with a configurable system prompt, temperature, and max tokens.
• Embeddings: generate embeddings and inspect their dimensions and token counts.
• Images: generate images with configurable size, quality, and count.

The model picker is populated from the Model Gateway’s /models endpoint, so it reflects the providers you have configured.

​

Observability

The Model Gateway records a trace for each request. Read traces for one key from the Traces tab on that key’s detail page, and choose where the Model Gateway exports them under Model Gateway → Telemetry.

​

Sampling policy

A sampling policy decides which traces the Model Gateway keeps, and whether it records prompt and completion content. Edit it under Model Gateway → Routing → Sampling policy.

The policy is an ordered list of rules. The first rule whose target matches a request supplies both the keep rate and the content setting for that request, and requests matching no rule fall to the policy defaults. A rule matches on one of:

• The x-chalk-request-tag header the caller sends.
• The calling key.
• The calling key’s usage pool.

Keep rate and content logging are independent. A trace the policy keeps may carry no content, and a request a rule marked for content logging may still be sampled away.

Outcome rules run once the response is known. Each reads a field from the response body and compares it to a value, and the first one that matches replaces the keep rate chosen at request time.

Some behavior applies regardless of which rule matches:

• A request that returns an error is always exported.
• Conversation rate floor sets a keep rate, between 0 and 1, that a session-keyed trace gets whatever the rules decided.
• A deployment-wide setting turns content logging off entirely, overriding every rule that would record it.

​

See also

• Authentication: service credentials and RBAC for the Chalk API.
• MCP Gateway: govern the external MCP servers your agents reach.
• LLM Toolchain: building AI features and agents on Chalk.