Rate and Concurrency Limits

---

Two settings limit how a Chalk Function is called: rate_limit caps requests over time, and concurrency caps how many calls run at once.

---

​

Rate limiting

Cap the outbound request rate of a function using rate_limit. Pass an integer for a simple “N per second” cap, or a RateLimitPolicy for full control:

import chalkcompute
from chalkcompute import RateLimitPolicy

# Simple: up to 100 requests per second
@chalkcompute.function(rate_limit=100)
def call_provider(query: str) -> str:
    ...

# Full control: rate + time unit + shared-bucket key
@chalkcompute.function(
    rate_limit=RateLimitPolicy(rate=1000, per="minute", key="provider-api"),
)
def call_other_endpoint(query: str) -> str:
    ...

​

RateLimitPolicy

Parameter	Type	Default	Description
rate	int	required	Maximum requests per per unit
per	str	"second"	Time unit: "second", "minute", or "hour"
key	str	function name	Shared-bucket identifier (see below)

​

Sharing a bucket across functions

Two functions declaring the same key share a single rate-limit bucket. This is useful when multiple functions call the same downstream service and the service imposes a limit across all callers:

shared = RateLimitPolicy(rate=1000, per="second", key="provider-api")

@chalkcompute.function(rate_limit=shared)
def lookup(query: str) -> str: ...

@chalkcompute.function(rate_limit=shared)
def enrich(record: dict) -> dict: ...
# Calls to both functions count against the same 1000/s budget.

If key is omitted, each function gets its own bucket keyed by the function name, so rate limits are per-function by default.

---

​

Concurrency limits

Cap the in-flight count of a function using concurrency. This is distinct from rate_limit: rate limits gate requests-over-time, concurrency gates how many calls are executing simultaneously:

import chalkcompute
from chalkcompute import ConcurrencyPolicy

# Simple: at most 10 calls in flight at once
@chalkcompute.function(concurrency=10)
def gpu_infer(text: str) -> list[float]:
    ...

# Shared gate across multiple functions hitting the same resource
gpu_pool = ConcurrencyPolicy(max_concurrent=4, key="gpu-pool")

@chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4")
def embed(text: str) -> list[float]: ...

@chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4")
def rerank(query: str, docs: list[str]) -> list[int]: ...
# Only 4 calls across both functions run simultaneously.

​

ConcurrencyPolicy

Parameter	Type	Default	Description
max_concurrent	int	required	Maximum in-flight handler invocations
key	str	function name	Shared-bucket identifier

​

Rate limit vs concurrency

Both can be set independently and they interact:

Policy	Controls	Example use case
rate_limit	Requests per unit time	“1000 calls/sec across all replicas”
concurrency	Parallel in-flight count	“4 GPU slots, don’t exceed memory”