# Rate and Concurrency Limits
source: https://docs.chalk.ai/docs/functions/rate-and-concurrency-limits

## Cap how often a Chalk Function is called and how many of its calls run at once.

Two settings limit how a Chalk Function is called: rate_limit caps requests over time, and concurrency caps how many calls run at once.

### Rate limiting

Cap the outbound request rate of a function using rate_limit. Pass an
integer for a simple "N per second" cap, or a RateLimitPolicy for full
control:

```
import chalkcompute
from chalkcompute import RateLimitPolicy

# Simple: up to 100 requests per second
@chalkcompute.function(rate_limit=100)
def call_provider(query: str) -> str:
    ...

# Full control: rate + time unit + shared-bucket key
@chalkcompute.function(
    rate_limit=RateLimitPolicy(rate=1000, per="minute", key="provider-api"),
)
def call_other_endpoint(query: str) -> str:
    ...
```

### RateLimitPolicy

| Parameter | Type  | Default       | Description                                    |
| --------- | ----- | ------------- | ---------------------------------------------- |
| `rate`    | `int` | *required*    | Maximum requests per `per` unit                |
| `per`     | `str` | `"second"`    | Time unit: `"second"`, `"minute"`, or `"hour"` |
| `key`     | `str` | function name | Shared-bucket identifier (see below)           |

### Sharing a bucket across functions

Two functions declaring the same key share a single rate-limit bucket. This
is useful when multiple functions call the same downstream service and the
service imposes a limit across all callers:

```
shared = RateLimitPolicy(rate=1000, per="second", key="provider-api")

@chalkcompute.function(rate_limit=shared)
def lookup(query: str) -> str: ...

@chalkcompute.function(rate_limit=shared)
def enrich(record: dict) -> dict: ...
# Calls to both functions count against the same 1000/s budget.
```

If key is omitted, each function gets its own bucket keyed by the function
name, so rate limits are per-function by default.

### Concurrency limits

Cap the in-flight count of a function using concurrency. This is distinct
from rate_limit: rate limits gate requests-over-time, concurrency gates how
many calls are executing simultaneously:

```
import chalkcompute
from chalkcompute import ConcurrencyPolicy

# Simple: at most 10 calls in flight at once
@chalkcompute.function(concurrency=10)
def gpu_infer(text: str) -> list[float]:
    ...

# Shared gate across multiple functions hitting the same resource
gpu_pool = ConcurrencyPolicy(max_concurrent=4, key="gpu-pool")

@chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4")
def embed(text: str) -> list[float]: ...

@chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4")
def rerank(query: str, docs: list[str]) -> list[int]: ...
# Only 4 calls across both functions run simultaneously.
```

### ConcurrencyPolicy

| Parameter        | Type  | Default       | Description                           |
| ---------------- | ----- | ------------- | ------------------------------------- |
| `max_concurrent` | `int` | *required*    | Maximum in-flight handler invocations |
| `key`            | `str` | function name | Shared-bucket identifier              |

When concurrency is not specified, the function queue consumer applies a
default of max_concurrent=1 per function, so calls to a single function are
serialized. Set concurrency=N explicitly to allow parallel execution.

### Rate limit vs concurrency

Both can be set independently and they interact:

| Policy        | Controls                 | Example use case                     |
| ------------- | ------------------------ | ------------------------------------ |
| `rate_limit`  | Requests per unit time   | "1000 calls/sec across all replicas" |
| `concurrency` | Parallel in-flight count | "4 GPU slots, don't exceed memory"   |