# Model Inference with vLLM
source: https://docs.chalk.ai/docs/compute/model-inference

## Deploy open-weight models like Gemma 4 with vLLM on Chalk Compute.

### Overview

Running your own inference endpoint gives you full control over model selection, GPU
allocation, and cost. Chalk Compute supports scaling groups with GPU access — deploy
a vLLM server once and let Chalk scale it across multiple replicas.

This tutorial deploys Gemma 4 using
vLLM with autoscaling and persistent model caching.

### Cache model weights with a volume

Large model files (multi-GB) should live in a Volume so they persist across scaling-group
replicas. This avoids re-downloading weights every time a replica starts.

```
from chalkcompute import Volume

vol = Volume(name="gemma4-weights")
```

On first boot, vLLM downloads the model into the Hugging Face cache directory. By
mounting the volume at that path, subsequent replicas start serving immediately.

### Define the scaling group

Create deploy_gemma4.py:

```
from chalkcompute import Image, ScalingGroup, Volume

vol = Volume(name="gemma4-weights")

image = (
    Image.base("vllm/vllm-openai:latest")
    .run_commands(
        "pip install huggingface_hub",
    )
)

sg = ScalingGroup(
    image=image,
    name="gemma4-vllm",
    env={
        "HF_TOKEN": "hf_...",                    # Hugging Face access token
        "HUGGING_FACE_HUB_TOKEN": "hf_...",
    },
    port=8000,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=1,
    max_replicas=4,
    entrypoint=[
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "google/gemma-3-27b-it",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--tensor-parallel-size", "1",
        "--max-model-len", "8192",
        "--dtype", "auto",
    ],
).deploy().wait_ready()

print(f"Inference endpoint: {sg.web_url}")
```

### Key parameters

| Parameter        | Purpose                                                   |
| ---------------- | --------------------------------------------------------- |
| `min_replicas=1` | Keep at least one replica warm — no cold starts.          |
| `max_replicas=4` | Scale up to 4 replicas under load.                        |
| `volumes=[...]`  | Mount the weight cache so new replicas skip the download. |

### Deploy it

```
python deploy_gemma4.py
# Inference endpoint: https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai
```

### Query the endpoint

vLLM exposes an OpenAI-compatible API. Point any OpenAI client at your scaling-group URL:

```
from openai import OpenAI

client = OpenAI(
    base_url="https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai/v1",
    api_key="not-needed",  # no auth required within Chalk
)

response = client.chat.completions.create(
    model="google/gemma-3-27b-it",
    messages=[
        {"role": "user", "content": "Explain feature stores in two sentences."},
    ],
)

print(response.choices[0].message.content)
```

Or with curl:

```
curl https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-27b-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

### Scaling behavior

Chalk monitors the scaling group and adds replicas as traffic increases. Each new
replica pulls model weights from the shared volume instead of downloading them again.
When traffic drops, Chalk scales back down to min_replicas.

```
# Dev configuration — scale to zero when idle
sg = ScalingGroup(
    image=image,
    name="gemma4-dev",
    port=8000,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=0,
    max_replicas=2,
    target_cpu_utilization_percentage=70,
    entrypoint=[...],
).deploy().wait_ready()
```