Model Inference with vLLM

---

​

Overview

Running your own inference endpoint gives you full control over model selection, GPU allocation, and cost. Chalk Compute supports scaling groups with GPU access, so you can deploy a vLLM server once and let Chalk scale it across multiple replicas.

This tutorial deploys Gemma 4 using vLLM with autoscaling and persistent model caching.

---

​

Cache model weights with a volume

Large model files (multi-GB) should live in a Volume so they persist across scaling-group replicas. This avoids re-downloading weights every time a replica starts.

import chalkcompute

vol = chalkcompute.Volume(name="gemma4-weights")

On first boot, vLLM downloads the model into the Hugging Face cache directory. By mounting the volume at that path, subsequent replicas start serving immediately.

---

​

Define the scaling group

Create deploy_gemma4.py:

import chalkcompute

vol = chalkcompute.Volume(name="gemma4-weights")

image = (
    chalkcompute.Image.base("vllm/vllm-openai:latest")
    .run_commands(
        "pip install huggingface_hub",
    )
)

sg = chalkcompute.ScalingGroup(
    image=image,
    name="gemma4-vllm",
    env={
        "HF_TOKEN": "hf_...",                    # Hugging Face access token
        "HUGGING_FACE_HUB_TOKEN": "hf_...",
    },
    port=8000,
    authenticated=True,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=1,
    max_replicas=4,
    entrypoint=[
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "google/gemma-3-27b-it",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--tensor-parallel-size", "1",
        "--max-model-len", "8192",
        "--dtype", "auto",
    ],
).deploy().wait_ready()

print(f"Inference endpoint: {sg.web_url}")

​

Key parameters

Parameter	Purpose
min_replicas=1	Keep at least one replica warm, so requests do not wait for a cold start.
max_replicas=4	Scale up to 4 replicas under load.
volumes=[...]	Mount the weight cache so new replicas skip the download.
authenticated=True	Require a Chalk access token on every request. See Authentication.

---

​

Deploy it

python deploy_gemma4.py
# Inference endpoint: https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai

---

​

Query the endpoint

vLLM exposes an OpenAI-compatible API. Because the group sets authenticated=True, every request needs a Chalk access token. Put the group’s URL and a token in your shell:

export WEB_URL=<the inference endpoint printed by deploy_gemma4.py>
export CHALK_TOKEN=$(chalk token get)

The OpenAI client sends its api_key as a bearer token, so pass the Chalk token there:

import os

from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['WEB_URL']}/v1",
    api_key=os.environ["CHALK_TOKEN"],
)

response = client.chat.completions.create(
    model="google/gemma-3-27b-it",
    messages=[
        {"role": "user", "content": "Explain feature stores in two sentences."},
    ],
)

print(response.choices[0].message.content)

Or with curl:

curl "$WEB_URL/v1/chat/completions" \
  -H "Authorization: Bearer $CHALK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-27b-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

---

​

Scaling behavior

Chalk monitors the scaling group and adds replicas as traffic increases. Each new replica pulls model weights from the shared volume instead of downloading them again. When traffic drops, Chalk scales back down to min_replicas.

# Dev configuration: scale to zero when idle
sg = chalkcompute.ScalingGroup(
    image=image,
    name="gemma4-dev",
    port=8000,
    authenticated=True,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=0,
    max_replicas=2,
    target_cpu_utilization_percentage=70,
    entrypoint=[...],
).deploy().wait_ready()