Volumes

---

​

Overview

Volumes provide persistent file storage that can be shared across sandboxes. Under the hood, volumes are backed by a Rust-based FUSE driver that mounts directly into the container’s filesystem. This is file-level storage, not block storage — workloads interact with volumes through normal filesystem operations, and the driver handles tiered caching and replication to object storage transparently.

Volumes can be managed through the Python SDK, the CLI, or the web dashboard.

---

​

Creating and using volumes

​

CLI

# Create a volume
chalk volume create chalk://my-data

# Upload a file
chalk volume cp ./local-model.bin chalk://my-data/models/latest.bin

# Upload a directory
chalk volume cp ./models/ chalk://my-data/models --recursive

# List contents
chalk volume ls chalk://my-data

# Download a file
chalk volume cp chalk://my-data/models/latest.bin ./downloaded-model.bin

​

Chalk Volume URIs

A Chalk Volume URI starts with chalk://VOLUME, where VOLUME is the volume name. Everything after the next / is a path inside that volume.

chalk://my-data/models/latest.bin
        └─────┘
        volume name

Chalk Volume URI	Meaning
chalk://my-data	Volume named my-data
chalk://my-data/models/latest.bin	File inside my-data
chalk://my-data:experiment	Fork experiment of my-data
chalk://my-data@3	Version 3 of my-data

​

Python SDK

from chalkcompute import Volume

vol = Volume("my-data")

# Upload files
vol.put_file("models/latest.bin", model_bytes)

# Batch upload for efficiency
with vol.batch_upload() as batch:
    batch.put("data/train.csv", train_csv)
    batch.put("data/eval.csv", eval_csv)

# Read files
data = vol.read_file("models/latest.bin")

# List files
for f in vol.listdir("data/"):
    print(f"{f.path}  {f.size} bytes")

​

Mounting into sandboxes

Create a sandbox with a volumes parameter to mount that volume before starting it. The mounted volume is available as an ordinary directory inside the sandbox:

from chalkcompute import Image, Sandbox, Volume

vol = Volume("my-data")
vol.put_file("inputs/message.txt", b"hello from the volume\n")

sandbox = Sandbox(
    image=Image.debian_slim("3.12"),
    name="volume-reader",
    cpu="1",
    memory="2Gi",
    volumes=[("my-data", "/data")],
).run()

result = sandbox.exec(
    "python3",
    "-c",
    "print(open('/data/inputs/message.txt').read())",
)
print(result.stdout_text)

sandbox.terminate()

​

Syncing a mounted volume

To persist modified files, run sync from the mount to commit modifications:

sync /data

This creates a new immutable version of the mounted volume. This can be run after writing files from a shell session or long-running process.

The mount also contains a control file at .chalk-volume/commit. Run it to commit the files instead of sync. The .chalk-volume directory is not saved in the volume.

/data/.chalk-volume/commit

To reload the mount with the latest committed version, run the reload control file:

/data/.chalk-volume/reload

​

Web browser

Volumes are also browsable from the Chalk dashboard. You can navigate the file tree, preview file contents, and upload or download files directly from the browser.

---

​

Architecture

​

FUSE driver

The volume mount is implemented as a Rust-based FUSE (Filesystem in Userspace) driver that runs inside the sandbox. The driver exposes the volume as a standard POSIX directory, so workloads don’t need special libraries or APIs to read and write files — any tool that works with the filesystem works with a volume.

The FUSE driver handles:

• Tiered caching. Frequently accessed files are cached on the sandbox’s local disk. Cold data is fetched from object storage on demand.
• Transparent replication. All persisted data is durably stored in object storage (S3 or GCS, depending on your environment). Local cache is ephemeral and rebuilt automatically.

​

Copy-on-write semantics

Volumes use batch copy-on-write. When a sandbox writes to a mounted volume, the writes are buffered locally on the sandbox’s filesystem. These local writes are not visible to other sandboxes until a commit is made to persist writes via a sync call and sandboxes reload their current state.

# Writes inside the sandbox are local until committed
sandbox.exec("cp", "output.parquet", "/volumes/my-data/output.parquet")

This design avoids partial-write visibility — other consumers of the volume see a consistent snapshot, not a stream of in-progress file mutations. If the sandbox terminates before sync is called, uncommitted writes are discarded.

---

​

Versioning and fork semantics

Every commit creates an immutable snapshot of the volume’s state. Past versions are retained and can be retrieved by version ID:

# List available versions
versions = vol.versions()

# Open a previous version (read-only)
old = vol.at_version(versions[-2].id)
data = old.read_file("models/latest.bin")

Because prior versions are immutable and cheaply addressable, volumes support fork semantics. You can spawn multiple sandboxes from the same volume version, let them diverge independently, and sync their results into separate version lineages — without copying the underlying data.

This is particularly useful for coding agents that need to explore multiple solution paths in parallel:

from chalkcompute import Image, Sandbox

base_version = vol.latest_version()

# Fork two sandboxes from the same starting state
sandbox_a = Sandbox(
    image=Image.debian_slim(),
    volumes=[vol.at_version(base_version.id).fork("sandbox_a")],
).run()
sandbox_b = Sandbox(
    image=Image.debian_slim(),
    volumes=[vol.at_version(base_version.id).fork("sandbox_b")],
).run()

# Each sandbox writes independently — no interference
sandbox_a.exec("python", "approach_a.py")
sandbox_b.exec("python", "approach_b.py")
sandbox_a.terminate()
sandbox_b.terminate()

Each sandbox’s writes are isolated until explicitly synced, and the original version remains available regardless of what the forks produce.

---

​

Setting up a volume bucket

To use volumes on your own EKS or GKE cluster, link an object storage bucket and bind it to the cluster. Chalk volumes are backed by that bucket.

​

Bucket permissions

The following permissions are required on the bucket you’d like to use:

GCP: The service account used by background persistence (configured via “Service Account Name”) needs roles/storage.objectAdmin on the bucket, plus iam.serviceAccounts.signBlob on the service account.

AWS: Background persistence obtains AWS credentials through IRSA. The IAM role needs s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, and s3:GetBucketCors on the bucket.

​

CORS policy

The Chalk dashboard uploads and downloads volume files directly from the bucket using signed URLs, so the bucket needs a CORS rule that allows:

• Origin: the URL you open the dashboard at, for example https://chalk.ai
• Methods: GET and PUT
• Headers: Content-Type

Without it, dashboard uploads and downloads are proxied via the metadata server instead of going directly to the bucket, and the cluster’s volume health check reports a CORS error. To add the rule, follow Set up and view CORS configurations for GCS or Using cross-origin resource sharing for S3.

​

1. Create a cloud storage reference

Create the bucket first if you haven’t already. In the team navigation (click on the Chalk logo in the top left of the page), go to Cloud storage and click Link Storage. Fill in:

Field	Description
Storage Type	The bucket’s provider: Google Cloud Storage, AWS S3, or Azure Blob Storage.
URI	The bucket URI, optionally with a prefix (for example gs://my-bucket/volumes, s3://my-bucket/volumes, or https://myaccount.blob.core.windows.net/my-container).
Cloud Account	The cloud account whose credentials grant access to the bucket.

​

2. Bind the storage reference to the cluster

In the team navigation click on Clusters and select the cluster to open the Settings tab. Under Volume Bucket Binding, select the cloud storage reference created in step 1 and click Save Binding. This sets the bucket the cluster uses for volumes; the cluster manager picks it up automatically. To change or remove the binding later, select a different reference and save, or click Delete Binding.