# Resources and GPUs
source: https://docs.chalk.ai/docs/workload-settings/resources-and-gpus

## Set CPU, memory, and GPU requests for sandboxes, scaling groups, and functions.

CPU, memory, and GPU settings apply to sandboxes,
scaling groups, and functions. All three
take the same cpu, memory, and gpu parameters.

### CPU and memory

The cpu and memory arguments (SDK) and the --cpu and --memory flags (scaling group CLI)
set the resource requests and limits. Both the Kubernetes request and limit are set to the same
value, guaranteeing your workload the resources it asks for, so it is not subject to
noisy-neighbor throttling.

CPU values follow Kubernetes conventions: 1 means one full core, 500m means
half a core. Memory values use standard suffixes: Mi (mebibytes) and Gi
(gibibytes).

CPU must be 250m, 500m, or a power-of-two number of cores (1, 2, 4, 8, 16, 32, ...).
Memory must be at least 2Gi per core: 512Mi for 250m, 1Gi for 500m, 2Gi for 1,
4Gi for 2, and so on. If you set CPU and omit memory, the workload gets
that minimum.

A sandbox:

```
from chalkcompute import Image, Sandbox

sandbox = Sandbox(
    image=Image.debian_slim(),
    cpu="4",
    memory="16Gi",
).run()
```

A function:

```
import chalkcompute

@chalkcompute.function(cpu="1", memory="2Gi")
def normalize_name(name: str) -> str:
    return " ".join(part.capitalize() for part in name.split())

normalize_name.wait_ready()
print(normalize_name("ada lovelace"))  # Ada Lovelace
```

A scaling group, from the CLI:

```
$ chalk scaling-group create \
    --image=my-registry/my-app:v1 \
    --name=my-service \
    --max-replicas=1 \
    --port=8080 \
    --cpu=2 \
    --memory=4Gi
```

If you omit CPU and memory, a sandbox gets 250m CPU and 512Mi memory. Each scaling group
replica gets the same defaults, set as both the request and the limit.

A sandbox's request must fit on a single host in a configured host pool.
A sandbox that asks for more CPU or memory than any host provides fails to schedule with
Unschedulable: no configured host pool can satisfy .... To run it, configure a host pool whose
hosts are large enough for the request.

### GPUs

The gpu argument (SDK) and the --gpu flag (scaling group CLI) take a type:count value, or
just type or count:

| Value         | GPU type      | GPUs |
| ------------- | ------------- | ---- |
| `nvidia-l4:2` | `nvidia-l4`   | 2    |
| `nvidia-l4`   | `nvidia-l4`   | 1    |
| `2`           | Any available | 2    |

To build an image with GPU libraries, see GPU images.

### Scaling groups and functions

A function runs on a scaling group, so the same rules apply to both.

```
import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="gpu-inference",
    port=8080,
    cpu="8",
    memory="32Gi",
    gpu="nvidia-l4:1",
    min_replicas=1,
    max_replicas=4,
).deploy()
```

```
$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-t4:1 \
    --cpu=4 \
    --memory=16Gi
```

The type portion selects the GPU node the replicas are scheduled on, and the
count sets the number of GPUs each replica gets (the nvidia.com/gpu request
and limit). Chalk adds the nvidia.com/gpu toleration automatically, so replicas
can schedule onto GPU-tainted nodes.

Write type as a GKE accelerator name on both GCP and AWS:

- GCP: the value is used as the cloud.google.com/gke-accelerator node
selector, so any accelerator in your cluster's node pools works.
- AWS: Chalk maps the value to Karpenter's karpenter.k8s.aws/instance-gpu-name
label. The types in the table below are supported.

| GPU type                | GPU                 |
| ----------------------- | ------------------- |
| `nvidia-tesla-t4`       | NVIDIA T4           |
| `nvidia-a10g`           | NVIDIA A10G         |
| `nvidia-l4`             | NVIDIA L4           |
| `nvidia-l40s`           | NVIDIA L40S         |
| `nvidia-tesla-v100`     | NVIDIA V100         |
| `nvidia-tesla-a100`     | NVIDIA A100         |
| `nvidia-a100-80gb`      | NVIDIA A100 80GB    |
| `nvidia-h100-80gb`      | NVIDIA H100 80GB    |
| `nvidia-h100-mega-80gb` | NVIDIA H100 Mega    |
| `nvidia-h200-141gb`     | NVIDIA H200         |
| `nvidia-rtx-pro-6000`   | NVIDIA RTX PRO 6000 |
| `nvidia-b200`           | NVIDIA B200         |

Which types you can schedule depends on the GPU node pools configured in your
cluster.

To run on any available GPU node without selecting a type, pass only the count:

```
$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=1
```

To request multiple GPUs, increase the count:

```
$ chalk scaling-group create \
    --image=my-registry/my-training-server:v1 \
    --name=multi-gpu \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-a100:4
```

To scale on GPU utilization, see Autoscaling.

### Sandboxes

A sandbox requests a GPU type at creation time. This example runs on the
k8s compute class:

```
from chalkcompute import ComputeClass, Image, Sandbox

sandbox = Sandbox(
    image="nvcr.io/nvidia/pytorch:24.01-py3",
    gpu="nvidia-l4",
    cpu="8",
    memory="64Gi",
    compute_class=ComputeClass.K8S,
).run()
```

The Chalk scheduler matches the request to a node with the appropriate GPU
hardware and configures the NVIDIA device plugin and driver mounts
automatically.

A k8s sandbox does not support exec, so pass its workload as entrypoint
when you create it. To run commands in a GPU sandbox with exec, use the host
compute class.

On AWS and GKE, you can also run a GPU sandbox on the default host compute class when
a host pool with the same GPU type is configured.
Omit compute_class or set it to ComputeClass.HOST. Each GPU host pool host
provides one GPU. GKE GPU host pools support the accelerator types listed in the
host pool guide. On other clouds,
use ComputeClass.K8S for GPU sandboxes.