Autoscaling

---

Autoscaling applies to scaling groups and functions. A function runs on a scaling group and passes these arguments to it. Functions can also scale on their own request queue depth.

---

​

Parameters

Parameter	Applies to	Description
min_replicas	Both	Lower bound for the autoscaler. Set to 0 to allow scale to zero. On a scaling group, defaults to 1.
max_replicas	Both	Upper bound for the autoscaler. On a scaling group, defaults to 1.
target_cpu_utilization_percentage	Both	Average CPU utilization the autoscaler targets, from 1 to 100. The autoscaler adds replicas when the average exceeds it and removes them when the average is below it.
target_gpu_utilization_percentage	Both	Average GPU utilization the autoscaler targets, from 1 to 100. Requires gpu and min_replicas of at least 1.
target_queue_depth	Functions	Target pending-request queue depth per replica. When set, the autoscaler scales on the function’s own queue depth.
scaling_interval_seconds	Both	How often the autoscaler scrapes metrics and makes scaling decisions. Defaults to 60s on the server.
shutdown_delay_seconds	Both	Grace period when removing replicas, to drain in-flight requests. See Readiness and Shutdown.
scale_from_zero_request_policy	Both	How requests to a scaling group are handled while it scales up from zero. See Scale to zero.
cron_scaling_trigger	Both	Replica floors for scheduled time windows. See Scheduled replica floors.

A scaling group:

import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="inference",
    port=8080,
    min_replicas=2,
    max_replicas=20,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=30,
).deploy()

From the CLI, --min-replicas and --max-replicas set the replica range, and --target-cpu sets the CPU utilization target for autoscaling. --max-replicas is required. To run a fixed number of replicas, set both flags to the same value.

A function:

import chalkcompute

@chalkcompute.function(
    min_replicas=1,
    max_replicas=8,
    scaling_interval_seconds=30,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=120,
    gpu="nvidia-l4",
)
def infer(text: str) -> str:
    ...

---

​

CPU-driven vs queue-driven scaling

These two signals apply to functions:

• CPU (target_cpu_utilization_percentage): good for compute-bound functions, such as model inference or data transformation, where CPU is the bottleneck.
• Queue depth (target_queue_depth): good for I/O-bound or variable-latency functions, or when you want to scale on demand rather than resource saturation. Scales up when calls pile up regardless of CPU.

Both can be set together; the autoscaler takes the higher signal.

---

​

Scale to zero

Set min_replicas=0 to let a scaling group scale down to zero replicas when it is not receiving traffic. A function also accepts min_replicas=0. The group reports the ScaledToZero status while it has no replicas.

A request that arrives while the group has no replicas starts a scale-up. By default, Chalk responds to that request with HTTP 503, and your client has to retry it. To hold the request until a replica is ready to serve it, set scale_from_zero_request_policy to ScaleFromZeroRequestPolicy.HOLD:

import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="on-demand-api",
    port=8080,
    min_replicas=0,
    max_replicas=5,
    scale_from_zero_request_policy=chalkcompute.ScaleFromZeroRequestPolicy.HOLD,
).deploy()

From the CLI, pass --min-replicas=0 and --scale-from-zero-request-policy=hold.

​

Scheduled replica floors

To keep replicas running during set hours, pass a CronScalingTrigger. Each CronScalingWindow sets a replica floor from its start cron expression to its end cron expression, evaluated in the trigger’s IANA timezone. During a window the group runs at least desired_replicas replicas. Outside its windows the group scales between min_replicas and max_replicas.

This group runs at least three replicas from 8:00 to 18:00 New York time on weekdays, and can scale to zero the rest of the time:

import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="business-hours-api",
    port=8080,
    min_replicas=0,
    max_replicas=10,
    cron_scaling_trigger=chalkcompute.CronScalingTrigger(
        timezone="America/New_York",
        windows=[
            chalkcompute.CronScalingWindow(
                start="0 8 * * 1-5",
                end="0 18 * * 1-5",
                desired_replicas=3,
            ),
        ],
    ),
).deploy()

A function takes the same cron_scaling_trigger argument. Scheduled replica floors are set from the SDK. The CLI has no flag for them.