# Autoscaling
source: https://docs.chalk.ai/docs/workload-settings/autoscaling

## Set replica ranges, scaling targets, scale to zero, and scheduled replica floors for scaling groups and functions.

Autoscaling applies to scaling groups and
functions. A function runs on a scaling group and passes these
arguments to it. Functions can also scale on their own request queue depth.

### Parameters

| Parameter                           | Applies to | Description                                                                                                                                                            |
| ----------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `min_replicas`                      | Both       | Lower bound for the autoscaler. Set to `0` to allow [scale to zero](#scale-to-zero). On a scaling group, defaults to `1`.                                              |
| `max_replicas`                      | Both       | Upper bound for the autoscaler. On a scaling group, defaults to `1`.                                                                                                   |
| `target_cpu_utilization_percentage` | Both       | Average CPU utilization the autoscaler targets, from 1 to 100. The autoscaler adds replicas when the average exceeds it and removes them when the average is below it. |
| `target_gpu_utilization_percentage` | Both       | Average GPU utilization the autoscaler targets, from 1 to 100. Requires `gpu` and `min_replicas` of at least `1`.                                                      |
| `target_queue_depth`                | Functions  | Target pending-request queue depth per replica. When set, the autoscaler scales on the function's own queue depth.                                                     |
| `scaling_interval_seconds`          | Both       | How often the autoscaler scrapes metrics and makes scaling decisions. Defaults to 60s on the server.                                                                   |
| `shutdown_delay_seconds`            | Both       | Grace period when removing replicas, to drain in-flight requests. See [Readiness and Shutdown](/docs/workload-settings/readiness-and-shutdown#shutdown-delay).         |
| `scale_from_zero_request_policy`    | Both       | How requests to a scaling group are handled while it scales up from zero. See [Scale to zero](#scale-to-zero).                                                         |
| `cron_scaling_trigger`              | Both       | Replica floors for scheduled time windows. See [Scheduled replica floors](#scheduled-replica-floors).                                                                  |

A scaling group:

```
import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="inference",
    port=8080,
    min_replicas=2,
    max_replicas=20,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=30,
).deploy()
```

From the CLI, --min-replicas and --max-replicas set the replica range, and
--target-cpu sets the CPU utilization target for autoscaling. --max-replicas
is required. To run a fixed number of replicas, set both flags to the same value.

A function:

```
import chalkcompute

@chalkcompute.function(
    min_replicas=1,
    max_replicas=8,
    scaling_interval_seconds=30,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=120,
    gpu="nvidia-l4",
)
def infer(text: str) -> str:
    ...
```

### CPU-driven vs queue-driven scaling

These two signals apply to functions:

- CPU (target_cpu_utilization_percentage): good for compute-bound
functions, such as model inference or data transformation, where CPU is the
bottleneck.
- Queue depth (target_queue_depth): good for I/O-bound or variable-latency
functions, or when you want to scale on demand rather than resource
saturation. Scales up when calls pile up regardless of CPU.

Both can be set together; the autoscaler takes the higher signal.

### Scale to zero

Set min_replicas=0 to let a scaling group scale down to zero replicas when it is not
receiving traffic. A function also accepts min_replicas=0. The group reports the ScaledToZero status while it has no
replicas.

A request that arrives while the group has no replicas starts a scale-up. By
default, Chalk responds to that request with HTTP 503, and your client has to
retry it. To hold the request until a replica is ready to serve it, set
scale_from_zero_request_policy to ScaleFromZeroRequestPolicy.HOLD:

```
import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="on-demand-api",
    port=8080,
    min_replicas=0,
    max_replicas=5,
    scale_from_zero_request_policy=chalkcompute.ScaleFromZeroRequestPolicy.HOLD,
).deploy()
```

From the CLI, pass --min-replicas=0 and --scale-from-zero-request-policy=hold.

### Scheduled replica floors

To keep replicas running during set hours, pass a CronScalingTrigger. Each
CronScalingWindow sets a replica floor from its start cron expression to its
end cron expression, evaluated in the trigger's IANA timezone. During a
window the group runs at least desired_replicas replicas. Outside its windows
the group scales between min_replicas and max_replicas.

This group runs at least three replicas from 8:00 to 18:00 New York time on
weekdays, and can scale to zero the rest of the time:

```
import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="business-hours-api",
    port=8080,
    min_replicas=0,
    max_replicas=10,
    cron_scaling_trigger=chalkcompute.CronScalingTrigger(
        timezone="America/New_York",
        windows=[
            chalkcompute.CronScalingWindow(
                start="0 8 * * 1-5",
                end="0 18 * * 1-5",
                desired_replicas=3,
            ),
        ],
    ),
).deploy()
```

A function takes the same cron_scaling_trigger argument. Scheduled replica floors are set from the SDK. The CLI has no flag for them.