# Tracing Policy
source: https://docs.chalk.ai/docs/workload-settings/tracing

## Set how many traces Chalk keeps for a function, scaling group, or sandbox with the tracing setting.

The tracing setting applies to functions,
scaling groups, and sandboxes. It sets how
many traces Chalk keeps for the workload. Its default and behavior differ by component.

### Functions

Tracing helps you understand where time is spent when a function runs. Use it
to debug slow calls, follow nested function calls, and see which parts of your
application are doing work.

After a traced call runs, view it from the function's Scaling Group page under
Remote Call Traces.

### Enabling tracing

Set tracing=True on a function:

```
import chalkcompute

@chalkcompute.function(
    cpu="1",
    memory="2Gi",
    tracing=True,
)
def score_transaction(transaction_id: str) -> float:
    return 0.42
```

Nested calls appear in the same trace when tracing is enabled on each function:

```
import chalkcompute

@chalkcompute.function(tracing=True)
def fetch_profile(user_id: str) -> dict:
    return {"user_id": user_id, "segment": "active"}


@chalkcompute.function(tracing=True)
def score_user(user_id: str) -> float:
    with chalkcompute.span("score_user.fetch_profile"):
        profile = fetch_profile.remote(user_id)

    return 0.9 if profile["segment"] == "active" else 0.1


with chalkcompute.span("demo.score_user"):
    score = score_user.remote("user_123")
```

In the trace you can see the outer function, the call to fetch_profile, and
the inner function's execution together. Add manual spans like
score_user.fetch_profile to label important parts of your own code.

### Manual spans

Use chalkcompute.span() to label meaningful application work:

```
with chalkcompute.span("checkout.score_transaction"):
    score = score_transaction.remote("txn_123")
```

For a whole caller function, use @chalkcompute.traced:

```
@chalkcompute.traced("batch.score_transactions")
def run_batch(transaction_ids: list[str]) -> list[float]:
    return [score_transaction.remote(txn_id) for txn_id in transaction_ids]
```

Manual spans are useful around model calls, retrieval steps, preprocessing, and
other code you want to separate in the trace. To see a remote function's own
execution, enable tracing=True or a TracingPolicy on that function.

### Tracing policies

Use TracingPolicy when you only want to collect a percentage of calls:

```
@chalkcompute.function(
    tracing=chalkcompute.TracingPolicy(sample_rate=0.25),
)
def recommend(user_id: str) -> list[str]:
    ...
```

sample_rate must be between 0.0 and 1.0.

| Configuration                             | Behavior                                                                                                       |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `tracing=True`                            | Collect traces for every call.                                                                                 |
| `tracing=TracingPolicy(sample_rate=0.25)` | Collect traces for about 25% of calls.                                                                         |
| `tracing=TracingPolicy(sample_rate=0.0)`  | Trace this function only when its caller is already traced.                                                    |
| `tracing=False`                           | Turn tracing off for the function.                                                                             |
| Omitted                                   | Use the default behavior: only continue a caller's trace if tracing support is already installed in the image. |

Once a call is traced, nested function calls stay in the same trace.

tracing=True and TracingPolicy(...) automatically install tracing support
for functions deployed with @chalkcompute.function or RemoteFunction. If
you omit tracing, Chalk does not add those packages for you. The default can
still continue an existing trace if your image already includes tracing support,
for example because it installs chalkcompute or
chalk-remote-call-python[tracing].

### Troubleshooting

If you do not see remote function spans:

- Redeploy the function after changing tracing.
- Enable tracing on both outer and inner functions to see both functions'
execution in a nested trace.
- Wrap local tests in chalkcompute.span() if you want the call to be part of
an explicit trace.
- Make sure you are looking at the called function's Scaling Group page and a
time range that includes the call.

### Scaling groups

Chalk collects and stores the traces your app sends with
OpenTelemetry. Chalk sets the standard OpenTelemetry
environment variables in each replica, including OTEL_EXPORTER_OTLP_ENDPOINT,
which OpenTelemetry SDKs read to find where to send traces. An app produces
traces only if it is instrumented with an OpenTelemetry SDK.

By default, Chalk keeps about 1% of traces. Set tracing to change how many
traces it keeps:

```
import chalkcompute

img = chalkcompute.Image.debian_slim("3.12").entrypoint(
    ["python", "-m", "http.server", "8080"]
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="hello-api",
    port=8080,
    tracing=chalkcompute.TracingPolicy(sample_rate=0.25),
).deploy()
```

| Configuration                             | Behavior                  |
| ----------------------------------------- | ------------------------- |
| Omitted                                   | Keep about 1% of traces.  |
| `tracing=True`                            | Keep every trace.         |
| `tracing=TracingPolicy(sample_rate=0.25)` | Keep about 25% of traces. |
| `tracing=False`                           | Keep no traces.           |

sample_rate must be between 0.0 and 1.0. Unless tracing is False, a
request that arrives as part of a trace Chalk is already keeping stays in that
trace, whatever the sample rate.

tracing sets the OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG
environment variables. Set the sample rate with tracing or with those
variables in env; a group cannot use both.

### Sandboxes

tracing sets trace sampling for processes inside the sandbox. tracing=True traces every request,
a TracingPolicy sets the root sample rate, and tracing=False turns tracing off.