Monitors

​

Overview

Monitors continuously evaluate a monitored entity — a metric formula, a platform healthcheck, or a log query — against a configured threshold. When the threshold is met, the monitor opens an incident, and routes notifications to the configured channels. When the monitored value recovers, the incident is closed. Each monitor keeps a timeline of its events (incidents opened, triggering values, incidents closed) so you can see exactly when a problem started and when it resolved.

Monitors live in the dashboard under Alerting > Monitors. From there you can see every monitor in your environment, whether it currently has an active incident, and create new monitors with the New monitor button.

Every monitor, regardless of type, is configured with:

• Name and description: user-provided metadata about the purpose of this monitor, shown in the monitors table.
• Monitored entity: the metric chart, a log query, or environment being evaluated healthcheck, depending on the monitor type.
• Threshold: a comparison (above, below, greater than or equal, less than or equal, equal, or not equal) against a target value.
• Alert channels: where notifications are sent when the threshold is crossed. Monitors can route to a Slack channel, a PagerDuty service, or an incident.io service.

​

Monitor Types

​

Metric Chart Monitors

Metric chart monitors evaluate one or more metric series, optionally combined into a single value with a formula. Each series is defined the same way as a series on a dashboard metrics chart — pick a metric kind, window, grouping function, and filter conditions — and multiple series can be aggregated together with a formula that references the series by letter (a, b, …). The resulting value is compared against the threshold on every evaluation.

Use metric chart monitors to notify on things like request counts, latencies, error ratios, or feature drift metrics. Series and formulas are validated when the monitor is created or updated, so a misconfigured query fails at save time instead of at the first evaluation.

​

Log Monitors

Log monitors run a query against your environment’s logs on a schedule. You configure:

• Query string: a literal or wildcard expression match against log contents.
• Window period: the period over which matching log lines are aggregated.

The number of matching log lines in each window is compared against the threshold. Use log monitors to open incidents on error messages, exception signatures, or any log pattern that should be rare (or that should always be present). Log monitors require Chalk-managed (ClickHouse-backed) log storage in your environment.

​

Environment Health Monitors

Use environment health monitors to get paged when a piece of your Chalk infrastructure degrades, without needing to construct a metric query for it. Not all systems are metric queryable — health monitors provide additional functionality on top of metrics.

Environment health monitors watch one of Chalk’s built-in environment healthchecks — the same checks that power the Settings > Connections page, covering components such as the online store, offline store, query engines, streaming and branch servers, and supporting infrastructure. Pick the healthcheck to monitor, and the monitor opens an incident when that check reports unhealthy.

​

Incidents

When a monitor’s threshold is met, an incident is opened for the monitor. Additional evaluations while the incident remains open are attached to the same incident rather than opening new ones. When the monitored value returns within the threshold, the incident is closed. The monitor’s detail page shows the full event timeline alongside the monitor’s configuration, including every failing evaluation, incident opening, and incident closure. opening, and incident closure — alongside the monitor’s configuration.