Observability
Continuously evaluate metrics, platform health, or log queries, and open incidents when thresholds are met.
Monitors continuously evaluate a monitored entity (a metric formula, a log query, or a platform healthcheck) against a configured condition. When the condition is met, the monitor opens an incident and routes notifications to the configured channels. When the monitored value recovers, the incident is closed. Each monitor keeps a timeline of its events (incidents opened, triggering values, incidents closed) so you can see exactly when a problem started and when it resolved.
Monitors live in the dashboard under Alerting > Monitors. From there you can see every monitor in your environment with its status (OK, Failing, or Muted), type, frequency, description, alert channels, creator, and creation time. You can search by name, filter by type, creator, or alert channel, and create a new monitor with the New monitor button.
Every monitor, regardless of type, is configured with:
Metric chart and log monitors also have a threshold: a comparison (above, below, greater than or equal, less than or equal, equal, or not equal) against a target value. Environment health monitors have no threshold.
On the New monitor page, choose the type in the Monitor type list: Metric, Logs, Access Logs, Kube Events, or Environment health.
Metric chart monitors evaluate one or more metric series, optionally combined into a single value
with a formula. Each series is defined the same way as a series on a dashboard metrics chart: pick
a metric kind, window, grouping function, and filter conditions. Multiple series can be aggregated
together with a formula that references the series by letter (a, b, …). The resulting value
is compared against the threshold on every evaluation.
Use metric chart monitors to notify on things like request counts, latencies, error ratios, or feature drift metrics. Series and formulas are validated when the monitor is created or updated, so a misconfigured query fails at save time instead of at the first evaluation.
The editor previews the series over a time range you choose, and shades the alerting region red once you set a threshold.
Log monitors run a query against your environment’s logs, access logs, or Kube events, depending on the type you choose, on a schedule. You configure:
The number of matching entries in each lookback window is compared against the threshold. Use log monitors to open incidents on error messages, exception signatures, or any log pattern that should be rare (or that should always be present). Log monitors require Chalk-managed (ClickHouse-backed) log storage in your environment.
The editor previews the matching entries and a chart of how often they occur.
Use environment health monitors to get paged when a piece of your Chalk infrastructure degrades, without needing to construct a metric query for it. Not all systems are metric queryable, so health monitors provide additional functionality on top of metrics.
Environment health monitors watch one of Chalk’s built-in environment healthchecks, the same
checks that power the Settings > Connections page, covering components such as the online store,
offline store, query engines, streaming and branch servers, and supporting infrastructure. Pick
the healthcheck to monitor. The monitor opens an incident while that check reports unhealthy, and
closes it once the check reports healthy again.
When a monitor’s condition is met, an incident is opened for the monitor. Additional evaluations while the incident remains open are attached to the same incident rather than opening new ones. When the monitored value returns within the threshold, the incident is closed.
A monitor’s detail page has two tabs. Configuration holds the monitor’s settings, and Events shows the full timeline, newest first, including every failing evaluation, incident opening, and incident closure. Click Edit schedule at the top of the page to change how often the monitor runs, then click Save changes on the Configuration tab to apply it. Every incident also appears on the Alerting > Incidents page, and clicking an incident’s title in the timeline opens it.
Mute a monitor to stop it from sending alerts while it is failing. Click the bell icon next to the monitor’s status, then choose how long to mute it. A muted monitor shows Muted in the monitors list. Click the bell icon again to unmute it.