Chalk Catalog Components

The Chalk Catalog is an umbrella term encompassing all things Chalk. This page describes how various concepts within Chalk are translated into the Catalog and made available to Chalk SQL.

Like traditional relational databases, the Catalog is organized under a three-tiered namespace system of catalogs, schemas, and tables.

​

The chalk catalog

Most concepts in Chalk will be exposed as tables under the chalk database catalog.

​

chalk.datasets schema

The chalk.datasets schema contains a view for each named dataset created within the last 60 days, and the view’s name is the dataset’s name. Note that if two offline queries reference the same named dataset, only the latest dataset revision is available in this schema.

To reference dataset revisions that are not the most recent or were created without a name, consider using the get_dataset_revision() table-valued function.

​

chalk.historical_values schema

The chalk.historical_values schema contains a view for each feature, as well as a view for each feature namespace within the deployment.

• For each feature, a view with the same name as the feature’s fully-qualified name is available in the Catalog (i.e. chalk.historical_values.user.name). This view contains information about all observations of the feature, with the following schema:

  • observation_id, string, non-nullable: The unique identifier for the observation of the feature value.
  • pkey, (type of feature namespace’s pkey), non-nullable: The value of the primary feature associated with this observation
  • value, (type of feature namespace’s value): The observed value of the feature.
  • value_json, string, DEPRECATED: The observed value of the feature, as JSON, for backwards compatibility.
  • observed_at, timestamp, non-nullable: The time at which the feature value was observed.
  • inserted_at, timestamp, non-nullable: The time at which the feature value was inserted into the offline store.
  • resolver_fqn, string, nullable: The resolver that computed this feature.
  • deployment_id, string, nullable: The deployment id where the observation was made

• For each feature namespace, a view with the same name as the namespace is available in the Catalog (i.e. chalk.historical_values). This view contains the freshest values of each feature for each primary key at the current point in time, and has the following schema:

  • __pkey__: (type of feature namespace’s pkey), non-nullable: The value of the primary feature associated with this row
  • <feature-name>, (type of feature), for each feature in the namespace: The freshest observed value of the feature for the primary key.
  • <feature-name>.__observed_at__, timestamp, for each feature in the namespace: The time at which the freshest value was observed.

​

chalk.query_log schema

The chalk.query_log schema consists of exactly one table, data, which is the query log table. This table provides a list of query metadata for each operation id, including which tables in the offline store correspond to values queried in that operation id. The data table has the following schema:

• operation_id, string, non-nullable: The unique identifier for the operation.
• environment_id, string, non-nullable: The environment id where the operation was run.
• deployment_id, string: The deployment id where the operation was run.
• operation_kind, string, The kind of operation that was run (inference, streaming, etc.).
• query_timestamp, timestamp: The value of now() that was used by the query.
• execution_started_at, timestamp: The time at which the operation started executing. Null if there were errors during planning.
• execution_finished_at, timestamp: The time at which the operation finished executing. Null if there were errors during execution.
• query_status, string: The status of the query (success, error, running, etc.).
• query_name, string: The query name for named queries, null otherwise.
• query_name_version, string: The version of the query for named queries, null otherwise.
• branch_name: string: The branch the query was run on, null if the query was not run on a branch.
• correlation_id, string: The correlation id for the query, null if the query was not run with a correlation id.
• value_tables: array of strings: The list of tables in the offline store that correspond to result values from this query.

​

chalk.resolvers schema

The chalk.resolvers schema contains a view for each dataframe-returning resolver that takes in no inputs. When querying this view, a resolver run will be triggered and have the resolver’s outputs as the view’s contents.

​

chalk.online_store schema

The chalk.online_store schema contains a fixed view keys, which contains different information depending on which online store is configured for your environment.

• For Redis online stores, information about the keys’ type (Scalar, HasMany, TimeSeries, etc.), memory usage, TTL, and feature fqn are available in this view.
• For DynamoDB online stores, information about the keys’ namespace and value are available in this view.

Additionally, for each feature namespace, a view with the same name (i.e. chalk.online_store.user) is available, consisting of the following schema:

• __id__
• <feature_namespace>.<feature> for each feature
• <feature_namespace>.<feature>_ts for each feature

​

The offline_store catalog

The offline store is available as a catalog. Similar to data sources as catalogs, this catalog works as a direct interface to querying the underlying database that powers it.

​

Data Sources as Catalogs

Each data source that you register with Chalk will be automatically registered in the Chalk Catalog as a database catalog. The catalog shares the same name as the data source and provides a more direct interface into querying that data source.

As an example, if you define a Postgres source and a BigQuery source in your code:

from chalk.sql import PostgresSource
from chalk.sql import BigquerySource
pg = PostgresSource(name='pg_source')
bq = BigquerySource(name='bq_source')

You can write a federated query in Chalk SQL to join a transactions table in pg with a users table in bq:

SELECT
  bq_users.user_id AS user_id,
  bq_users.full_name AS full_name,
  pg_transactions.transaction_id AS transaction_id,
  pg_transactions.amount AS transaction_amount
FROM bq_source.my_dataset.users AS bq_users
OUTER JOIN pg.public.transactions AS pg_transactions
ON user_id = pg_transactions.user_id
/* pg_source and bq_source are the name of the datasources
 and therefore also the name of their respective catalogs */

The following data source types will be registered to the Chalk Catalog, and will be able to be queried in Chalk SQL:

• PostgreSQL
• Spanner
• MySQL
• Clickhouse
• BigQuery
• Redshift
• Snowflake
• Databricks

​

Attaching a catalog with ATTACH

A query can also bring its own catalog, without registering a data source, using ATTACH:

ATTACH 'host=db.internal port=5432 dbname=analytics user=reader' AS analytics (TYPE POSTGRES);

SELECT * FROM analytics.public.transactions LIMIT 10;

The attached catalog lives for the session, is always read-only, and opens no connection until one of its tables is queried. ATTACH IF NOT EXISTS and ATTACH OR REPLACE behave as they do elsewhere in SQL.

TYPE chooses the backend. It may be omitted when the string before AS is a URI, in which case the scheme picks the backend:

ATTACH 'mysql://reader:hunter2@db.internal:3306/analytics' AS analytics;

The supported types are postgres, mysql, clickhouse, mssql, redshift, snowflake, bigquery, spanner, dynamodb, iceberg and turbopuffer. Naming a type that does not exist lists the ones that do.

​

Connection options

The SQL backends take their connection target from the URI, from options, or from a mixture of the two — an option overrides the same value from the URI:

ATTACH '' AS analytics (
  TYPE POSTGRES,
  HOST 'db.internal',
  PORT 5432,
  DATABASE 'analytics',
  USER 'reader',
  PASSWORD 'hunter2'
);

Backends that need more than a connection target read it from options too — S3_BUCKET and IAM_ROLE for Redshift, WAREHOUSE / ROLE for Snowflake, PROJECT_ID / DATASET_ID for BigQuery, and so on. An option the backend does not recognize fails the query with the offending name rather than being ignored.

​

Secrets

Rather than writing credentials into the statement, name a secret that your environment has made available to the session:

ATTACH '' AS analytics (TYPE POSTGRES, SECRET analytics_reader, DATABASE 'analytics_replica');

The secret’s entries are used as options, and any option written in the statement — DATABASE above — wins over the same key from the secret.

​

Iceberg

An Iceberg catalog attaches by REST endpoint, by Glue, or as a bare warehouse:

ATTACH 's3://my-bucket/warehouse' AS iceberg_catalog (
  TYPE ICEBERG,
  ENDPOINT 'https://rest.example.com',
  SECRET iceberg_secret
);

SELECT * FROM iceberg_catalog.my_namespace.my_table LIMIT 10;

The table’s Iceberg namespace occupies the schema position. Options other than ENDPOINT, REGION and ROLE_ARN are passed to the Iceberg catalog client as its own property keys (catalog, warehouse, token, credential, prefix, header.<name>, …); catalog defaults to rest when an endpoint is given.

​

Turbopuffer

Turbopuffer exposes no namespace or schema listing, so an attached Turbopuffer catalog declares the namespaces it exposes and their column types:

ATTACH 'turbopuffer://<api_key>@gcp-us-central1' AS tpuf (
  TYPE TURBOPUFFER,
  NAMESPACES '{
    "docs": {
      "id": "uuid",
      "vector": "[1536]f32",
      "title": "string",
      "body": {"type": "string", "full_text_search": true}
    }
  }'
);

SELECT id, title FROM tpuf.docs WHERE tpuf_bm25(body, 'checking account') LIMIT 10;

id and vector are reserved: they give the namespace’s key and vector types. Every other entry is an attribute that can be selected and filtered, and full_text_search marks the ones usable with tpuf_bm25.