Vector Databases

Vector databases enable efficient storage and retrieval of high-dimensional vectors for nearest neighbor search operations. Chalk integrates with vector databases to provide scalable vector search capabilities for your feature pipelines.

---

​

Configuration

Vector databases can be configured through your Chalk dashboard at Integrations > Vector Database. Once configured, your vector database becomes available for use with vector search operations.

---

​

Vector Search

After configuring a vector database integration, you can leverage Chalk’s nearest neighbor search capabilities to find similar vectors across your feature classes. When you define a has_many relationship using the is_near method, the query is delegated to the underlying vector store for efficient processing.

from chalk.features import DataFrame, embed, features, has_many, Vector

@features
class SearchQuery:
    query: str
    embedding: Vector = embed(...)
    topic: str
    nearest_documents: DataFrame[FAQDocument] = has_many(
        lambda: SearchQuery.embedding.is_near(
            FAQDocument.question_embedding
        ) & (SearchQuery.topic == FAQDocument.topic) # Additional filters can optionally be added to the join
    )

@features
class FAQDocument:
    question: str
    question_embedding: Vector = embed(...)
    topic: str
    link: str

In this example, the has_many response from the join will be delegated to the configured vector database which performs the efficient nearest neighbor search. The FAQDocument.question_embedding is stored in the vector store: the following section discusses how the embeddings are loaded into the vector store.

---

​

Persisting Vector Features

Vector features — those used in nearest neighbor joins — are automatically persisted to your vector store. This ensures that your vectors are readily available for efficient querying. Typically, embeddings are either generated from streams or offline/scheduled runs.

​

Using Streams

A stream resolver can load embeddings computed outside Chalk, such as by a service that publishes each document and its embedding to Kafka:

from chalk.features import DataFrame, Vector, _, features, has_many
from chalk.features.resolver import make_stream_resolver
from chalk.streams import KafkaSource
from pydantic import BaseModel


@features
class FAQDocument:
    id: str
    question: str
    question_embedding: Vector[384]
    topic: str


@features
class SearchQuery:
    id: str
    embedding: Vector[384]
    topic: str
    nearest_documents: DataFrame[FAQDocument] = has_many(
        lambda: SearchQuery.embedding.is_near(FAQDocument.question_embedding)
        & (SearchQuery.topic == FAQDocument.topic)
    )


class FAQDocumentMessage(BaseModel):
    id: str
    question: str
    question_embedding: list[float]
    topic: str


faq_documents_stream = KafkaSource(name="faq_documents")

make_stream_resolver(
    name="ingest_faq_documents",
    source=faq_documents_stream,
    message_type=FAQDocumentMessage,
    output_features={
        FAQDocument.id: _.id,
        FAQDocument.question: _.question,
        FAQDocument.question_embedding: _.question_embedding,
        FAQDocument.topic: _.topic,
    },
)

The stream resolver outputs FAQDocument.question_embedding, the stored side of the SearchQuery.nearest_documents join, so Chalk writes each message’s embedding to the vector store. The searching side of the join, SearchQuery.embedding, is not written to the vector store.

​

Using Offline or Scheduled Jobs

Alternatively, you can ingest vector features through offline or scheduled jobs. These jobs compute and load vector features to the vector database in batch:

from chalk import offline
from chalk import ScheduledQuery

@offline
def ingest_embeddings(
    document_text: Document.text
) -> Document.embedding:
    return compute_embedding(document_text)


ScheduledQuery(
    name="generate_document_embeddings",
    schedule="0 0 * * *",
    output=[Document.embedding],
    # store online will cause the vector to be persisted
    store_online=True,
    store_offline=True,
    incremental_resolvers="documents",
)

The scheduled query will load newly calculated embeddings into the vector database, making them queryable through nearest neighbor operations.

---

​

Query Flow

When you execute a query that includes a nearest neighbor relationship:

1. Chalk identifies the vector search operation in your query
2. The query is delegated to your configured vector database
3. The vector store performs efficient approximate nearest neighbor search
4. Results are returned as a Chalk DataFrame, ready to use in your resolvers

This delegation ensures that vector search operations leverage the specialized indexing and query capabilities of your vector database, providing optimal performance for high-dimensional similarity search.