Infrastructure
Build a Chalk engine image and push it to a registry you own, without creating a deployment — and mirror it into a separate cloud partition such as AWS GovCloud.
chalk deploy build builds the engine image for your project and pushes it to container
registries that you name. Unlike chalk apply, it does not deploy
anything.
Concretely, it does not:
:latest tag on your environment’s engine repository.The result is an image in a registry you control, and nothing else. Your active deployment keeps serving exactly what it was serving before.
Use chalk deploy build when the image is an artifact you want to handle yourself rather
than a change you want live:
If you just want to deploy, use chalk apply.
chalk deploy build --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1Repeat --destination to push the same image to several registries in one build:
chalk deploy build \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25 \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidateThe build pushes to every destination in a single pass, so a second destination costs almost nothing.
On success the command prints the build’s identifier and the references it pushed to:
Build ID: chalk-build-argo-8f3c21d4
→ 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25
→ 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidate
The command runs from your project directory, exactly like chalk apply — it archives the
same source tree, honouring .chalkignore.
Every destination must be a fully-qualified reference with an explicit tag:
<registry-host>/<repository>:<tag>
| Rule | Accepted | Rejected |
|---|---|---|
| Registry host | 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 | myorg/engine:v1 (Docker Hub shorthand) |
| Explicit tag | .../engine:v1 | .../engine |
| Tag, not digest | .../engine:v1 | .../engine@sha256:… |
| Supported registry | Amazon ECR, Google Artifact Registry, GCR | Azure Container Registry, other hosts |
An untagged repository is rejected rather than defaulted to :latest, because silently
writing latest in a registry you own can overwrite an image your other systems already
pull.
Chalk pushes with the build’s own cloud identity. It never accepts, stores, or transmits registry credentials.
That means you grant Chalk’s build identity push access to the destination before running the command. What that identity is depends on your cloud:
This model implies two limits on AWS:
Cross-account works. A destination in a different AWS account is fine, as long as that repository’s policy grants your environment’s management role.
Cross-region does not. An ECR authorization token is only valid in the region that issued it, so the destination must be in the same region as your environment’s own registry. A destination elsewhere is rejected before the build starts:
image destination "123456789012.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1"
is in region us-gov-west-1 but this environment's registry is in us-east-1;
an ECR authorization token is only valid in the region that issued it
Destinations must also be on the same cloud as your environment — a GCP environment cannot push to ECR, and vice versa — because the build pod holds exactly one cloud identity.
Because there is no deployment record, this build does not appear in
chalk deployment list, the deployments page, or chalk deployment build-steps. Those
surfaces are all keyed on a deployment.
Today, the way to observe completion is to watch for the image to appear at its destination:
crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1or equivalently aws ecr describe-images. The GovCloud example below builds this wait into
the mirroring workflow.
Automating on top of this? Poll the destination registry rather than any Chalk deployment API. The build ID that `chalk deploy build` prints identifies the build workflow, not a deployment, so deployment-scoped endpoints will not find it.
The image is the same engine image chalk apply produces: Chalk’s platform software and the
resolved Python virtualenv for your requirements.
Your project source is not baked in. The engine downloads it at boot, from the source
bundle the deploy uploaded to your environment’s bucket. That is why one image serves every
deployment whose requirements are unchanged — and it is what makes
chalk apply --from-image possible.
Runtime configuration is *not* baked in. When Chalk deploys an image it supplies environment variables pointing at your online store, offline store, and graph artifacts. An image you run outside Chalk's control plane needs equivalent configuration supplied by whatever runs it.
AWS GovCloud is a separate AWS partition (aws-us-gov). This is a stronger boundary
than a region:
sts:AssumeRole does not cross partitions, so you cannot assume a GovCloud role from a
commercial account.So when your Chalk environment runs in the commercial partition, Chalk cannot push into
GovCloud directly, and --destination will reject a GovCloud reference on the region rule
above. The split is:
Step 2 is your code, running in your account, holding your GovCloud credentials. Chalk is not involved and never sees them. What follows is a working example you can adapt.
Because role assumption cannot cross partitions, the copy needs a credential issued in GovCloud. In practice that is an IAM user in your GovCloud account with permission to push to the destination repository, stored as a Kubernetes secret:
kubectl create secret generic govcloud-ecr \
--from-literal=access_key_id="$GOVCLOUD_ACCESS_KEY_ID" \
--from-literal=secret_access_key="$GOVCLOUD_SECRET_ACCESS_KEY"The commercial side needs no secret — give the workflow’s service account an IRSA role with read access to the source repository.
This WorkflowTemplate logs into both registries, copies the image, and verifies that the
digest survived the copy. All three containers share one pod so they can share a docker
config through an emptyDir.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: mirror-chalk-engine-to-govcloud
spec:
entrypoint: mirror
# IRSA role with ECR read on the commercial source repository.
serviceAccountName: chalk-image-mirror
arguments:
parameters:
- name: source-image
value: 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1
- name: destination-image
value: 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1
- name: source-region
value: us-east-1
- name: destination-region
value: us-gov-west-1
templates:
- name: mirror
volumes:
- name: docker-config
emptyDir: {}
containerSet:
volumeMounts:
- name: docker-config
mountPath: /docker
containers:
- name: login
image: amazon/aws-cli:2
env:
- name: GOVCLOUD_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: govcloud-ecr
key: access_key_id
- name: GOVCLOUD_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: govcloud-ecr
key: secret_access_key
command: [sh, -c]
args:
- |
set -eu
SRC='{{workflow.parameters.source-image}}'
DST='{{workflow.parameters.destination-image}}'
SRC_HOST="${SRC%%/*}"
DST_HOST="${DST%%/*}"
# Commercial partition: the pod's own IRSA role.
SRC_PW=$(aws ecr get-login-password \
--region '{{workflow.parameters.source-region}}')
# GovCloud is a different partition, so this must use the static
# credential. `env -u` drops the IRSA variables so the SDK cannot
# prefer web identity over the keys we are handing it.
DST_PW=$(env -u AWS_ROLE_ARN -u AWS_WEB_IDENTITY_TOKEN_FILE \
AWS_ACCESS_KEY_ID="$GOVCLOUD_ACCESS_KEY_ID" \
AWS_SECRET_ACCESS_KEY="$GOVCLOUD_SECRET_ACCESS_KEY" \
aws ecr get-login-password \
--region '{{workflow.parameters.destination-region}}')
SRC_AUTH=$(printf 'AWS:%s' "$SRC_PW" | base64 -w0)
DST_AUTH=$(printf 'AWS:%s' "$DST_PW" | base64 -w0)
umask 077
cat > /docker/config.json <<EOF
{"auths":{
"$SRC_HOST":{"auth":"$SRC_AUTH"},
"$DST_HOST":{"auth":"$DST_AUTH"}
}}
EOF
echo "wrote docker config for $SRC_HOST and $DST_HOST"
- name: copy
image: gcr.io/go-containerregistry/crane:debug
dependencies: [login]
env:
- name: DOCKER_CONFIG
value: /docker
command: [crane]
args:
- copy
- '{{workflow.parameters.source-image}}'
- '{{workflow.parameters.destination-image}}'
- name: verify
image: gcr.io/go-containerregistry/crane:debug
dependencies: [copy]
env:
- name: DOCKER_CONFIG
value: /docker
command: [sh, -c]
args:
- |
set -eu
SRC_DIGEST=$(crane digest '{{workflow.parameters.source-image}}')
DST_DIGEST=$(crane digest '{{workflow.parameters.destination-image}}')
echo "source: $SRC_DIGEST"
echo "destination: $DST_DIGEST"
if [ "$SRC_DIGEST" != "$DST_DIGEST" ]; then
echo "digest mismatch; the copy did not preserve the image" >&2
exit 1
fi
echo "digests match"Run it with:
argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \
-p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
-p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1crane copycrane copy transfers the manifest and its layers as-is, so the image keeps its digest
across partitions. That is what makes the verify step meaningful: a matching digest is
evidence that the bits authorized in GovCloud are byte-identical to the ones Chalk built,
which is usually what an authorization boundary review wants to see. A docker pull and
docker push through an intermediate host can rewrite the manifest and change the digest.
skopeo copy --all is an equivalent choice if it fits your tooling better.
# 1. Chalk builds and pushes to your commercial repository.
chalk deploy build \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1
# 2. Wait for the image to land (there is no deployment to poll).
until crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
>/dev/null 2>&1; do sleep 30; done
# 3. Mirror it into GovCloud.
argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \
-p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
-p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1Tag each build with something immutable — a git SHA or a release version — rather than reusing one tag. The digest check in step 3 only proves the copy was faithful; an immutable tag is what lets you say which build was authorized.
Once an image exists in a registry your environment’s cluster can pull from, deploy it with
chalk apply --from-image:
chalk apply --from-image \
210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256:<digest>This creates a normal deployment — a deployment record, a graph, a rollout — but builds
nothing. The reference you give becomes the deployment’s engine image verbatim, and the
Python source uploaded by that same chalk apply is what the engine downloads and serves.
Prefer a digest over a tag. A digest is the same identifier the mirroring step verified, so pinning one is how you say that the deployment runs exactly the image that was authorized.
Because no image is built, anything that describes how to build one is rejected rather than
ignored: a base image override or a build profile cannot be combined with --from-image.
An image build also produces artifacts beside the image — the source bundle it was built from, the virtualenv tarball, and the graph export an indexing job wrote. Three optional flags serve those instead of producing them again:
| Flag | What it replaces |
|---|---|
--with-source-bundle | The source bundle the engine downloads at boot, in place of the archive chalk apply just uploaded |
--with-export | The deployment’s export.pb, in place of the one an indexing job would have written |
--with-venv | Seeds your environment’s virtualenv cache, so a later indexing or branch build reuses it instead of resolving the requirements again |
Each takes an object that is already in your environment’s source bundle bucket — a bare
object key, or a full s3://, gs:// or az:// URI naming that same bucket. Chalk copies
within that one bucket; it does not read from anywhere else, and an object in another bucket
is rejected. Copy the artifacts across the partition boundary yourself, the same way you
copied the image.
--with-venv must keep the filename the cache uses, shallow_copy_<hash>.tar.zst, because
that name is the cache key. --with-export is optional in a different sense: without it the
engine builds its graph from the source bundle at boot, which is slower but correct.
chalk apply \
--from-image 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256:<digest> \
--with-source-bundle s3://my-chalk-bundles/authorized/build-v1.tar.gz \
--with-export s3://my-chalk-bundles/authorized/build-v1-export.pb \
--with-venv s3://my-chalk-bundles/authorized/shallow_copy_<hash>.tar.zstchalk apply Works — the deploying counterpart, and what the
image build stage does in detail.