Fontana CLI

The fontana CLI is the single operator entry point for a Fontana host. You declare desired state in fontana.yaml and reconcile the box with fontana apply. The CLI manages the platform k3d cluster (shared services), one k3d cluster per workspace tenant (the Fontana backend stack), and the host edge that serves the Fontana web app and welcome page as static assets. It ships inside the verified deploy bundle that your host pulls from GHCR (or from your private registry mirror).

Supported platforms

Run fontana on the host that runs Docker and your Fontana clusters. Edge TLS, DNS, and cluster lifecycle reconcile on that same machine.

Platform	Typical use
Linux (x86_64 or arm64)	Production self-hosted servers, private cloud VMs, and customer VPC hosts
macOS (Apple Silicon or Intel, with Homebrew)	Local and lab *.localhost installs on a laptop or workstation

On Linux production hosts you usually install with administrator privileges; on macOS you install as your user (see Install below). Windows is not supported as a CLI host today.

If Fontana operates the host for you, you may not need the CLI on your own machines. See Cloud deployment for managed cloud and customer VPC, or Self-hosted deployment when your team runs the box.

Install

export FONTANA_GHCR_PAT=…   # read:packages token from Fontana
curl -fsSL <pages-host>/cli/install.sh | sudo -E bash   # Linux
curl -fsSL <pages-host>/cli/install.sh | bash           # macOS (no sudo)

The installer links fontana into your PATH and seeds fontana.yaml from the bundle template. All CLI-owned state (releases, snapshots, kubeconfigs) lives under one FONTANA_HOME directory (/opt/fontana on Linux production hosts, ~/.fontana on macOS).

macOS: never sudo fontana

Run fontana as your user. Only the managed /etc/hosts block prompts for an admin password during fontana apply.

Configuration

fontana.yaml is the sole declarative install config the CLI reads at apply time. Fontana publishes committed per-environment examples in the product repository (fontana.local.yaml, fontana.flow-dev.yaml, fontana.flow.yaml, fontana.flow-staging.yaml). On each host you activate one of those files as ./fontana.yaml (symlink or copy). If ./fontana.yaml is missing when you run a command in a terminal, the CLI offers to symlink a fontana.*.yaml file or create a starter from the template. The installer can also seed a starter from the bundle template for external installs.

Key	Purpose
baseDomain	Platform hostname (localhost for lab installs; your production domain otherwise). Tenants serve at <name>.<baseDomain>.
source.kind	ghcr (pull bundle + images from GHCR or your private registry mirror).
source.tag	Release tag; pin an immutable sha-… tag in production.
snapshots.on_apply	When true, take a deploy snapshot before upgrade/destroy when the release changes. Local hosts often set false.
snapshots.retain.deploy	How many pre-upgrade (deploy) snapshots to keep per tenant.
snapshots.retain.daily	How many daily scheduled snapshots to keep (0 disables daily timers).
snapshots.retain.monthly	How many monthly snapshots to keep (promoted from daily; no second cluster stop).
snapshots.schedule.daily	Optional UTC HH:MM; fontana apply installs Linux systemd timers when set and retain.daily is greater than zero.
platform.<service>.url	HTTPS origin on the EKS platform plane. Apply skips the local pod for that service. Services: docling, infinity, tts (Kokoro), opensandbox (MCP at {url}/mcp). Requires FONTANA_PLATFORM_BEARER. Do not set url and disabled on the same key.
platform.docling.disabled	When true, skip platform-cluster Docling (saves RAM; PDF preload fails hard at use time). Omitted defaults to enabled.
platform.tts.disabled	When true, skip platform-cluster Kokoro TTS (saves RAM; local TTS unavailable in Flow). Omitted defaults to enabled.
platform.opensandbox.disabled	When true, skip OpenSandbox charts and image imports (sandbox MCP fails hard at use time). Omitted defaults to enabled.
platform.observability.url	Remote OTLP gateway on platform EKS (for example https://otlp.platform.dev.fontana-ai.com). Tenant otel-collector forwards logs here; pair with platform.observability.uiUrl for Grafana. Requires FONTANA_PLATFORM_BEARER.
platform.observability.uiUrl	Operator Grafana URL (for example https://observe.platform.dev.fontana-ai.com). Admin login only; filter by fontana.environment and fontana_tenant_id.
platform.observability.disabled	When false on platform k3d, run local LGTM on the platform cluster (https://status.<baseDomain>/observe). Mutually exclusive with platform.observability.url.
tenants.<id>.zitadel.mfa	When false, skip enforced MFA for that tenant. Allowed only when baseDomain is exactly localhost. Omitted or true keeps MFA on.
tenants	Desired workspace list. fontana apply creates missing tenants and upgrades existing ones.

Secrets never belong in fontana.yaml. At apply time, export:

• FONTANA_GHCR_PAT (and optionally FONTANA_GHCR_USER) so the CLI can pull the release bundle and images
• FONTANA_PLATFORM_BEARER when any platform.<service>.url is set. Export it in the same shell as fontana apply. Mint it with terraform output -raw bearer_token in infra-k8s/envs/flow-staging-eks. On a laptop, source gitignored .env.staging (set -a && . .env.staging && set +a). fontana secrets does not print this key. After apply, the tenant overlay copies it in-cluster as PLATFORM_BEARER. OpenSandbox remotes use this same bearer; they do not mint per-tenant Vault tokens.
• Convex instance secrets are auto-generated in-cluster by the vault-seed hook

Platform remotes

Set platform.<service>.url when you want the tenant box to call shared EKS origins instead of running those pods locally. Staging and laptop examples point at *.platform.flow-staging.fontana-ai.com (Docling, Infinity, Kokoro TTS, OpenSandbox). Hosts that still deploy local pods need a fresh build/ folder. On a macOS folder-source checkout, run bash scripts/ops/local-build-apply.sh on the host (build inside the Cursor Fontana container, then fontana apply on the host). Remotes only skip local deploy on the host that sets url.

Override the config path with --config <path> or FONTANA_CONFIG.

Example production config:

version: 1
baseDomain: flow.example.com
source:
  kind: ghcr
  tag: sha-abc123def456
snapshots:
  on_apply: true
  retain:
    deploy: 5
    daily: 10
    monthly: 24
  schedule:
    daily: "03:15"
tenants:
  demo: {}

Example local / lab config (edit loops without a pre-upgrade cluster stop):

version: 1
baseDomain: localhost
source:
  kind: folder
  path: ./build
snapshots:
  on_apply: false
  retain:
    deploy: 3
    daily: 0
    monthly: 0
tenants:
  demo: {}

Full-stack snapshots (GFS retention)

Fontana keeps full-stack workspace snapshots on the host: release metadata plus the tenant data volume (data.tgz). You configure retention in fontana.yaml with a grandfather-father-son (GFS) policy so rapid redeploys cannot erase longer-horizon recovery points. On Linux production hosts that mount a persistent data volume at /var/lib/fontana/persistent, archives are stored under that mount (fontana-snapshots/) so infrastructure volume snapshots (for example AWS DLM) include the rollback trail.

Kind	When it is created	Retention key
deploy	Pre-upgrade / pre-destroy when snapshots.on_apply is true and the bundle release changed; or manual	snapshots.retain.deploy
daily	Linux systemd timer from snapshots.schedule.daily, or fontana snapshot create --kind=daily	snapshots.retain.daily
monthly	Promoted from the newest daily in the current UTC month (no second cluster stop)	snapshots.retain.monthly

On disk (under FONTANA_HOME):

snapshots/<tenant>/
  deploy/<stamp>_<version>/    # data.tgz + meta.json
  daily/<stamp>_<version>/
  monthly/<stamp>_<version>/

Each meta.json records kind, tenant, release version, creation stamp, and image pins. If an older install still has flat directories directly under snapshots/<tenant>/, the CLI moves them into deploy/ the next time you list, create, prune, or roll back.

Apply behaviour

1. When on_apply is true and the deployed release differs from the bundle, fontana apply creates a deploy snapshot (brief cluster stop while the volume is archived), then upgrades.
2. When the release is unchanged, apply skips the snapshot so a no-op reconcile does not stop the cluster.
3. After a successful apply, the CLI reconciles daily timers from schedule.daily when retain.daily is greater than zero (Linux with systemd, typically as root). Timers run fontana snapshot create <tenant> --kind=daily at the configured UTC time, with a small per-tenant minute stagger inside that hour.
4. After the volume archive, the CLI starts the tenant cluster and unseals Vault so workloads can load secrets. When remoteStore is set, it checks S3 credentials (FONTANA_REMOTE_S3_ACCESS_KEY / FONTANA_REMOTE_S3_SECRET_KEY) before it stops the cluster. Daily timers load /root/fontana-operator.env as a required EnvironmentFile in that case.
5. On macOS, or when systemd is unavailable, apply logs that the schedule was skipped; you can still create dailies manually.

Monthly promote: after each daily create, if the current UTC month has no entry under monthly/, the CLI copies the newest daily for that month into monthly/ and prunes by retain.monthly. Deploy, daily, and monthly counts prune independently.

CLI

fontana snapshot create demo                      # kind=deploy
fontana snapshot create demo --kind=daily
fontana snapshot list demo                        # TENANT KIND SNAPSHOT VERSION
fontana rollback demo                             # newest deploy
fontana rollback demo deploy/20260801T031500Z_sha-abc
fontana rollback demo daily/20260801T031500Z_sha-abc

Prefer kind/name for rollback. A bare name works only when it is unique across the three kind folders.

CLI snapshots and host disk DR

CLI GFS snapshots are the per-tenant rollback unit (fontana rollback). On AWS production hosts, EBS volume snapshots (DLM of the data volume) are the host-level disaster-recovery control and also cover the GFS archive directory when it lives on that volume. The two planes answer different failures: upgrade rollback vs disk or host loss. See Backup and restore.

Light local installs (RAM-constrained hosts)

On a laptop or workstation with limited RAM, you can omit heavy platform-cluster singletons you do not need for day-to-day Flow work. Platform-cluster skips go under platform:. Omit platform.observability entirely to skip the per-tenant otel-collector DaemonSet (~128 Mi request). Configure platform.observability.url (remote EKS) or platform.observability.disabled: false (local platform LGTM) when you want centralized logs in Grafana.

platform:
  docling:
    disabled: true # skips Docling (~2Gi request); PDF preload fails hard if used
  tts:
    disabled: true # skips Kokoro TTS (~1Gi request); local chat TTS unavailable
  opensandbox:
    disabled: true # skips OpenSandbox control plane; sandbox MCP fails hard if used
  # observability: omit url/disabled → no otel-collector on tenants

fontana apply logs disabled services, skips their image import and Helm deploy, and reconciles away workloads that were previously running. Re-enable a service by setting disabled: false or removing the key, then run fontana apply again. Helm request and limit figures for every workload are on Hardware requirements.

Production hosts

Leave platform: unset on production and integration hosts unless you accept the feature trade-offs above. For platform-central observability, set platform.observability.url and platform.observability.uiUrl. Cloud runbooks assume Docling, Kokoro TTS, and OpenSandbox are enabled on the platform cluster.

fontana apply

fontana apply is the day-to-day reconcile command.

Laptop folder-source updates

On a macOS checkout with source.kind: folder, run bash scripts/ops/local-build-apply.sh on the host. The script builds the deploy folder inside the Cursor Fontana Dev Container, then runs fontana apply on the host. First-time CLI install stays FONTANA_SOURCE=./build bash published/cli/install.sh on the host.

It:

1. Ensures the platform k3d cluster is running and deploys the gateway + shared charts (Docling, Kokoro TTS, OpenSandbox, and related platform services when not disabled in fontana.yaml).
2. For each tenant in fontana.yaml: creates or upgrades the tenant cluster and deploys the Fontana platform Helm release (with a deploy snapshot when on_apply and the release changed; see Full-stack snapshots above).
3. Refreshes Caddy edge routes and the managed /etc/hosts block once at the end (not mid-tenant, so routes stay stable during maintenance).
4. Runs fail-closed post-apply health checks (clusters, workloads, HTTPS probes).
5. Reconciles daily snapshot timers from snapshots.schedule.daily when configured (Linux systemd).

Tenants that exist on the box but are absent from YAML are warned only. Removal always requires an explicit fontana tenant destroy <name>.

Upgrade pipeline (per tenant)

When the bundle release version differs from what is deployed in-cluster and snapshots.on_apply is true, the CLI takes a deploy snapshot (stops the cluster briefly, archives the data volume), then redeploys. While upgrading, Caddy serves a static maintenance page for / on that tenant’s app hostnames. Auth hostnames stay proxied so OIDC discovery keeps working. Paths under /_* (workflow engine, Convex, collab) stay reverse-proxied so an already-open Flow tab can keep fetching port data.

Gates you will see in logs:

Gate	Why
cluster_wait_ready	Blocks kubectl/helm until server-0, serverlb, and API /readyz succeed (required after snapshot stop/start).
Vault unseal	Vault seals on cluster restart; deploy fails closed until unsealed.
Image import verify	After k3d image import, the CLI confirms the image ref is present on the node.
apply_status_check	Exits non-zero if any component or HTTPS route is unhealthy.

Unchanged releases skip the pre-upgrade snapshot automatically. To skip snapshots even when the release changes, set snapshots.on_apply: false in fontana.yaml (there is no FONTANA_SKIP_SNAPSHOT env var).

Command reference

Top-level

Command	Description
fontana apply	Sync release bundle; reconcile platform + all YAML tenants (create or upgrade). Shared static SPAs upgrade together.
fontana status	Component health checks (✅ running, ✅ 💤 YAML-disabled, ❌ failed); exits non-zero on failure.
fontana stop	Stop every live tenant cluster, then the platform cluster. Data volumes and edge config stay on disk.
fontana audit	Compare live k3d clusters and edge hostnames to fontana.yaml.
fontana resources <tenant>	List HTTPS URLs and browser console logins (Gatus, Grafana, platform Gatus). Points at fontana secrets for Convex admin key and Flow break-glass.
fontana secrets <tenant>	Print deploy credentials for .env.local (Convex, Zitadel client id, Vault keys, ImmuDB WORM audit, break-glass login/password, ZITADEL_PAT). Operator console logins live under fontana resources.

Platform

Command	Description
fontana platform create	Install platform cluster + gateway (alias: add).
fontana platform upgrade	Redeploy platform charts from the current release.
fontana platform destroy	Delete the platform cluster (tenants untouched).
fontana platform start / stop / status / list	Cluster lifecycle and listing.

Tenant

Command	Description
fontana tenant add <name>	Create tenant cluster + deploy the app stack.
fontana tenant destroy <name>	Final deploy snapshot when on_apply is true, then delete cluster + data volume + routes.
fontana tenant start / stop / status <name>	Cluster lifecycle.
fontana tenant list	List tenants with app and auth hostnames.

Snapshots and rollback

Command	Description
fontana snapshot create <name> [--kind=deploy|daily|monthly]	Full-stack snapshot (default kind deploy).
fontana snapshot list [name]	List snapshots (tenant, kind, name, version).
fontana rollback <name> [kind/name|name]	Restore data + redeploy pinned release (default: newest deploy snapshot).

Credentials and URLs

Command	Description
fontana resources <name>	HTTPS URLs for Flow, APIs, Gatus, Convex dashboard, Grafana (platform.observability.uiUrl or local /observe), plus browser console logins (Gatus, Grafana, platform Gatus). Convex admin key and Flow break-glass stay in fontana secrets.
fontana secrets <name>	Full .env.local credential block for devcontainers (Convex, Zitadel client id, ImmuDB, service keys, break-glass login/password, ZITADEL_PAT). No Gatus/Grafana browser logins.

Use fontana resources for browser URLs and operator-console sign-in. Use fontana secrets when wiring a local dev loop or copying keys into .env.local.

State on disk

All CLI-owned paths sit under FONTANA_HOME:

Path	Contents
releases/ / current	Downloaded GHCR release bundles for the pinned tag.
snapshots/<tenant>/{deploy,daily,monthly}/	GFS snapshot tarballs + meta.json (kind, release, image pins).
kubeconfigs/	Per-cluster kubeconfig files (not ~/.config/k3d).
cluster-ports/<cluster>	Recorded API + HTTP port pairs for recreate after Docker cleanup.
image-import-stamps/<cluster>/	Content-hash stamps for skipped tarball imports.

Each k3d cluster mounts a named Docker volume fontana-data-<cluster> at the k3s storage path. That volume is the unit for snapshots and rollback.

Host ports are discovered from the live serverlb container (no separate clusters.json registry). During snapshot stop, edge routing uses HostConfig.PortBindings so Caddy keeps stable :808x mappings while containers are stopped.

Recovery after manual Docker cleanup

If you remove k3d containers or images outside fontana (Docker Desktop prune, manual docker rm, and similar), k3d can lose its node records while the data volume still holds PVC data. Symptoms include:

• No nodes found for given cluster during fontana apply or after a snapshot restart
• Docling or tarball import errors such as content digest … not found
• Edge HTTPS failing after a partial apply

The CLI is designed to recover automatically when the data volume remains:

1. Snapshots can proceed even when k3d node records are missing.
2. When a cluster fails to start but fontana-data-<cluster> still exists, the CLI recreates the k3d cluster on the same host ports and re-attaches the data volume.
3. Port mappings are read from the live cluster or from ${FONTANA_HOME}/cluster-ports/<cluster> if containers are gone.
4. fontana apply treats a tenant with a data volume but no k3d cluster as an upgrade (not a fresh create), so you do not lose tenant data.
5. Image imports: after recreate, the CLI re-pulls images from your registry and verifies each ref is present on the node before deploy continues.

What you should do:

export FONTANA_GHCR_PAT=…   # required if Docker pruned cached images
fontana apply

Vault seals when a cluster is recreated; the deploy pipeline unseals it before hook jobs run. If apply still fails, check fontana status and logs. A full reset is fontana tenant destroy <name> then fontana apply (destroys in-cluster state unless you restore from a snapshot).

Port assignment after a full wipe

If both k3d containers and the cluster-ports/ file are gone, the CLI may allocate new HTTP ports (you will see a warning in logs). Edge routes and /etc/hosts update on the next successful apply. To preserve URLs, restore from snapshot or avoid deleting FONTANA_HOME/cluster-ports/.

Environment variables

Variable	Purpose
FONTANA_CONFIG	Path to fontana.yaml (alternative to --config).
FONTANA_HOME	Root for releases, snapshots, kubeconfigs, port records.
FONTANA_GHCR_PAT / FONTANA_GHCR_USER	GHCR pull credentials at apply time.
FONTANA_PLATFORM_BEARER	Required when any platform.<service>.url is set. Same shell as fontana apply.
FONTANA_SKIP_MAINTENANCE=1	Skip maintenance-page Caddy routes during upgrade.
FONTANA_LOG_FORMAT=json	Structured JSON logs (OTel-friendly).
FONTANA_K8S_CLUSTER_READY_TIMEOUT	Seconds to wait for API /readyz (default 180).

Pre-upgrade snapshot skip is snapshots.on_apply: false in fontana.yaml, not an environment variable.

Logging

Install and deploy logs use a unified format:

2026-06-30T06:11:48Z INFO [tenant-demo] apply: tenant 'demo': exists - upgrading
2026-06-30T06:12:38Z INFO [tenant-demo] cluster: recreating k3d cluster (api=127.0.0.1:6446 http=8082, data volume preserved)

Cluster context appears in brackets (platform, tenant-demo). Components include apply, cluster, snapshot, source, deploy, and edge.

Related documentation

• Architecture: platform and workspace clusters
• Hardware requirements: host RAM, CPU, and disk
• Backup and restore: snapshots, rollback, Admin configuration export
• Cloud deployment: managed cloud and customer VPC
• Self-hosted deployment: on-premises and private registry mirrors
• Deployment overview: deployment models and release pinning
• Observability: platform Grafana and Loki during fontana apply