# Fontana product documentation

This file is a single plain-text export of every Starlight documentation page on the Fontana website, in sidebar order. Each section starts with a page title, description, and canonical URL, followed by unrendered markdown body content. MDX import lines and Starlight-only frontmatter are removed.

# Getting Started
## Description: Welcome to Fontana documentation: agent harness, knowledge graph, workflow engine, and core runtime components.
## url: https://fontana-ai.com/docs/welcome/getting-started/

Fontana is an **Agent Harness**, **Knowledge Graph**, and **Workflow Engine** - integrated as a governed **AI platform** for organisations with strict compliance, audit, and operational control requirements.

You use Fontana to design data pipelines on an interactive canvas, run them on a reactive server-side engine, and layer **governed AI agents** on top with tools, skills, and approved model access. Every workspace is isolated; every privileged action can leave audit evidence.

#### What you can do in the Flow app:

- **Create AI agents** and assign modular [Skills](/docs/ai/skills/)
- **Build Knowledge Graphs** so agents retrieve approved operational knowledge
- **Configure agentic workflows** with agents, tools, and graphs on the Flow canvas
- **Ingest data** from files, APIs, and 600+ connectors
- **Transform and validate** data with built-in operations
- **Analyze results** with visual reports and interactive queries
- **Export or deliver** outputs to downstream systems

#### Explore the documentation

- **[Platform overview](/docs/welcome/platform-overview/)** - runtime stack, platform capabilities, and external service boundaries
- **[AI](/docs/ai/)** - gateways, models, agents, Knowledge Graph, tools, MCP, BYOK, and ZDR
- **[Workflows](/docs/workflows/)** - visual pipelines, execution engine, nodes, and operations
- **[Identity and access](/docs/auth/)** - Zitadel SSO, identity providers, roles, and application RBAC
- **[Data](/docs/data/)** - ingress, secure Arrow storage, egress, and connector secrets
- **[Observability](/docs/observability/)** - operational telemetry, Gatus health, and security audit (WORM)
- **[Deployment](/docs/deployment/)** - cloud, self-hosted, architecture, backup and restore, Fontana CLI
- **[Security](/docs/security/)** - workspace isolation, secrets, encryption, network hardening, supply chain, SOC 2 control summary
- **[Compliance evidence](/docs/compliance/)** - lineage, immutable audit trail

<DocsTermAside title="What is an Agent Harness?">

An agent harness is the control layer that surrounds an AI agent, providing access to data, tools, workflows, and enterprise systems. At Fontana, the harness also delivers governance capabilities such as audit logs, human-in-the-loop approvals, permissions, validation rules, and monitoring, ensuring that AI agents operate safely and in line with organisational controls.

</DocsTermAside>

<DocsTermAside title="What is the Workflow Engine?">

The server-side execution layer for DAG data transformation pipelines. It schedules idempotent node runs, persists run-scoped files, executes transforms and connectors, and writes audit and lineage metadata alongside result datasets.

</DocsTermAside>


# Platform overview
## Description: Core runtime components, platform capabilities, and external service boundaries - AI, workflows, identity, data, observability, deployment, and compliance.
## url: https://fontana-ai.com/docs/welcome/platform-overview/

Fontana ships as an integrated stack you can run on major clouds or on-premises: a modern web app, real-time backend, workflow engine, identity plane, secrets store, observability, and connector runtime, each with a clear role in production. The sections below cover platform capabilities, the technology stack in a typical workspace, and how external services are bounded.

#### Platform capabilities

Fontana delivers governed capabilities for data, workflows, and AI. Each area below is a first-class part of what you run in production.

- **[Governed AI](/docs/ai/)** - Your approved LLM gateways and models, BYOK key scoping at org, team, or person level, agents and skills, Knowledge Graph retrieval, tools, MCP, and agent interop under one control surface.
- **[Workflow orchestration](/docs/workflows/)** - Visual pipelines on the Flow canvas, reactive server-side execution, node families for transforms and connectors, plus built-in validations, conditions, and document parsing.
- **[Identity and access](/docs/auth/)** - Self-hosted Zitadel per workspace with OIDC, corporate SSO, MFA, SCIM provisioning, and application RBAC in Flow for teams and admin surfaces.
- **[Data](/docs/data/)** - File uploads, PyAirbyte connectors, encrypted Arrow storage, REST triggers, and governed export
- **[Observability](/docs/observability/)** - Platform-cluster HyperDX for OTLP logs and dashboards, in-cluster Gatus health probes, and observability-api routing operational telemetry alongside the security audit ledger.
- **[Deployment](/docs/deployment/)** - cloud, self-hosted, [architecture](/docs/deployment/architecture/), [backup and restore](/docs/deployment/backup-and-restore/), Fontana CLI
- **[Security](/docs/security/)** - cluster-per-tenant isolation, Vault secrets, encryption, network hardening, supply chain integrity, and SOC 2-aligned control summary
- **[Compliance evidence](/docs/compliance/)** - workflow data lineage and immutable security audit trail

#### Architecture

Fontana uses a **platform cluster** plus one **workspace cluster per tenant** for isolation. Shared services (edge gateway, observability, Docling, OpenSandbox) run on the platform; Flow, Convex, workflow engine, Vault, Zitadel, and workspace data run inside each workspace cluster.

<Aside type="note" title="Cluster-per-tenant">

Multiple workspaces can share one physical host, each in its own Kubernetes cluster. That separation is the primary **data and secrets boundary**. See **[Workspace isolation](/docs/security/workspace-isolation/)** for soft vs dedicated-host tenancy.

</Aside>

#### Topology diagram

<TechnicalDocArchitectureTopology client:only="react" />

Component breakdown and deployment models: **[Deployment architecture](/docs/deployment/architecture/)**.

#### Technology stack

The table below summarises the major components in a typical workspace and what each one does.

| Component                   | Technology                  | Role                                                                                              |
| --------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------- |
| **Host**                    | AWS / Azure / GCP / on-prem | Deploy anywhere - platform-agnostic infrastructure                                                |
| **Container orchestration** | Kubernetes (K3s)            | Multi-tenant container orchestration with per-workspace cluster segregation                       |
| **Flow App**                | Modern React PWA            | Web UI - canvas, grids, AI chat, admin panels                                                     |
| **Workflow engine**         | Node.js, Python             | Server-side execution of workflow graphs                                                          |
| **Real-time backend**       | Convex                      | Application backend - config, AI orchestration, persistence                                       |
| **Auth**                    | Zitadel                     | Self-hosted OIDC identity per workspace - SSO, MFA, SCIM provisioning                             |
| **Secrets storage**         | HashiCorp Vault             | Per-workspace encrypted secret storage (API keys, connector credentials) - SOC 2-aligned controls |
| **Observability**           | HyperDX                     | Platform-cluster OTLP collector and operator dashboards (shared across workspaces)                |
| **AI code sandbox**         | OpenSandbox                 | Isolated code execution for agents via MCP                                                        |
| **Data replication**        | Airbyte (PyAirbyte)         | Data ingress and egress - catalog connectors in the workflow engine                               |

#### External services

Fontana is designed for security and contractual control: the platform stack above runs inside **your** Kubernetes boundary with per-workspace Postgres, Convex, Vault, and Zitadel. Fontana does not operate a shared LLM pool or route production inference through Fontana-owned model accounts. Your workspace data is not sent to external **data processors** for platform operation beyond what **you** configure and approve.

When you use AI, requests leave the workspace only through **LLM gateways and providers you choose**. You store credentials with **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** in per-workspace Vault and, in **Admin**, approve which gateways and model IDs are reachable in production. Fontana supports **12+ gateways** (unified routers such as OpenRouter and LLM API, direct vendor APIs, and cloud-hosted foundation models on AWS Bedrock) and **300+ models** behind one governed routing layer. Spend, retention, training policy, and Zero Data Retention (ZDR) posture follow **your** provider contracts and gateway settings, not a shared vendor account Fontana operates on your behalf. See **[Bring Your Own Key (BYOK)](/docs/ai/byok/)**, **[Zero Data Retention (ZDR)](/docs/ai/zdr/)**, **[Governed AI](/docs/ai/)**, and **[Gateways](/docs/ai/gateways/)** for key scoping, retention posture, and approval workflows.

You may also connect to external systems through **connectors**, **REST and GraphQL triggers**, and **MCP** servers when your workflows require them. Connector and API secrets resolve from Vault at execution time; nothing is embedded in workflow config. Operational telemetry and security audit routing stay within the model described in **[Observability](/docs/observability/)**.


# Overview
## Description: Governed AI across workflows, chat, and automation with BYOK gateways, approved models, and auditable agent tooling.
## url: https://fontana-ai.com/docs/ai/

Fontana AI gives you **governed agents** for Flow chat, workflow automation, and operational tasks. You choose which gateways and models are approved, scope API keys with **[BYOK](/docs/ai/byok/)**, attach **[Knowledge Graph](/docs/ai/knowledge-graph/)** namespaces for retrieval, and extend agents with **[Skills](/docs/ai/skills/)**, built-in tools, and **[MCP](/docs/ai/tools-and-mcp/)** integrations. Every tool call and delegation can leave audit metadata aligned with your compliance posture.

#### Topics

- **[Gateways](/docs/ai/gateways/)** - **12+ LLM gateways** with BYOK routing, from unified routers (OpenRouter, LLM API) to direct vendor APIs and AWS Bedrock.
- **[Models](/docs/ai/models/)** - **300+ frontier and open-weight models** in an Admin-approved catalogue, all routed through the gateways and keys you configure.
- **[Knowledge Graph](/docs/ai/knowledge-graph/)** - Namespace-scoped operational knowledge with embeddings and versioned documents so agents retrieve approved context instead of rediscovering rules every run.
- **[Agents](/docs/ai/agents/)** - Configurable in-app personas for Flow chat and automation, each with its own model, tools, skills, Knowledge Graph boundaries, and workflow-aware context.
- **[Human in the Loop](/docs/ai/human-in-the-loop/)** - Approval gating for AI operations: structured `askHuman` forms, deferred CITL execution, canvas suggestion apply, and NL Operations compile review.
- **[Skills](/docs/ai/skills/)** - Modular agentskills.io instruction packages that extend agents with governed, on-demand procedures loaded only when the task requires them.
- **[Tools and MCP](/docs/ai/tools-and-mcp/)** - Built-in tools for workflows, data, canvas editing, and documents, plus Model Context Protocol (MCP) integrations and isolated OpenSandbox execution.
- **[Agent Interop](/docs/ai/agent-interop/)** - Delegate to specialist agents, hand off conversations in-thread, or federate with external agent platforms over A2A, ACP, and related open protocols with full audit coverage.
- **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** - API keys you control in per-workspace Vault, scoped at org, team, or person level, with no shared Fontana-operated model account.
- **[Zero Data Retention (ZDR)](/docs/ai/zdr/)** - Align LLM routing with your retention and no-training requirements through provider policies and your BYOK gateway settings.

#### Related documentation

- **[Security](/docs/security/)** - Vault secrets, BYOK alignment, SOC 2 control summary (AI governance row)
- **[Compliance evidence](/docs/compliance/)** - tool-call audit and immutable audit trail
- **[Tools and MCP](/docs/ai/tools-and-mcp/)** - OpenSandbox isolation (see **[Workspace isolation](/docs/security/workspace-isolation/)**)


# Gateways
## Description: Built-in support for 12+ LLM gateways with BYOK routing through unified routers, direct vendor APIs, and AWS Bedrock.
## url: https://fontana-ai.com/docs/ai/gateways/

Fontana provides built-in support for a wide range of LLM gateways, from unified routers such as OpenRouter and LLM API to direct vendor APIs and cloud-hosted foundation models on AWS Bedrock. All calls route through one gateway abstraction with **[Bring Your Own Key (BYOK)](/docs/ai/byok/)**: you supply the API keys your organisation holds, and administrators approve which gateways and models are reachable in production. Knowledge Graph embeddings use the same stack, with configurable gateway priority (OpenRouter → OpenAI → LLM API).

For retention and no-training posture on each gateway, see **[Zero Data Retention (ZDR)](/docs/ai/zdr/)**.

#### Approved gateways

<DocsAiGatewaysTable />


# Models
## Description: 300+ frontier and open-weight models from major providers, routed through your approved BYOK gateways.
## url: https://fontana-ai.com/docs/ai/models/

You control which LLM provider families and model IDs are available in your organisation. Fontana ships **300+ supported models** across leading providers: frontier chat and reasoning models, open-weight endpoints, and domain-specific hosts, all routed through your **approved gateways and [BYOK](/docs/ai/byok/) keys** (see **[Gateways](/docs/ai/gateways/)**). Usage stays aligned with your contracts, compliance posture, and budget because inference never runs through a shared Fontana-operated model account.

<Aside type="note" title="Admin approval">

Admins enable gateway families and individual model IDs in **Admin → AI**. Agents and workflows can only call models that remain on the approved list for your workspace.

</Aside>

#### Supported model providers

<DocsAiModelsTable />

#### Custom or internal providers

If your preferred provider or model is not listed above, or you run internal inference endpoints, contact the Fontana Engineering team. We can add support for custom gateways, private hosts, or enterprise-only model access to match your deployment boundary.

#### Related documentation

- **[Gateways](/docs/ai/gateways/)** - unified routers, direct vendor APIs, AWS Bedrock
- **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** - Vault-backed keys scoped at org, team, or person level
- **[Zero Data Retention (ZDR)](/docs/ai/zdr/)** - retention and no-training posture per provider


# Knowledge Graph
## Description: Governed operational knowledge with namespace-scoped retrieval, embeddings, and versioned documents for agents.
## url: https://fontana-ai.com/docs/ai/knowledge-graph/

The Knowledge Graph allows you to store, version, and govern the operational knowledge your agents need: firm rules, mappings, approval criteria, exception history, and process documentation. You organise documents in namespaces and folders, assign each agent the corpora they may search, and agents pull **approved, versioned context** on each turn instead of re-stating the same logic in every chat. Namespace boundaries keep regulated corpora segregated, so retrieval stays least-privilege and focused on the domain you intend.

<Aside type="note" title="What is the Knowledge Graph?">

A context store for operational knowledge: firm rules, mappings, approvals, exception history, and process documentation. Documents are namespace-scoped with version history; agents retrieve approved context instead of rediscovering logic on every run.

</Aside>

<KnowledgeGraphContextDiagram />

#### How agents get context

Before each chat or automation turn, Fontana **automatically builds context** from the Knowledge Graph. It runs vector search over the namespaces you assigned to that agent, ranks documents by similarity to the current message, and injects full text for the best matches above your configured threshold. You do not paste rules or procedures into every prompt; the agent starts each turn with **approved, versioned knowledge** already in place.

This pre-turn pass is fast, scoped to the corpora you permit, and bounded so only likely-relevant articles enter the thread before the model responds.

#### Agentic search

Automatic injection covers the strongest matches up front; **agentic search** lets agents improve results **during the work itself**. As the task develops, agents can explore the Knowledge Graph again: run semantic searches with **vector similarity**, inspect ranked summaries and scores, and pull **full document text only for hits that support the current step**. They can also retrieve exact articles with `@slug` references when a specific governed document applies.

Exploration stays inside the namespaces and permissions you configured, so agents can deepen context without loading your entire corpus into every conversation.

#### Embeddings and retrieval

Knowledge Graph documents are indexed and embedded as **1536-dimensional vectors** (`text-embedding-3-small`). Both automatic pre-turn injection and agentic search rely on this index; injection uses the agent's `kbInjectionScoreThreshold` (minimum similarity score; default `0.35`).

<Aside type="note" title="What are embeddings?">

Dense vector representations of text used for semantic search. Fontana stores 1536-dimensional vectors on Knowledge Graph documents and workflow templates; only title, description, and vectorSource are sent to the embedding provider, not full document bodies.

</Aside>

When you upload or save documents, Fontana parses structured files (CSV, TSV, JSON) and prose (Markdown, text) into governed records, indexes them for search, and builds a **relationship graph** linking cross-references, semantically similar neighbours, and folder structure within each namespace.

#### Knowledge Graph Document content types

Structured documents use **CSV**, **TSV**, or **JSON** dataTypes. Unstructured prose and **Markdown** (YAML frontmatter) use **text**. **PSV** and other delimited formats are supported on workflow File Input - see **Data** docs.

| Data type           | Usage                                                                                                                  |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Text / Markdown** | Unstructured prose and Markdown articles with YAML frontmatter. Default `text` dataType in the Knowledge Graph editor. |
| **CSV**             | Comma-delimited tabular sources. Parsed to delimited JSON rows for structured agent retrieval.                         |
| **TSV**             | Tab-delimited tabular sources. Parsed to delimited JSON rows for structured agent retrieval.                           |
| **JSON**            | Structured JSON document bodies with schema-aware editing and validation in the admin UI.                              |

#### Knowledge Graph namespaces

Knowledge Graph documents are organised in three levels: a **namespace** (top-level corpus boundary), a dot-path **folder** within that namespace, and individual **documents** addressed by slug. Namespaces and folders are managed in the **Knowledge Graph** admin UI; each document carries version history.

<TechnicalDocKgNamespacesDiagram client:only="react" />

Each agent searches only the namespaces you assign in **Admin → Agents**. Retrieval and relationship expansion stay inside those boundaries, which keeps regulated corpora segregated and agent answers focused on the domain you intend.


# Agents
## Description: Configurable in-app AI personas for Flow chat, with tools, skills, Knowledge Graph access, and workflow-aware context.
## url: https://fontana-ai.com/docs/ai/agents/

In Flow, **agents** are governed AI personas for chat and automation. You configure each one in **Admin → Agents** with its own model, tools, skills, and [Knowledge Graph](/docs/ai/knowledge-graph/) boundaries. Every tool call is audited; every agent runs inside the permissions you set.

Agent definitions follow the open [AGENTS.md](https://agents.md) pattern. You can edit each agent as a structured form or as one raw document; both views produce the same portable definition your team can version, review, and adapt for production.

<Aside type="note" title="What is an agent?">

A configured AI persona in Flow: system prompt, approved model, allowlisted tools, optional skills, Knowledge Graph namespaces, and delegation rules. Agents are not open-ended model access; they are bounded assistants that operate on your data and workflows under RBAC and audit.

</Aside>

#### Shipped defaults

Fontana ships ready-to-use agents you can tailor for your organisation:

- **Workflow Architect** (orchestrator) - default chat agent for clarifying intent, planning work, querying workflows and data, managing task lists, and delegating specialised tasks. It does not mutate the canvas directly.
- **Canvas agent** - specialist for building and editing workflow graphs (nodes, edges, ports, and canvas config), typically invoked by delegation from the orchestrator.

You can clone these defaults, adjust prompts and tool allowlists, and assign additional [Skills](/docs/ai/skills/) as your operating model requires.

#### Configure each agent

In **Admin → Agents**, you control how each persona behaves in production:

| Setting                           | What you control                                                                                                              |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Id and display name**           | Stable identity and the label users see in chat                                                                               |
| **Description and system prompt** | Short summary plus Markdown instructions that define tone, scope, and operating rules                                         |
| **Language model**                | Approved model override; leave empty to use the deployment default                                                            |
| **Tools**                         | Allowlisted entries from the audited catalog; see [Tools and MCP](/docs/ai/tools-and-mcp/)                                    |
| **Scopes**                        | Where the agent may run (for example Flow chat vs internal programmatic calls)                                                |
| **Sub-agents**                    | Delegation targets that run work in child threads                                                                             |
| **Handoff agents**                | Same-thread transfer targets when the conversation needs a different specialist; see [Agent Interop](/docs/ai/agent-interop/) |
| **Skills**                        | Assigned instruction packages; see [Skills](/docs/ai/skills/)                                                                 |
| **Knowledge Graph namespaces**    | Which governed corpora the agent may search and retrieve                                                                      |
| **Vector source**                 | Keywords that help route and discover this agent and related workflow templates                                               |
| **Call settings**                 | Optional JSON for temperature, topP, maxOutputTokens, and related LLM parameters                                              |
| **Reasoning effort**              | Agent default; per-thread chat controls can override                                                                          |
| **Context window slots**          | How much recent transcript the agent keeps in context; per-thread overrides available                                         |
| **KB injection score threshold**  | Minimum similarity score for automatic Knowledge Graph injection on each turn; per-thread overrides available                 |

#### Automatic and agentic context

Agents receive Knowledge Graph context in two ways, described fully on **[Knowledge Graph](/docs/ai/knowledge-graph/)**:

- **Automatic pre-turn injection** - fast vector search over assigned namespaces before each message, so likely-relevant approved documents are already in context.
- **Agentic search** - during the work, agents can search again, explore ranked matches, and pull full text only for documents that support the current step.

You assign namespaces per agent so regulated corpora stay segregated and retrieval stays least-privilege.

#### Workflow-aware assistance

Agents can search community workflow templates by semantic similarity and, with user consent, read workflow configuration from the canvas. That lets assistants help you design and refine pipelines with awareness of graph structure, node types, and connections.

Runtime execution data is queried separately from canvas configuration, so design assistance and live run results stay strictly separated.

#### Delegation and handoff

Complex work can span multiple agents without losing governance:

- **Sub-agents** run delegated tasks in child threads while the parent conversation continues under audit.
- **Handoff** transfers the same thread to another in-app agent when the task surface changes (for example from planning to canvas editing).

Peer-agent federation with external platforms uses separate interop protocols. See **[Agent Interop](/docs/ai/agent-interop/)** for A2A, ACP, and related patterns.


# Human in the Loop
## Description: Approval gating for AI operations in Flow chat, canvas edits, and natural-language operations, with structured askHuman forms and auditable deferred execution.
## url: https://fontana-ai.com/docs/ai/human-in-the-loop/

Fontana treats **human approval as a first-class control**. Agents can analyse, draft, and propose, but consequential changes are not applied until approved. Chat threads pause on structured prompts, canvas edits require an explicit apply step, and natural-language operation compiles stay pending until you confirm. Every path leaves audit metadata you can review alongside **[Compliance evidence](/docs/compliance/)**.

<div class="md:w-2/3 not-content">
  <HitlDiagrams />
</div>

<Aside type="note" title="What is human in the loop here?">

Governed checkpoints where AI work stops until a person approves, answers a structured question, or rejects a proposal. Fontana implements this across chat tools, canvas suggestions, and Operations-node NL compile review, not only in free-text conversation.

</Aside>

#### Assist vs approval boundaries

Not every AI step needs a human click. Fontana separates **assist** work (safe to run automatically under your agent and RBAC settings) from **approval-gated** work (changes that affect production rules, outputs, or control policies).

| Agents can assist automatically                          | Tasks that require your approval                           |
| -------------------------------------------------------- | ---------------------------------------------------------- |
| Analyse files, schemas, and source outputs               | Change production rules or workflow boundaries             |
| Infer fields, mappings, and validation checks for review | Approve exceptions, outputs, or control changes            |
| Draft operating specifications and mapping proposals     | Execute operational steps in production                    |
| Explain breaks, exceptions, and supporting evidence      | Bypass prompt, output, routing, version, and decision logs |
| Propose workflow configuration before it is applied      | Override named owners, approval gates, or control policies |
| Summarise evidence packs for reviewers                   | Act outside defined permissions and data-class boundaries  |

Your **[Agents](/docs/ai/agents/)** allowlist, **[Roles and RBAC](/docs/auth/roles-and-rbac/)**, and per-thread data-sharing toggles further narrow what server-side tools can run without an explicit human step.

#### The `askHuman` tool (structured chat prompts)

The primary chat HITL mechanism is the built-in **`askHuman`** tool (catalog id **`chat_askHuman`**). Agents call it whenever they need a question, confirmation, or choice from you. The model must use the tool rather than relying on plain-text questions alone, so the thread **pauses until you submit an answer** and the UI always renders a governed form.

**Input shape:**

| Field          | Required | Purpose                                                                                             |
| -------------- | -------- | --------------------------------------------------------------------------------------------------- |
| **`schema`**   | Yes      | JSON Schema object describing fields, types, validation, and question copy (`title`, `description`) |
| **`uiSchema`** | No       | Optional UI hints (widgets, layout) compatible with react-jsonschema-form conventions               |

**Response:** a **stringified JSON object** that conforms to `schema`. The agent receives that JSON as the tool result and only then continues the turn.

Flow renders the form in the chat **supplementary drawer** (HITL slot). When multiple `askHuman` calls are pending, you page through them one at a time. Submitting an answer calls `submitToolResult`, which persists the result and schedules the next model turn.

##### Example schemas agents can emit

**Boolean approval:**

```json
{
  "type": "object",
  "properties": {
    "approved": {
      "type": "boolean",
      "title": "Approve this change?",
      "description": "Do you want to proceed with this modification?"
    }
  },
  "required": ["approved"]
}
```

**Single select:**

```json
{
  "type": "object",
  "properties": {
    "priority": {
      "type": "string",
      "enum": ["low", "medium", "high"],
      "title": "Priority level"
    }
  },
  "required": ["priority"]
}
```

**Multi-select checkboxes** (via `uiSchema`):

```json
{
  "type": "object",
  "properties": {
    "categories": {
      "type": "array",
      "items": {
        "type": "string",
        "enum": ["urgent", "important", "optional"]
      },
      "title": "Select categories"
    }
  },
  "uiSchema": {
    "categories": { "ui:widget": "checkboxes" }
  }
}
```

**Number slider:**

```json
{
  "type": "object",
  "properties": {
    "threshold": {
      "type": "number",
      "minimum": 0,
      "maximum": 100,
      "title": "Threshold"
    }
  },
  "uiSchema": {
    "threshold": { "ui:widget": "range" }
  }
}
```

**Multiple fields in one prompt:** use an object schema with several `properties`. Boolean and single-select fields can render as one-click controls when they are the only field in the form.

Assign **`chat_askHuman`** to agents in **Admin → Agents** like any other built-in tool. See **[Tools and MCP](/docs/ai/tools-and-mcp/)** for the wider tool catalog.

#### Deferred execution (CITL)

`askHuman` is a **deferred tool**: it has no server-side executor. After the model turn completes, Fontana persists the tool call and waits for the browser.

```text
Model turn completes → tool call persisted (pending)
       ↓
Flow detects pending askHuman → renders form (CITL)
       ↓
You submit → submitToolResult → tool result persisted
       ↓
Thread schedules next model turn
```

**Client In The Loop (CITL)** orchestration applies only to `askHuman` / `chat_askHuman`. All other agent tools (workflow queries, canvas mutations, Knowledge Graph search, MCP calls) execute **server-side** after the same deferred pipeline, subject to allowlists and thread flags such as **Share workflow configuration with AI** and **Share run data with AI**.

One active browser tab holds the CITL lease per conversation root so duplicate submissions do not race. While a prompt is open, the thread run status shows that the agent is **waiting for you**, not still analysing.

#### Canvas suggestion approval

The **[Canvas agent](/docs/ai/agents/)** mutates a **draft** workflow graph stored per chat thread (`ai_thread_canvas`), not your saved workflow directly. When the agent finishes a batch of edits, Flow shows a **suggestion card** comparing the draft to your baseline. Nothing is written to the live canvas until you click **Apply**.

| Step        | What happens                                                                                                     |
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| Agent edits | Typed canvas tools update server-side `aiConfig` on the thread canvas row                                        |
| Review      | `getCanvasSuggestions` returns a structured diff in the chat or canvas panel                                     |
| Apply       | `applyCanvasSuggestions` promotes the draft, with three-way merge if you edited locally since the thread started |
| Conflict    | If agent and live edits overlap, you choose merge resolution or overwrite via an explicit modal                  |

Auto-apply is **opt in** (user preference, default off). Production canvas changes always go through the same apply mutation when auto-apply is disabled.

#### Natural-language Operations compile review

On Operations nodes, plain-English rules compile through the NL Operation Compiler agent. Successful compiles land in a **pending card** (dashed border) until you:

- **Confirm** - merge (possibly edited) operations into the node
- **Rephrase** - send the description back to the composer for another compile attempt
- **Reject** - discard the proposal

Policy validation and automatic repair retries run in Convex; you always see the final proposal before it affects workflow execution. Confirmed operations collapse in the editor; the composer is disabled while a pending card is visible.

#### Governance and audit

Human-in-the-loop checkpoints stack with other AI controls:

| Layer                | How it gates AI work                                                                   |
| -------------------- | -------------------------------------------------------------------------------------- |
| **Tool allowlist**   | Unassigned tools (built-in or MCP) are not callable                                    |
| **RBAC**             | Flow hides admin and mutation surfaces when your role lacks permission                 |
| **Thread toggles**   | Disable data or config sharing to block query tools until you re-enable via `askHuman` |
| **Deferred tools**   | `askHuman` cannot complete without your browser submission                             |
| **Suggestion apply** | Canvas drafts never replace saved workflows silently                                   |
| **NL compile HITL**  | Operations are not persisted until Confirm                                             |
| **Audit trail**      | Tool calls, submissions, and apply actions write reviewable records                    |

For immutable security audit and operational telemetry boundaries, see **[Security audit (WORM)](/docs/observability/security-audit-worm/)** and **[Compliance evidence](/docs/compliance/)**.

#### Related documentation

- **[Agents](/docs/ai/agents/)** - configure which personas may call `askHuman` and other tools
- **[Tools and MCP](/docs/ai/tools-and-mcp/)** - built-in catalog, MCP allowlists, and audit
- **[Skills](/docs/ai/skills/)** - temporary tool expansion during governed playbooks
- **[Workflows](/docs/workflows/)** - Operations nodes and NL compile behaviour


# Skills
## Description: Modular agentskills.io instruction packages that extend agents with governed, on-demand procedures.
## url: https://fontana-ai.com/docs/ai/skills/

**Skills** let you package repeatable expertise once and assign it to [agents](/docs/ai/agents/) in **Admin → Agents**. Each skill is a governed playbook: when to use it, the steps to follow, optional reference files, and which tools may run while the skill is active. Agents see a lightweight catalog on every turn and load the full procedure **only when the task requires it**, so chat stays focused without sacrificing depth.

Fontana implements the open [agentskills.io](https://agentskills.io) format, so skills remain portable, versionable, and auditable inside and outside your workspace.

<Aside type="note" title="What are Skills?">

Modular instruction packages in the agentskills.io format (`SKILL.md` plus optional bundled files). Assignment is per agent; activation is explicit. Skills are playbooks for _how_ to perform a task, distinct from [Knowledge Graph](/docs/ai/knowledge-graph/) articles that hold _what_ your organisation knows.

</Aside>

#### Why use skills

Without skills, long procedures either bloat every prompt or live only in an agent's system message where they are hard to reuse across personas. Skills give you:

- **One authored source of truth** for a procedure (canvas edits, reconciliations, approval checklists, domain playbooks)
- **Per-agent assignment** so only the right personas may activate a skill
- **Governed tool expansion** while a skill session is open, without permanently widening an agent's allowlist
- **Audit-friendly sessions** with activation, resource fetches, and close events on the chat transcript

Fontana ships default skills for common canvas and workflow tasks; you can author additional packages in the **Skills** admin UI or import validated `SKILL.md` packages from your repository.

#### Progressive disclosure

Skills use **progressive disclosure** so context stays lean until the model commits to a playbook:

1. **Catalog on each turn** - assigned skill names and descriptions appear in agent context so the model can decide whether a skill applies.
2. **Activation** - when a skill matches the work, the agent calls `activate_skill` with the skill's stable id. Only then is the full procedure body injected.
3. **Bundled references** - while a session is open, the agent can fetch manifest-bound files (patterns, examples, checklists) with `fetch_skill_resource` instead of inlining every attachment up front.
4. **Close** - when the playbook is complete, the agent calls `close_skill`. Provider context redacts closed skill internals on later turns; the transcript retains audit rows for review.

Assignment in **Admin → Agents** means a skill _may_ be used in that agent's threads. Unassigned skills are invisible to the model. Activation is always explicit; vector search never auto-activates a skill.

#### Author a skill package

Each skill is a versioned **`SKILL.md`** file with YAML frontmatter and a Markdown procedure body. Fontana validates packages at create, edit, and import (frontmatter keys, allowed tools, resource manifest, and file hashes).

| Frontmatter key   | Required | What it controls                                                                                              |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
| **name**          | Yes      | Human-readable title in the catalog and admin UI                                                              |
| **description**   | Yes      | When the model should consider this skill; shown in the catalog before activation                             |
| **allowed-tools** | No       | Tool ids from the audited catalog; unioned into the agent's effective tools while the skill session is active |

The Markdown body after the frontmatter holds the full procedure: steps, guardrails, and examples. It is not sent on every turn.

Manage skills under **Skills** in the Flow sidebar (same admin surface family as **Agents** and **Knowledge Graph**). The skill editor shows which agents currently use each package.

#### Tools during a skill session

An agent's baseline tool allowlist is configured in **Admin → Agents**. When a skill session is **active**, Fontana temporarily unions that agent's manual tools with the skill's **allowed-tools** list, so specialised playbooks can unlock canvas, workflow, or data tools only for the duration of the work. See **[Tools and MCP](/docs/ai/tools-and-mcp/)** for the full catalog.

Multiple skill sessions can be active concurrently; effective tools are the union of the agent baseline and all open skill sessions. When a session closes, its extra tools drop out of the next turn.

#### Skills and the Knowledge Graph

Skills and the Knowledge Graph solve different problems and stay separate in Fontana:

|                 | **Skills**                                    | **Knowledge Graph**                                                           |
| --------------- | --------------------------------------------- | ----------------------------------------------------------------------------- |
| **Purpose**     | Repeatable _procedures_ and playbooks         | Governed _operational knowledge_ and reference articles                       |
| **Content**     | `SKILL.md` packages with activation lifecycle | Namespace-scoped documents with version history                               |
| **Retrieval**   | Catalog plus explicit activation              | Automatic pre-turn injection plus [agentic search](/docs/ai/knowledge-graph/) |
| **Typical use** | "How to build this canvas pattern"            | "What our NAV approval rules say"                                             |

Assign both to the same agent when it needs firm knowledge _and_ structured runbooks for complex tasks.


# Tools and MCP
## Description: Built-in agent tools plus Model Context Protocol integrations and isolated OpenSandbox execution.
## url: https://fontana-ai.com/docs/ai/tools-and-mcp/

**Tools** are how [agents](/docs/ai/agents/) act on your workspace: query data, edit workflows, search the [Knowledge Graph](/docs/ai/knowledge-graph/), run canvas changes, call external systems, and complete governed procedures. Fontana exposes one **unified catalog** of built-in capabilities and **Model Context Protocol (MCP)** integrations. You assign tools per agent in **Admin → Agents**; the model never receives open-ended access to your estate. Every invocation is audited for compliance review.

<div class="md:w-1/2">
  <McpDiagram />
</div>

<Aside type="note" title="What is MCP?">

An open protocol for connecting AI applications to data sources, tools, and workflows ([modelcontextprotocol.io](https://modelcontextprotocol.io/)). In Fontana, agents invoke allowlisted MCP tools over HTTP or SSE. Hosted OAuth completes in the browser; bearer tokens and client credentials stay write-only in Vault. MCP connects agents to **tools and context**; peer-agent delegation is **[Agent Interop](/docs/ai/agent-interop/)**, a separate integration model.

</Aside>

#### Built-in tools

Built-in tools run **server-side inside your workspace** (Convex tenant). They do not require an external MCP server. You allowlist them per agent; Fontana executes them on the server and returns results into the same chat thread.

Typical built-in capabilities include:

| Area                   | What agents can do                                                                                                                                   |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Workflows and data** | List resources, query workflow configuration, inspect run datasets, and start workflow runs                                                          |
| **Knowledge Graph**    | Search governed corpora, fetch documents by slug, and manage agent memory where permitted                                                            |
| **Canvas**             | Query and mutate workflow graphs (nodes, edges, ports, and canvas config) through typed canvas tools, often via [Skills](/docs/ai/skills/) playbooks |
| **Documents**          | Create, read, update, and rename workflow documents (specs, design notes, reports) tied to a workflow                                                |
| **Chat orchestration** | Ask humans structured questions ([Human in the Loop](/docs/ai/human-in-the-loop/)), manage task lists, delegate to sub-agents, and hand off threads                                                     |
| **Skills runtime**     | Activate and close skill sessions and fetch bundled skill reference files                                                                            |

Built-in tools are the default surface for day-to-day Flow work. Prefer them when the capability already exists in Fontana so execution stays in-tenant, audited, and aligned with RBAC.

#### MCP connections

Use **MCP** when an agent must reach an **external system** you operate: SaaS APIs, databases, search services, calculators, or firm-specific HTTP tool servers.

In **Admin → Tools**, you register MCP connections (HTTP or SSE transport), test connectivity, and refresh the discovered tool list. Approved tools appear in the agent catalog under namespaced ids (`mcp_<connectionSlug>_<toolName>`). You can assign individual tools or a connection wildcard (`mcp_<connectionSlug>.*`) that expands against the cached snapshot at runtime.

| Topic               | What you should know                                                                                                                                  |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Authentication**  | Bearer tokens and OAuth app credentials (including BYOK Client ID/Secret) are stored write-only in Vault and used only during discovery and execution |
| **OAuth providers** | Hosted flows (for example Notion-style MCP URLs) use browser OAuth with PKCE; redirect URIs are shown in the admin UI                                 |
| **Discovery**       | Tool lists refresh when you test a connection; assign tools only after a successful refresh                                                           |
| **Transport**       | Remote HTTP/SSE only; local stdio MCP servers cannot run from Convex actions                                                                          |
| **Governance**      | Agent assignment is the approval boundary in V1; unassigned MCP tools are not callable                                                                |

Fontana acts as an **MCP client**: your agents call external tools; Fontana does not expose Flow as an MCP server for third-party clients in this integration path.

#### Configure tools for agents

Tool governance spans three admin surfaces that work together:

1. **Admin → Tools** - register MCP connections, inspect schemas, refresh discovery, and manage credentials
2. **Admin → Agents** - allowlist built-in and MCP tools (or wildcards) for each persona
3. **Admin → Skills** - optional temporary tool expansion while a skill session is active; see **[Skills](/docs/ai/skills/)**

[Skills](/docs/ai/skills/) can union additional tools for the duration of an active playbook without permanently widening an agent's baseline allowlist.

#### OpenSandbox (isolated code execution)

When agents need to run commands, scripts, or exploratory code safely, **OpenSandbox** provides **isolated execution** on the platform cluster. Each workspace reaches OpenSandbox through a **single shared MCP connection** seeded at deploy time; sandboxes are ephemeral, TTL-bound, and scoped with per-workspace bearer authentication.

<Aside type="note" title="What is OpenSandbox?">

Shared platform-cluster service for isolated agent code execution. Agents reach it via MCP; each workspace uses its own bearer token; sandboxes are ephemeral pods with no cross-tenant persistence.

</Aside>

**Agent (MCP)** → **OpenSandbox platform** → **Ephemeral sandbox pod**

Exploratory compute stays separated from your tenant data plane while remaining available to allowlisted agents through the same audited tool pipeline as other MCP calls.

#### Audit and review

Every built-in and MCP tool invocation writes to **tool-call audit records** with protocol metadata, latency, and outcomes. Operators with appropriate permissions can review agent tool usage in Flow admin surfaces and correlate activity with the immutable audit trail where your deployment enables it. See **[Observability](/docs/observability/)**, **[Compliance evidence](/docs/compliance/)**, and **[Security](/docs/security/)** (OpenSandbox controls in **[Workspace isolation](/docs/security/workspace-isolation/)**).

#### Tools, MCP, and Agent Interop

| Integration        | Connects agents to                                                   | Configure in                                              |
| ------------------ | -------------------------------------------------------------------- | --------------------------------------------------------- |
| **Built-in tools** | Fontana workflows, data, canvas, Knowledge Graph, chat orchestration | **Admin → Agents** (+ **Skills** for temporary expansion) |
| **MCP**            | External tool and context servers (HTTP/SSE)                         | **Admin → Tools** + **Admin → Agents**                    |
| **Agent Interop**  | Peer agents on other platforms (A2A, ACP, and related protocols)     | **Admin → Agent Interop**                                 |

Do not confuse MCP tool servers with peer-agent federation. For delegation and handoff between agents, see **[Agent Interop](/docs/ai/agent-interop/)**.


# Agent Interop
## Description: Connect Fontana agents to peer-agent ecosystems via A2A, ACP, and in-app delegation with full audit coverage.
## url: https://fontana-ai.com/docs/ai/agent-interop/

In **Admin → Agent Interop**, you register external agent connections and control which protocols are in use. Delegate work to specialist agents inside Fontana, hand off conversations when the task changes, or federate with external platforms over open protocols (**A2A**, **ACP**, **ANP**, **OFP**). Every delegation runs with allowlisted origins, bearer authentication, and audit metadata.

<Aside type="note" title="What is Agent Interop?">

Peer-agent federation distinct from MCP tools and in-app handoff/sub-agents. Fontana registers external agent connections, enforces allowlisted origins and bearer auth, and records delegation in the audit trail.

</Aside>

#### In-app orchestration patterns

<div
  class="not-content grid items-stretch gap-4 sm:grid-cols-2 lg:grid-cols-3"
  role="group"
  aria-label="Agent orchestration patterns: sub-agent delegation, handoff, and interop"
>
  <SubagentDelegationDiagram class="h-full" />
  <AgentHandoffDiagram class="h-full" />
  <AgentInteropDiagram class="h-full" />
</div>

| Pattern                 | Status    | What it does                                                                                                                        | Configuration                                                 |
| ----------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **SubAgent delegation** | Available | Spawn a child thread for delegated work, then return results to the parent. Isolates deep work while the parent keeps context lean. | `subagents` on the agent definition in **Admin → Agents**     |
| **Agent handoff**       | Available | Transfer the current conversation to another in-app agent on the same thread. Use when the task surface changes.                    | `handoffAgents` on the agent definition in **Admin → Agents** |

#### External protocols

Configure connections in **Admin → Agent Interop**. Supported and planned federation protocols:

<Aside type="note" title="What is A2A?">

JSON-RPC 2.0 over HTTPS for delegating tasks between agents. Fontana supports outbound delegation to registered connections, public Agent Card discovery, and bearer-authenticated inbound JSON-RPC.

</Aside>

<Aside type="note" title="What is ACP?">

REST-native agent messaging with MIME-typed multipart payloads, session identifiers, and manifest-based discovery. Fontana persists ACP connections alongside A2A for enterprises standardising on IBM/BeeAI-style agent buses.

</Aside>

| Protocol                               | Status      | Technical description                                                                                                                                                                                                                                                                                                         | Documentation                                                   |
| -------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| **A2A** (Agent2Agent)                  | Available   | JSON-RPC 2.0 over HTTPS for peer-agent task delegation. Outbound: in-app agents delegate tasks to registered connections with allowlisted origins and bearer auth (env or Vault). Inbound: workspace Agent Card at `/.well-known/agent-card.json`, per-agent cards, and bearer-authenticated JSON-RPC at `/api/a2a/json-rpc`. | [A2A specification](https://a2a-protocol.org/latest/)           |
| **ACP** (Agent Communication Protocol) | Available   | REST-native agent messaging with MIME-typed multipart payloads, session identifiers, and manifest-based discovery.                                                                                                                                                                                                            | [ACP specification](https://agentcommunicationprotocol.dev/)    |
| **ANP** (Agent Network Protocol)       | Coming soon | Decentralised peer-to-peer agent networking using W3C Decentralized Identifiers (DIDs) and JSON-LD capability metadata. Complements enterprise A2A when agents must federate across organisational boundaries without a central registry.                                                                                     | [ANP repository](https://github.com/agent-network-protocol/ANP) |
| **OFP** (Open Floor Protocol)          | Coming soon | Linux Foundation standard for multi-party conversational agents: Conversation Envelopes, Assistant Manifests, and Dialog Events. Supports delegation, mediation, and shared floor control across heterogeneous voice and chat agents.                                                                                         | [OFP introduction](https://openfloor.dev/introduction)          |

<Aside type="note" title="What is ANP?">

Decentralised peer-to-peer agent networking using W3C Decentralized Identifiers (DIDs) and JSON-LD capability metadata. Complements enterprise A2A when agents must federate across organisational boundaries without a central registry.

</Aside>

<Aside type="note" title="What is OFP?">

Linux Foundation standard for multi-party conversational agents: Conversation Envelopes, Assistant Manifests, and Dialog Events. Supports delegation, mediation, and shared floor control across heterogeneous voice and chat agents.

</Aside>

#### Inbound and outbound behaviour

- **Outbound delegation** - in-app agents send A2A JSON-RPC tasks to registered peer connections with allowlisted origins and bearer authentication
- **Inbound discovery** - workspace publishes an Agent Card at `/.well-known/agent-card.json` plus per-agent cards
- **Inbound API** - bearer-authenticated JSON-RPC at `/api/a2a/json-rpc` for inbound task execution
- **Admin → Agent Interop** - connection CRUD, allowlisted HTTPS origins, audit metadata on tool calls

#### Related documentation

- **[Agents](/docs/ai/agents/)** - in-app personas, subagents, and handoff configuration
- **[Tools and MCP](/docs/ai/tools-and-mcp/)** - MCP integrations distinct from peer-agent federation
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** - delegation and tool-call audit evidence


# Bring Your Own Key (BYOK)
## Description: Connect Fontana to your approved LLM gateways with API keys you control, scoped in per-workspace Vault at org, team, or person level.
## url: https://fontana-ai.com/docs/ai/byok/

**Bring Your Own Key (BYOK)** is how you connect Fontana to LLM gateways and model providers using credentials **your organisation already holds**. Fontana does not operate a shared vendor LLM account for your workspace.

Your keys are stored securely in a **HashiCorp Vault**. You can approve which gateways and models are reachable in production, ensuring that chat, agent, workflow AI, and embedding calls are routed through **your** contracts—so spend, logging, retention, and training policy always remain under your control.

<Aside type="note" title="What is BYOK?">

Your API keys for approved LLM gateways, stored write-only in Vault and resolved at runtime. BYOK is for model inference and embeddings, not for connector credentials or MCP OAuth tokens (those use separate Vault paths).

</Aside>

#### Why BYOK matters

BYOK keeps AI spend and data policy aligned with procurement you already run:

- **No shared Fontana model pool** - inference uses keys and gateways you provision
- **Contractual control** - retention, training, and regional routing follow your provider agreements; see **[Zero Data Retention (ZDR)](/docs/ai/zdr/)** where supported
- **Cost attribution** - scope keys at org, team, or person level for clearer chargeback
- **Fail-closed secret handling** - keys are never returned to the browser or embedded in workflow config

After keys are set, choose **which gateways and models** are allowed in **[Gateways](/docs/ai/gateways/)** and **[Models](/docs/ai/models/)**. BYOK supplies credentials; those pages govern routing and the production model catalogue.

#### Key storage and scopes

BYOK secret values live in Vault KV mount `fontana` only. They are **write-only** from **Admin → Models → API Keys**: the UI shows whether a key is set, when it was updated, and who set it, but never returns the raw value to the browser.

You can scope each provider key at three levels:

| Scope            | Use when                                                         |
| ---------------- | ---------------------------------------------------------------- |
| **Org (global)** | One deployment-wide key shared by all teams                      |
| **Team**         | A team-specific key for cost attribution or provider segregation |
| **Person**       | An individual user's key for personal or experimental use        |

At runtime, Fontana resolves keys in order **person → team (when a team context applies) → org**. The first match wins, so narrower scopes override broader ones without shadow usage. Updates in Admin are visible on the next model call without restarting workspace pods.

#### Configure in Admin

Open **Admin → Models → API Keys** to set, verify, and clear keys per provider. Use the scope selector to choose org, team, or person before editing a provider card.

| Step                   | What you do                                                                                                               |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **Set a key**          | Paste or rotate the provider credential; Fontana writes it to Vault only                                                  |
| **Verify**             | Run the built-in check against the provider's authenticated endpoint; the card shows Verified, Invalid key, or Unverified |
| **Approve models**     | Move to **Models → LLMs** to choose which model ids are reachable in production                                           |
| **Embedding priority** | On **Models → Embeddings**, configure gateway fallback order (default OpenRouter → OpenAI → LLM API)                      |

**Permissions:** viewing Models requires `platform:read` or `ai:page`. Setting or clearing keys requires `platform:write`; team-scoped keys also require `teams:admin` for the selected team.

Supported providers include OpenAI, OpenRouter, LLM API, Anthropic, Groq, Google Generative AI, Mistral, NVIDIA, AWS Bedrock (three-field credentials), and additional gateways on **[Gateways](/docs/ai/gateways/)**.

#### What BYOK powers

BYOK credentials feed every governed LLM path in Fontana:

| Surface                                          | How BYOK is used                                                       |
| ------------------------------------------------ | ---------------------------------------------------------------------- |
| **[Agents](/docs/ai/agents/) and Flow chat**     | Model calls for conversation, tool use, and automation                 |
| **Workflow AI**                                  | Natural-language operation compile and AI-assisted canvas steps        |
| **[Knowledge Graph](/docs/ai/knowledge-graph/)** | Embedding queries and index maintenance through the same gateway stack |

Gateway choice, model catalogue approval, and ZDR posture are configured separately on **[Gateways](/docs/ai/gateways/)**, **[Models](/docs/ai/models/)**, and **[Zero Data Retention (ZDR)](/docs/ai/zdr/)**.

#### Distinct from connector secrets

BYOK keys are for **LLM gateways only**. Connector credentials, MCP OAuth tokens, and REST API secrets use a different Vault resolution path and admin surfaces (`{{secret.NAME}}` placeholders in workflow node config). See **[Data](/docs/data/)** for connector secret handling and **[Tools and MCP](/docs/ai/tools-and-mcp/)** for MCP credential storage.

#### Audit and backup

BYOK configuration changes are in scope for the immutable security audit trail. This allows tracking any key changes within your org.


# Zero Data Retention (ZDR)
## Description: Align LLM routing with your retention and no-training requirements through provider ZDR policies and your BYOK gateway settings.
## url: https://fontana-ai.com/docs/ai/zdr/

**Zero Data Retention (ZDR)** is a provider policy not to store prompt or completion payload after the request completes. In Fontana, ZDR posture comes from the **gateway and [BYOK](/docs/ai/byok/) account you configure** (for example OpenRouter privacy toggles or direct vendor enterprise terms), not from a Fontana-operated default.

Fontana supports ZDR on every gateway where the upstream provider exposes it, so you can align chat and completion routing with your retention and training requirements. Posture comes from your BYOK contract and, for unified gateways, the provider's own routing and policy controls. See **[Gateways](/docs/ai/gateways/)** for the approved gateway catalogue.

#### OpenRouter ZDR enforcement

When chat routes through **OpenRouter**, ZDR is governed by [OpenRouter's ZDR policy](https://openrouter.ai/docs/guides/features/zdr) on your account. OpenRouter tracks per-endpoint data policies (retention and training) and can restrict routing to ZDR-only endpoints at account, guardrail, or request scope. Fontana traffic uses your OpenRouter API key, so account-wide and guardrail ZDR settings apply to all calls through that key.

| Model group (OpenRouter) | Effect when ZDR is enabled for that group                                                       |
| ------------------------ | ----------------------------------------------------------------------------------------------- |
| **Anthropic**            | Removes first-party Anthropic endpoints (AWS Bedrock and Google Vertex routes remain available) |
| **OpenAI**               | Removes first-party OpenAI endpoints (Azure routes remain available)                            |
| **Google**               | Removes AI Studio endpoints (Vertex routes remain available)                                    |
| **Non-frontier**         | Removes all other non-ZDR endpoints                                                             |

- **Account-level:** each model-group toggle in [OpenRouter privacy settings](https://openrouter.ai/settings/privacy) restricts all requests on that account to ZDR endpoints for that group.
- **Guardrail-level:** separate fields per group - `enforce_zdr_anthropic`, `enforce_zdr_openai`, `enforce_zdr_google`, `enforce_zdr_other` - so different API keys or org members can carry different ZDR posture.
- **Per-request:** `provider: {"{ zdr: true }"}` in the API body ORs with account and guardrail settings (it can only tighten routing, not override ZDR off when account policy requires it). Useful for one-off ZDR-only calls without a global toggle.
- **Endpoint list:** ZDR-capable endpoints are published at [`openrouter.ai/api/v1/endpoints/zdr`](https://openrouter.ai/api/v1/endpoints/zdr) and updated when provider policies change. OpenRouter treats unknown endpoint policies conservatively (assumed retain + train).
- **Caching:** in-memory prompt caching on the provider side is not classified as retention for ZDR routing.
- **OpenRouter itself:** OpenRouter does not retain your prompts unless you opt in to prompt logging.

#### Direct gateways and ZDR

When you route through a **direct** gateway (not OpenRouter), ZDR and no-training posture comes from that provider's API and your enterprise agreement:

- **Anthropic, OpenAI, Google, Mistral, AWS Bedrock** - direct API calls with your keys; zero-retention and no-training modes depend on the product tier and account settings you hold with that vendor.
- **LLM API** and other OpenAI-compatible hosts - follow the host's published data policy for the key you configure.
- **Operational control in Fontana** - admins approve which gateway and model IDs are reachable in production; combined with BYOK, teams can standardise on direct enterprise endpoints or OpenRouter ZDR-only routing without shadow usage.

#### Provider vs platform logging

Upstream LLM providers apply ZDR and retention under your BYOK contract and gateway settings above. Separately, Fontana records token usage, latency, and routing metadata for cost and compliance. Optional in-platform AI audit capture (admin audit logs, `ai_tool_calls` records, and permission-gated LLM debug) can retain prompt and completion excerpts inside your deployment; scope those surfaces to investigators who need them, or rely on metadata-only logging when message bodies must not be stored.


# Workflow orchestration
## Description: Visual pipeline model, execution engine, node families, operations, and document processing.
## url: https://fontana-ai.com/docs/workflows/

On the **Flow** canvas, you design visual pipelines that move data from source to outcome. The **workflow engine** runs them reactively on the server: when inputs or configuration change, affected nodes re-run while independent branches execute in parallel. You reuse work through templates and subgraphs, and every run can leave **audit and lineage metadata** for compliance review. See **[Compliance evidence](/docs/compliance/)** for data lineage and **[Data](/docs/data/)** for connectors, storage, and file formats.

<Aside type="note" title="What is a workflow?">

A directed graph of **nodes** connected by **edges**. Each node performs one step and passes typed dataset ports downstream. Runs are server-side when files, connectors, or document parsing require the workflow engine.

</Aside>

#### Visual pipeline model

A **workflow** is a directed graph of **nodes** connected by **edges**. Each node performs one step and passes typed dataset ports downstream.

<WorkflowPipelineDiagram />

#### Execution engine

When you trigger a run, the engine:

- **Reacts to change** - hash-driven scheduling re-runs nodes when inputs or configuration change
- **Runs in parallel** - independent branches execute concurrently where the graph allows
- **Executes server-side** - large files, PDF parsing, connector fetch, and PyAirbyte jobs run on the workflow engine
- **Persists run-scoped data** - uploads, Arrow datasets, exports, and audit sidecars survive pod restarts within your workspace boundary

<ValidationsDiagram />

<TransformationsDiagram />

<ComputeDiagram />

#### Node families

The sidebar under **Workflows → Nodes** documents each node type. Representative families:

<DocsNodeFamiliesTable client:only="react" />

#### Operations and validations

The **Operations** node exposes column-level transforms (strings, dates, decimals, conditionals, lookups). You combine visual operation stacks in the UI or use **natural-language compile** ([Governed AI](/docs/ai/)) to generate operations from plain English.

The sidebar catalogues **Transformations**, **Validations**, and **Conditions** you attach to Operations nodes. Use them to enforce data quality before data leaves a pipeline branch.

#### Document processing

PDF and office documents uploaded to **File Input** are parsed server-side via **Docling** on the platform cluster (shared document-parse service for all workspaces). Optional AI structuring enriches results for review in the UI.

#### Subgraphs and templates

- Save workflows as **templates** and share them within teams or the community catalog
- Encapsulate reusable fragments as **subgraphs** and graft them onto larger pipelines

#### Related documentation

- **[Security](/docs/security/)** - platform security controls and SOC 2 summary
- **[Compliance evidence](/docs/compliance/)** - atomic data lineage and workflow file store
- **[Data](/docs/data/)** - File Input, PyAirbyte connectors, Arrow storage, REST triggers, exports
- **[Governed AI](/docs/ai/)** - AI transforms and natural-language operation compile
- **[Observability](/docs/observability/)** - run telemetry; canvas port audit items vs security audit ledger


# AI Node
## Description: Processes data with AI/LLM models. Supports entry-by-entry or whole-payload mode.
## url: https://fontana-ai.com/docs/workflows/nodes/ai/

## Overview

- **Template variables** – Use `{row}`, `{row.property}`, `{rowIndex}`, `{lookups}`, `{numRows}` in prompts
- **Entry mode** – One AI call per row
- **Payload mode** – Single AI call for entire dataset
- **Structured output** – Optional JSON schema for structured responses

## Configuration

```json
{
  "custom": {
    "execution": {
      "prompt": "Summarize: {{ row.description }}",
      "mode": "entry",
      "model": "openai/gpt-4o"
    }
  }
}
```

## Related

- [jinja](/docs/workflows/nodes/jinja/) – Template-based transformation
- [compute](/docs/workflows/nodes/compute/) – Custom JavaScript
- [Workflow schema](/docs/workflows/) – Full schema reference


# Compute Node
## Description: Executes custom JavaScript functions that can return multiple entries per input (1-to-many).
## url: https://fontana-ai.com/docs/workflows/nodes/compute/

## Overview

- **Function operations** – Only `function` operations are supported
- **1-to-many** – Functions can return arrays to expand or filter rows
- **Stateful context** – Context persists across entries within a function operation
- **Lookup data** – Access lookup payloads for enrichment

## Configuration

```json
{
  "custom": {
    "execution": {
      "operations": [
        {
          "type": "transformation",
          "action": "function",
          "params": {
            "function": "({ row }) => row.quantity > 0 ? [row] : []"
          }
        }
      ]
    }
  }
}
```

## Related

- [operation](/docs/workflows/nodes/operation/) – Transformations and validations
- [filter](/docs/workflows/nodes/filter/) – Remove invalid rows
- [function](/docs/workflows/operations/transformation/general/function/) – Function transformation


# File Input Node
## Description: Loads data from files (CSV, JSON, etc.).
## url: https://fontana-ai.com/docs/workflows/nodes/fileInput/

## Overview

- Supports CSV, JSON, and other formats
- Configurable import settings (delimiter, encoding, headers)
- PDF text extraction modes

## Related

- [Workflow schema](/docs/workflows/) – Full schema reference


# Filter Node
## Description: The filter node removes rows that fail validations. It uses the same operations as the operation node.
## url: https://fontana-ai.com/docs/workflows/nodes/filter/

## Overview

- Applies validation operations to each row
- Rows that fail are excluded from the output
- Same operation set as [operation](/docs/workflows/nodes/operation/)

## Configuration

```json
{
  "custom": {
    "execution": {
      "operations": [
        {
          "type": "validation",
          "action": "required",
          "params": {}
        }
      ]
    }
  }
}
```

## Related

- [operation](/docs/workflows/nodes/operation/) – Same operations, but transforms data instead of filtering
- [lessThan](/docs/workflows/operations/validation/number/lessThan/) – Example validation


# Jinja Node
## Description: Transforms data using Jinja templates. Supports entry-by-entry or whole-payload mode.
## url: https://fontana-ai.com/docs/workflows/nodes/jinja/

## Overview

- **Template engine** – Jinja2 syntax for string formatting
- **Entry mode** – Apply template to each row with `{row}`, `{rowIndex}`, `{lookups}`, `{numRows}`
- **Payload mode** – Apply template to entire dataset with `{rows}`, `{lookups}`, `{numRows}`
- **JSON output** – Rendered output is parsed as JSON when possible

## Configuration

```json
{
  "custom": {
    "execution": {
      "template": "{{ firstName }} {{ lastName }}",
      "mode": "entry"
    }
  }
}
```

## Related

- [operation](/docs/workflows/nodes/operation/) – Built-in transformations
- [compute](/docs/workflows/nodes/compute/) – Custom JavaScript functions
- [Workflow schema](/docs/workflows/) – Full schema reference


# Operation Node
## Description: The operation node applies transformations and validations to data rows.
## url: https://fontana-ai.com/docs/workflows/nodes/operation/

## Overview

- **Transformations** – Modify field values (e.g. uppercase, number, template)
- **Validations** – Check values and optionally reject rows (e.g. required, lessThan)

Operations are executed in order. Each operation can have a source field and target field.

## Configuration

See [Workflow schema](/docs/workflows/) for the full schema. The operation node uses:

```json
{
  "custom": {
    "execution": {
      "operations": [
        {
          "type": "transformation",
          "action": "uppercase",
          "params": {}
        }
      ]
    }
  }
}
```

## Related

- [filter](/docs/workflows/nodes/filter/) – Uses the same operations to filter rows
- [lessThan](/docs/workflows/operations/validation/number/lessThan/) – Example validation


# Schema Node
## Description: Defines data structure with JSON Schema. Used for validation and transformation.
## url: https://fontana-ai.com/docs/workflows/nodes/schema/

## Overview

- **outputSchema** – JSON Schema defining the expected structure
- **validationEnabled** – Whether to validate incoming data
- **transformData** – Whether to transform data to match schema

## Related

- [Workflow schema](/docs/workflows/) – Full schema reference
- [operation](/docs/workflows/nodes/operation/) – Uses schema for field detection


# Add
## Description: Add source field and operand.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/add/

<OperationReference type="transformation" action="add" />


# Divide
## Description: Divide source field by operand.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/divide/

<OperationReference type="transformation" action="divide" />


# Modulo
## Description: Compute remainder of source field divided by operand (sign follows dividend).
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/modulo/

<OperationReference type="transformation" action="modulo" />


# Multiply
## Description: Multiply source field by operand.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/multiply/

<OperationReference type="transformation" action="multiply" />


# Power
## Description: Raise source field to the power of operand.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/power/

<OperationReference type="transformation" action="power" />


# Subtract
## Description: Subtract operand from source field.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/arithmetic/subtract/

<OperationReference type="transformation" action="subtract" />


# Date
## Description: Convert date between formats.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/date/

<OperationReference type="transformation" action="date" />


# Date Add
## Description: Add a duration to a date field.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateAdd/

<OperationReference type="transformation" action="dateAdd" />


# Date Diff
## Description: Compute difference between two dates as a number.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateDiff/

<OperationReference type="transformation" action="dateDiff" />


# Date End Of
## Description: Set date to the end of a calendar period.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateEndOf/

<OperationReference type="transformation" action="dateEndOf" />


# Date Now
## Description: Write the current date/time to the target field.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateNow/

<OperationReference type="transformation" action="dateNow" />


# Date Start Of
## Description: Set date to the start of a calendar period.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateStartOf/

<OperationReference type="transformation" action="dateStartOf" />


# Date Subtract
## Description: Subtract a duration from a date field.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/date-and-time/dateSubtract/

<OperationReference type="transformation" action="dateSubtract" />


# Function
## Description: Custom JavaScript transformation function that returns an updated row with optional lineage and audit metadata.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/general/function/

<OperationReference type="transformation" action="function" />


# Swap
## Description: Swap values between two fields in one step (uses Source as field A and Target as field B; supports nested paths like parent.child).
## url: https://fontana-ai.com/docs/workflows/operations/transformation/general/swap/

<OperationReference type="transformation" action="swap" />


# Template
## Description: Create value using a template.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/general/template/

<OperationReference type="transformation" action="template" />


# Absolute Value
## Description: Get absolute value of number.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/abs/

<OperationReference type="transformation" action="abs" />


# Ceil
## Description: Round a number up to N decimal places.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/ceil/

<OperationReference type="transformation" action="ceil" />


# Floor
## Description: Round a number down to N decimal places.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/floor/

<OperationReference type="transformation" action="floor" />


# Floor Number
## Description: Round a number down to N decimal places (throws error if not a number).
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/floorNumber/

<OperationReference type="transformation" action="floorNumber" />


# Negate
## Description: Negate (flip the sign of) a number.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/negate/

<OperationReference type="transformation" action="negate" />


# Number
## Description: Convert value to number.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/number/

<OperationReference type="transformation" action="number" />


# Number Strict
## Description: Convert value to number (throws error if not a number).
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/numberStrict/

<OperationReference type="transformation" action="numberStrict" />


# Parse float
## Description: Parse string to float.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/parseFloat/

<OperationReference type="transformation" action="parseFloat" />


# Parse integer
## Description: Parse string to integer.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/parseInt/

<OperationReference type="transformation" action="parseInt" />


# Round
## Description: Round a number to N decimal places.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/round/

<OperationReference type="transformation" action="round" />


# Sqrt
## Description: Take the square root of a number.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/sqrt/

<OperationReference type="transformation" action="sqrt" />


# Trunc
## Description: Truncate a number toward zero to N decimal places.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/number/trunc/

<OperationReference type="transformation" action="trunc" />


# Concat
## Description: Concatenate multiple fields with delimiter.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/concat/

<OperationReference type="transformation" action="concat" />


# Lowercase
## Description: Convert text to lowercase.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/lowercase/

<OperationReference type="transformation" action="lowercase" />


# Replace
## Description: Replace pattern in text with flexible options.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/replace/

<OperationReference type="transformation" action="replace" />


# Substring
## Description: Extract substring from text with start and optional end positions.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/substring/

<OperationReference type="transformation" action="substring" />


# Title Case
## Description: Convert text to title case (first letter of each word capitalized).
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/titleCase/

<OperationReference type="transformation" action="titleCase" />


# Trim
## Description: Remove leading and trailing whitespace.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/trim/

<OperationReference type="transformation" action="trim" />


# Uppercase
## Description: Convert text to uppercase.
## url: https://fontana-ai.com/docs/workflows/operations/transformation/string/uppercase/

<OperationReference type="transformation" action="uppercase" />


# Array contains
## Description: Ensure array field contains specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayContains/

<OperationReference type="validation" action="arrayContains" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array does not contain
## Description: Ensure array field does not contain specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayDoesNotContain/

<OperationReference type="validation" action="arrayDoesNotContain" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array does not exist
## Description: Ensure array field does not exist.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayDoesNotExist/

<OperationReference type="validation" action="arrayDoesNotExist" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array is empty
## Description: Ensure array field is empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayIsEmpty/

<OperationReference type="validation" action="arrayIsEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array is not empty
## Description: Ensure array field is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayIsNotEmpty/

<OperationReference type="validation" action="arrayIsNotEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length equal to
## Description: Ensure array field has specified length.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthEqualTo/

<OperationReference type="validation" action="arrayLengthEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length greater than
## Description: Ensure array field length is greater than specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthGreaterThan/

<OperationReference type="validation" action="arrayLengthGreaterThan" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length greater than or equal
## Description: Ensure array field length is greater than or equal to specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthGreaterThanOrEqual/

<OperationReference type="validation" action="arrayLengthGreaterThanOrEqual" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length less than
## Description: Ensure array field length is less than specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthLessThan/

<OperationReference type="validation" action="arrayLengthLessThan" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length less than or equal
## Description: Ensure array field length is less than or equal to specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthLessThanOrEqual/

<OperationReference type="validation" action="arrayLengthLessThanOrEqual" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Array length not equal to
## Description: Ensure array field does not have specified length.
## url: https://fontana-ai.com/docs/workflows/operations/validation/array/arrayLengthNotEqualTo/

<OperationReference type="validation" action="arrayLengthNotEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean does not exist
## Description: Ensure boolean field does not exist.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanDoesNotExist/

<OperationReference type="validation" action="booleanDoesNotExist" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean equal to
## Description: Ensure boolean field equals specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanEqualTo/

<OperationReference type="validation" action="booleanEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean is empty
## Description: Ensure boolean field is empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanIsEmpty/

<OperationReference type="validation" action="booleanIsEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean is false
## Description: Ensure boolean field is false.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanIsFalse/

<OperationReference type="validation" action="booleanIsFalse" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean is not empty
## Description: Ensure boolean field is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanIsNotEmpty/

<OperationReference type="validation" action="booleanIsNotEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean is true
## Description: Ensure boolean field is true.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanIsTrue/

<OperationReference type="validation" action="booleanIsTrue" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Boolean not equal to
## Description: Ensure boolean field does not equal specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/boolean/booleanNotEqualTo/

<OperationReference type="validation" action="booleanNotEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Is date
## Description: Validate date format.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/date/

<OperationReference type="validation" action="date" />


# Date equal to
## Description: Ensure date field equals specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateEqualTo/

<OperationReference type="validation" action="dateEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Date is after
## Description: Ensure date field is after specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateIsAfter/

<OperationReference type="validation" action="dateIsAfter" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Date is after or equal
## Description: Ensure date field is after or equal to specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateIsAfterOrEqual/

<OperationReference type="validation" action="dateIsAfterOrEqual" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Date is before
## Description: Ensure date field is before specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateIsBefore/

<OperationReference type="validation" action="dateIsBefore" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Date is before or equal
## Description: Ensure date field is before or equal to specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateIsBeforeOrEqual/

<OperationReference type="validation" action="dateIsBeforeOrEqual" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Date not equal to
## Description: Ensure date field does not equal specified date.
## url: https://fontana-ai.com/docs/workflows/operations/validation/date-and-time/dateNotEqualTo/

<OperationReference type="validation" action="dateNotEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Custom function validation
## Description: Custom JavaScript validation function.
## url: https://fontana-ai.com/docs/workflows/operations/validation/general/custom/

<OperationReference type="validation" action="custom" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Function validation
## Description: Custom JavaScript validation function (returns row if valid, null if invalid).
## url: https://fontana-ai.com/docs/workflows/operations/validation/general/function/

<OperationReference type="validation" action="function" />


# Required
## Description: Ensure field exists and is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/general/required/

<OperationReference type="validation" action="required" />


# Between
## Description: Ensure value is between min and max.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/between/

<OperationReference type="validation" action="between" />


# Greater than
## Description: Ensure value is greater than threshold.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/greaterThan/

<OperationReference type="validation" action="greaterThan" />


# Greater than or equal
## Description: Ensure value is greater than or equal to threshold.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/greaterThanOrEqual/

<OperationReference type="validation" action="greaterThanOrEqual" />


# Is integer
## Description: Validate integer format.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/integer/

<OperationReference type="validation" action="integer" />


# Less than
## Description: Ensure value is less than threshold.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/lessThan/

<OperationReference type="validation" action="lessThan" />


# Less than or equal
## Description: Ensure value is less than or equal to threshold.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/lessThanOrEqual/

<OperationReference type="validation" action="lessThanOrEqual" />


# Maximum value
## Description: Ensure value is at most maximum.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/max/

<OperationReference type="validation" action="max" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Minimum value
## Description: Ensure value is at least minimum.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/min/

<OperationReference type="validation" action="min" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Is number
## Description: Validate number format.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/number/

<OperationReference type="validation" action="number" />


# Number does not exist
## Description: Ensure number field does not exist.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/numberDoesNotExist/

<OperationReference type="validation" action="numberDoesNotExist" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Number equal to
## Description: Ensure number field equals specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/numberEqualTo/

<OperationReference type="validation" action="numberEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Number is empty
## Description: Ensure number field is empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/numberIsEmpty/

<OperationReference type="validation" action="numberIsEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Number is not empty
## Description: Ensure number field is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/numberIsNotEmpty/

<OperationReference type="validation" action="numberIsNotEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Number not equal to
## Description: Ensure number field does not equal specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/number/numberNotEqualTo/

<OperationReference type="validation" action="numberNotEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Object does not exist
## Description: Ensure object field does not exist.
## url: https://fontana-ai.com/docs/workflows/operations/validation/object/objectDoesNotExist/

<OperationReference type="validation" action="objectDoesNotExist" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Object is empty
## Description: Ensure object field is empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/object/objectIsEmpty/

<OperationReference type="validation" action="objectIsEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Object is not empty
## Description: Ensure object field is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/object/objectIsNotEmpty/

<OperationReference type="validation" action="objectIsNotEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Is email
## Description: Validate email format.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/email/

<OperationReference type="validation" action="email" />


# Is lowercase
## Description: Ensure string field is lowercase.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/isLowercase/

<OperationReference type="validation" action="isLowercase" />

## Related
- [lowercase](/docs/workflows/operations/transformation/string/lowercase/) – Transform to lowercase before validation


# Is uppercase
## Description: Ensure string field is uppercase.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/isUppercase/

<OperationReference type="validation" action="isUppercase" />

## Related
- [uppercase](/docs/workflows/operations/transformation/string/uppercase/) – Transform to uppercase before validation


# Maximum length
## Description: Ensure string does not exceed maximum length.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/maxLength/

<OperationReference type="validation" action="maxLength" />


# Minimum length
## Description: Ensure string has minimum length.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/minLength/

<OperationReference type="validation" action="minLength" />


# String contains
## Description: Ensure string field contains specified substring.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringContains/

<OperationReference type="validation" action="stringContains" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String does not contain
## Description: Ensure string field does not contain specified substring.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringDoesNotContain/

<OperationReference type="validation" action="stringDoesNotContain" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String does not end with
## Description: Ensure string field does not end with specified suffix.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringDoesNotEndWith/

<OperationReference type="validation" action="stringDoesNotEndWith" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String does not exist
## Description: Ensure string field does not exist.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringDoesNotExist/

<OperationReference type="validation" action="stringDoesNotExist" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String does not match regex
## Description: Ensure string field does not match regex pattern.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringDoesNotMatchRegex/

<OperationReference type="validation" action="stringDoesNotMatchRegex" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String does not start with
## Description: Ensure string field does not start with specified prefix.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringDoesNotStartWith/

<OperationReference type="validation" action="stringDoesNotStartWith" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String ends with
## Description: Ensure string field ends with specified suffix.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringEndsWith/

<OperationReference type="validation" action="stringEndsWith" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String equal to
## Description: Ensure string field equals specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringEqualTo/

<OperationReference type="validation" action="stringEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String is empty
## Description: Ensure string field is empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringIsEmpty/

<OperationReference type="validation" action="stringIsEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String is not empty
## Description: Ensure string field is not empty.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringIsNotEmpty/

<OperationReference type="validation" action="stringIsNotEmpty" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Matches regex pattern
## Description: Ensure string field matches regex pattern.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringMatchesRegex/

<OperationReference type="validation" action="stringMatchesRegex" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String not equal to
## Description: Ensure string field does not equal specified value.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringNotEqualTo/

<OperationReference type="validation" action="stringNotEqualTo" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# String starts with
## Description: Ensure string field starts with specified prefix.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/stringStartsWith/

<OperationReference type="validation" action="stringStartsWith" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# Is URL
## Description: Validate URL format.
## url: https://fontana-ai.com/docs/workflows/operations/validation/string/url/

<OperationReference type="validation" action="url" />

## Related
- [required](/docs/workflows/operations/validation/general/required/)


# AND
## Description: Logical AND: all nested operations must pass. Transformations count as true.
## url: https://fontana-ai.com/docs/workflows/operations/condition/and/

<OperationReference type="condition" action="and" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# FALSE
## Description: Always returns false.
## url: https://fontana-ai.com/docs/workflows/operations/condition/false/

<OperationReference type="condition" action="false" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# IF
## Description: Evaluate a predicate (AND). Run true or false branch operations per row.
## url: https://fontana-ai.com/docs/workflows/operations/condition/if/

<OperationReference type="condition" action="if" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# NOT
## Description: Logical NOT: negates the result of 1 nested operation. Transformations count as true.
## url: https://fontana-ai.com/docs/workflows/operations/condition/not/

<OperationReference type="condition" action="not" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# OR
## Description: Logical OR: at least one nested operation must pass. Transformations count as true.
## url: https://fontana-ai.com/docs/workflows/operations/condition/or/

<OperationReference type="condition" action="or" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# SWITCH
## Description: Evaluate the discriminant field (source) and run the operations of the first matching case. Falls back to default operations when no case matches.
## url: https://fontana-ai.com/docs/workflows/operations/condition/switch/

<OperationReference type="condition" action="switch" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# TRUE
## Description: Always returns true.
## url: https://fontana-ai.com/docs/workflows/operations/condition/true/

<OperationReference type="condition" action="true" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# XOR
## Description: Logical XOR: exactly one of 2 nested operations must pass. Transformations count as true.
## url: https://fontana-ai.com/docs/workflows/operations/condition/xor/

<OperationReference type="condition" action="xor" />

## Related
- [if](/docs/workflows/operations/condition/if/)


# Identity and access
## Description: Self-hosted Zitadel per workspace - OIDC, SSO, MFA, SCIM, and application RBAC in Flow.
## url: https://fontana-ai.com/docs/auth/

Each workspace runs a **self-hosted Zitadel** instance: the identity plane for that tenant. You get enterprise sign-in with every deployment without bolting on a separate IdP product. Zitadel handles authentication, federation, and directory integration; **Flow** and Convex enforce **application RBAC** on top so permissions match how your teams actually work.

#### Topics

- **[Identity providers](/docs/auth/identity-providers/)** - federate corporate SSO (OIDC, SAML, LDAP), SCIM provisioning, social logins, and custom IdPs through Zitadel as your workspace identity broker
- **[Roles and RBAC](/docs/auth/roles-and-rbac/)** - default roles, the permission catalog, custom roles in Admin, and how Zitadel role assignment maps into Flow

#### What you get

| Capability                            | What it means for you                                                                                                                  |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Single sign-on (SSO)**              | Users sign in with corporate IdP credentials through your per-workspace Zitadel plane                                                  |
| **OIDC**                              | Standard browser login for Flow with Authorization Code + PKCE; issuer, client, and redirect URIs provisioned per workspace            |
| **SAML federation**                   | Link upstream SAML identity providers for enterprise SSO alongside OIDC clients                                                        |
| **Multi-factor authentication (MFA)** | TOTP and WebAuthn (security keys / passkeys) enforced through Zitadel login policy                                                     |
| **SCIM provisioning**                 | Inbound SCIM v2 through your workspace **Zitadel** instance: automated user create, update, and deprovision from corporate directories |
| **Token introspection**               | Backend services (workflow engine, observability-api) validate opaque tokens via RFC 7662 using dedicated service credentials          |
| **Service authentication**            | Dedicated OAuth clients for workflow-engine, observability-api, and automation jobs; credentials rotated via Vault-backed provisioning |
| **Session lifecycle**                 | Standard OIDC logout and token refresh; Flow fails closed when discovery or introspection is misconfigured                             |

#### Identity events and audit

Zitadel identity events (sign-in, MFA, role changes, deprovisioning) feed the immutable security audit trail automatically through observability-api. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** and **[Observability](/docs/observability/)**.

#### Related documentation

- **[Security](/docs/security/)** - workspace isolation, secrets, encryption, and SOC 2 control summary
- **[Compliance evidence](/docs/compliance/)** - immutable audit trail and workflow lineage
- **[Deployment](/docs/deployment/)** - cluster-per-tenant topology; Zitadel provisioned per workspace
- **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** - distinct from identity; LLM credentials in Vault


# Identity providers
## Description: Federate corporate SSO, social logins, and custom IdPs through Zitadel as the identity broker for each workspace.
## url: https://fontana-ai.com/docs/auth/identity-providers/

An **identity provider (IdP)** creates and maintains identity information and provides authentication services to your applications. **Single sign-on (SSO)** lets users access multiple services with one set of credentials, which improves convenience and reduces password sprawl.

Each Fontana workspace runs **self-hosted Zitadel** as its identity plane. Zitadel can act as your primary user store, or as an **identity broker** that federates **external identity providers** (corporate Entra ID, Okta, Google, SAML IdPs, LDAP, and more). Your Flow application talks only to Zitadel; Zitadel handles protocol differences with upstream IdPs.

<Aside type="note" title="What is an external identity provider?">

An IdP outside your Fontana workspace that users already trust: Google, GitHub, LinkedIn, Entra ID, Okta, or a custom OIDC or SAML service. Users sign in with those credentials; Zitadel links or creates the workspace account on first login.

</Aside>

#### External identity providers and SSO

Many users already have accounts with popular IdPs or workplace directories. Allowing them to sign in with those accounts means they do not need new usernames and passwords for Flow, which improves adoption and reduces support load.

External IdPs also bring mature security controls (password policies, conditional access, device compliance) that your organisation may already enforce. Integrating them through Zitadel keeps that investment while Fontana stays a standard OIDC client.

#### Adding external identity providers

With Zitadel you can integrate:

- **Social logins** - Google, GitHub, Apple, and similar templates
- **Enterprise IdPs** - Entra ID (OIDC or SAML), Okta, Auth0, Keycloak
- **Directory services** - LDAP and OpenLDAP
- **Custom IdPs** - any provider that speaks OIDC or SAML via generic templates

By default, Zitadel can serve as the primary user store (email and password plus MFA). You can also require or allow external IdPs only, depending on login policy.

<Aside type="note" title="B2B: different IdPs per customer">

Zitadel supports **organisation-level** IdP configuration. In multi-customer deployments, one customer can use Entra ID while another uses Okta, each scoped to their Zitadel organisation within the same workspace instance. Your platform team configures this in the Zitadel Management Console for that workspace.

</Aside>

#### Sign-in journey (external IdP)

1. User chooses **Sign in with** an external provider on the Flow login screen (when enabled in login policy).
2. Flow redirects to Zitadel; Zitadel forwards the user to the external IdP.
3. After successful authentication, the user returns through Zitadel to Flow with tokens and profile claims.
4. **New users:** Zitadel can auto-create an account, or prompt to link to an existing local account, depending on template settings.
5. **Returning users:** Zitadel issues session tokens; Flow projects the user's **Zitadel project role** into Convex on login (see **[Roles and RBAC](/docs/auth/roles-and-rbac/)**).

#### Configure identity providers in Zitadel

Your platform or security team configures IdPs in the **Zitadel Management Console** for each workspace instance.

##### Enable external IdP login

1. Open **Login Behavior and Security** on the instance or organisation login policy.
2. Enable **External IDP Allowed** so users can choose federated sign-in.

##### Add a provider

1. Go to **Identity Providers** on the instance or organisation.
2. Pick a template (Google, Entra ID, generic OIDC, SAML SP, LDAP, and others) or use **Generic OIDC** / **SAML Service Provider** when no template exists.
3. Complete the vendor-specific client IDs, secrets, and redirect URIs Zitadel displays.

Common template settings:

| Setting                | Purpose                                                                                            |
| ---------------------- | -------------------------------------------------------------------------------------------------- |
| **Scopes**             | Claims requested from the external IdP (`openid`, `profile`, `email`, plus vendor-specific scopes) |
| **Automatic creation** | Create a Zitadel user on first external login when no account exists                               |
| **Automatic update**   | Refresh profile fields from the IdP on each login                                                  |
| **Account linking**    | Allow linking external identity to an existing Zitadel user                                        |

##### Instance default vs organisation IdP

| Scope                | When to use                                                                     |
| -------------------- | ------------------------------------------------------------------------------- |
| **Instance default** | One SSO strategy for all organisations in the workspace; central administration |
| **Organisation**     | Per-customer or per-business-unit IdP; delegated administration                 |

#### Supported provider guides

Zitadel publishes step-by-step guides for common providers. Use these when wiring your workspace (links open ZITADEL Docs):

- [Google](https://zitadel.com/docs/guides/integrate/identity-providers/google)
- [Entra ID (OIDC)](https://zitadel.com/docs/guides/integrate/identity-providers/azure-ad-oidc)
- [Entra ID (SAML)](https://zitadel.com/docs/guides/integrate/identity-providers/azure-ad-saml)
- [GitHub](https://zitadel.com/docs/guides/integrate/identity-providers/github)
- [Okta (generic OIDC)](https://zitadel.com/docs/guides/integrate/identity-providers/okta-oidc)
- [LDAP](https://zitadel.com/docs/guides/integrate/identity-providers/ldap)

For the full introduction and additional templates, see [Let Users Login with Preferred Identity Provider](https://zitadel.com/docs/guides/integrate/identity-providers/introduction) on ZITADEL Docs.

#### SCIM provisioning

Each workspace **Zitadel** instance supports **inbound SCIM v2**. Your corporate directory or HR system pushes user lifecycle events (create, update, deactivate) to Zitadel; Fontana does not require a separate SCIM endpoint in Flow.

Typical setup:

1. Enable SCIM on the Zitadel organisation or project in the **Zitadel Management Console** and copy the SCIM endpoint URL and bearer token for your IdP (Entra ID, Okta, and others publish SCIM app templates for Zitadel).
2. Map directory groups or attributes to **Zitadel project roles** so provisioned users receive the correct role grant before first login.
3. On login (or SCIM-driven pre-provision), Flow projects the Zitadel role into Convex **`users.roleId`** (see **[Roles and RBAC](/docs/auth/roles-and-rbac/)**). There are no per-user permission overrides in the app.
4. Deprovisioning in the directory disables or removes the user in Zitadel; Flow enforces **`users.enabled`** so access ends without editing roles inside Admin.

Identity events from SCIM and sign-in feed the **immutable security audit trail** automatically. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**.

For Zitadel SCIM configuration details, see [SCIM provisioning](https://zitadel.com/docs/guides/manage/user/scim2) on ZITADEL Docs.

#### MFA and session policy

MFA (TOTP and WebAuthn) is enforced through **Zitadel login policy**, independent of which IdP authenticated the user. Your organisation can require MFA for all users or specific login methods. WebAuthn requires a registrable domain over HTTPS on production auth hostnames; local development typically uses TOTP.

#### Related documentation

- **[Roles and RBAC](/docs/auth/roles-and-rbac/)** - how Zitadel project roles map to Flow permissions after login
- **[Identity and access](/docs/auth/)** - OIDC, SCIM, service authentication, and audit
- **[Security](/docs/security/)** - platform security controls and SOC 2 summary
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** - automated Zitadel identity events in the WORM ledger


# Roles and RBAC
## Description: Fontana default roles, the permission catalog, custom roles in Admin, and how Zitadel assigns users to roles.
## url: https://fontana-ai.com/docs/auth/roles-and-rbac/

Fontana uses **role-based access control (RBAC)** in two layers:

1. **Zitadel** - who the user is, MFA, and **which role name** they hold (project role grant)
2. **Flow / Convex** - **what that role can do** via a static permission catalog and dynamic role documents

You assign roles in Zitadel (or via SCIM when enabled). Admins define permission sets in **Admin → Roles**. There are **no per-user permission overrides** in the app: effective permissions always come from the user's assigned role.

<Aside type="note" title="What is RBAC?">

**Role-based access control** maps named roles to permissions. Users receive one role; the role grants a set of `resource:action` permissions checked on every protected route and API call.

</Aside>

#### How the layers connect

```text
Zitadel project role (e.g. editor)
  → JWT claim on login
  → Convex storeUser → users.roleId
  → roles.permissions → effective permission list
  → Flow sidebar, pages, and mutations
```

| Layer             | You configure                                                                | Fontana stores                                            |
| ----------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------- |
| **Identity**      | Zitadel (SSO, MFA, project role grants, SCIM when enabled)                   | User profile, `enabled` flag, projected `roleId`          |
| **Authorization** | **Admin → Roles** (permission sets), **Admin → Users** (enable/disable only) | Convex `roles` table, permission checks on every mutation |

Zitadel project role **keys must match** Convex role **names** exactly (for example `admin`, `editor`, `viewer`, `member`). If the token carries a role name with no matching Convex role, the user signs in but receives no permissions until an admin creates the role or fixes the Zitadel grant.

<Aside type="caution" title="Users table is read-only for roles">

**Admin → Users** shows the effective role sourced from Zitadel. You cannot escalate privilege by editing roles in Flow. Change the project role grant in Zitadel (or SCIM), then the user signs in again (or refreshes the session) to pick up the new role.

</Aside>

#### Default roles

Four roles are upserted on every workspace seed. Custom roles you create in Admin are never modified by seed.

| Role         | Typical use                                                       | Permission summary                                                                                                                    |
| ------------ | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **`admin`**  | Platform and security administrators                              | Wildcard `*` (all permissions)                                                                                                        |
| **`editor`** | Builders and analysts who create workflows, agents, and knowledge | Full read/write on workflows, agents, skills, Knowledge Graph, files, and evals; AI model selection; no user admin or platform deploy |
| **`viewer`** | Read-only operators and reviewers                                 | Read workflows, agents, skills, Knowledge Graph, files, evals, and explore; edit own profile                                          |
| **`member`** | Minimal access                                                    | Explore, read workflows, edit own profile                                                                                             |

Grant these roles in Zitadel as **project roles** on the Flow application. Provisioning seeds a break-glass `admin@fontana.local` account with the `admin` project role on fresh installs.

#### Permission format

Permissions use **`resource:action`** ids (OAuth2-compatible), defined in the static catalog shipped with Fontana:

| Resource          | Example actions                               | What it gates                                  |
| ----------------- | --------------------------------------------- | ---------------------------------------------- |
| `workflows`       | `read`, `write`, `delete`                     | Canvas, workflow lists, run and edit           |
| `agents`          | `read`, `write`, `delete`                     | AI agent definitions and Admin agent surfaces  |
| `skills`          | `read`, `write`, `delete`                     | Skill packages                                 |
| `knowledge-graph` | `read`, `write`, `delete`                     | Knowledge Graph documents and namespaces       |
| `evals`           | `read`, `write`, `run`, `delete`              | Evaluation tests                               |
| `files`           | `read`, `write`, `delete`                     | File browser and uploads                       |
| `ai`              | `page`, `select_model`, `add_model`, …        | Usage page, model approval, defaults           |
| `users`           | `read`, `write`, `create`, `delete`, `enable` | **Admin → Users** and user management          |
| `teams`           | `admin`                                       | **Admin → Teams** (create, rename, membership) |
| `platform`        | `read`, `write`                               | **Admin → Deployment**, API keys, releases     |
| `profile`         | `read`, `write`, `allow_email_change`         | User profile and preferences                   |
| `explore`         | `read`                                        | Community template explore                     |
| `audit_logs_own`  | `read`                                        | LLM debug for the caller's chat threads        |
| `audit_logs_team` | `read`                                        | LLM debug for any user's threads (high trust)  |

Wildcards are supported when editing roles in Admin:

- `*` - all permissions (used by default `admin`)
- `workflows.*` - all actions on one resource area

#### Manage roles in Admin

**Admin → Roles** lets you:

- Create **custom roles** with any combination of catalog permissions or wildcards
- Edit permission sets for custom roles (default seed roles can be edited like any other Convex role document)
- Delete custom roles that are no longer needed

After you create a custom role in Flow, grant the matching **project role name** in Zitadel so users receive it on login.

#### Teams and workflow access

**Teams** add a second dimension for **team-bound workflows**:

- **`teams:admin`** - create teams, rename teams, and manage team membership (`viewer` or `editor` on the team)
- **Team-bound workflows** require **both** global `workflows:*` permissions **and** team membership with the right team role

Personal workflows (`teamId` unset) follow global RBAC only. See **Admin → Teams** and **Team Workflows** in the sidebar.

#### Route and UI gates

Flow hides sidebar entries and blocks routes when the signed-in user lacks the required permission. Examples:

| Route                          | Permission required    |
| ------------------------------ | ---------------------- |
| `/workflows`, `/workflow`      | `workflows:read`       |
| `/agents`, `/tools`            | `agents:read`          |
| `/knowledge-graph`             | `knowledge-graph:read` |
| `/usage`, `/threads`           | `ai:page`              |
| `/admin/users`, `/admin/roles` | `users:read`           |
| `/admin/teams`                 | `teams:admin`          |
| `/admin/deployment`            | `platform:read`        |
| `/audit-logs/chat` (own)       | `audit_logs_own:read`  |

Backend mutations call the same permission checks; missing permission fails closed with an error.

#### Disabled users

When an admin disables a user in **Admin → Users**, effective permissions become **empty** immediately regardless of role. Re-enable restores permissions from the assigned role.

#### Audit

Privileged RBAC changes (role edits, user enable/disable, team membership) should appear in admin audit logs and, when producers are enabled, the immutable security audit trail. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail)**.

#### Related documentation

- **[Identity providers](/docs/auth/identity-providers/)** - SSO and federation into Zitadel before roles apply
- **[Identity and access](/docs/auth/)** - MFA, SCIM provisioning, service authentication
- **[Security](/docs/security/)** - platform security controls and SOC 2 summary
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** - identity and admin events in the WORM ledger


# Overview
## Description: Governed data ingress, secure run-scoped storage, Apache Arrow processing, and export paths.
## url: https://fontana-ai.com/docs/data/

Fontana connects your workflows to **600+ upstream and downstream integrations** through governed ingestion, secure processing, and export in the workflow engine. You bring data in on **File Input**, REST triggers, or PyAirbyte connectors; the workflow engine stores and transforms it as **Apache Arrow** datasets on an encrypted, workspace-scoped file store; you push results to files, APIs, or catalog destinations on the way out.

<Aside type="note" title="Where data runs">

File uploads, connector fetch, transforms, and exports execute on the **workflow engine** in your workspace cluster. Connector secrets resolve from **Vault at runtime**; they are distinct from **[BYOK](/docs/ai/byok/)** LLM keys. See **[Workflow orchestration](/docs/workflows/)** for the reactive execution model.

</Aside>

#### Topics

- **[Ingress](/docs/data/ingress/)** - File Input, REST Input, PyAirbyte connectors, and the upstream connector catalog
- **[Supported formats](/docs/data/supported-formats/)** - File Input extensions for tabular, financial, office, markup, and image uploads
- **[Storage and processing](/docs/data/storage-and-processing/)** - immutable, durable, auditable Arrow storage and Core Fontana data principles
- **[Data lineage](/docs/data/data-lineage/)** - atomic provenance from ingress to export, OpenLineage export
- **[Egress](/docs/data/egress/)** - Save File Export, REST and GraphQL API nodes, downstream connectors, and export paths

#### Connector catalog

Fontana ships mirror-pinned, digest-pinned connector images on GHCR with **Trivy CRITICAL** scanning as part of the SOC 2 supply-chain control set. Browse representative sources and destinations on the ingress and egress pages; the full Airbyte catalog runs inside the workflow engine via PyAirbyte with no separate Airbyte platform server.

#### Secrets for connectors and APIs

Connector and API credentials never appear in workflow config as plaintext. You configure secrets in **Admin** or node settings; values resolve from **HashiCorp Vault** when a run executes.

- Distinct from **[BYOK](/docs/ai/byok/)** LLM keys (separate scope and Admin surfaces)
- Scoped to your workspace; never returned to the browser

#### Related documentation

- **[Security](/docs/security/)** - encryption at rest, workspace isolation, supply chain
- **[Compliance evidence](/docs/compliance/)** - data lineage and immutable audit trail
- **[Workflow orchestration](/docs/workflows/)** - reactive execution engine and node families
- **[Deployment](/docs/deployment/)** - supply-chain pinned connector images on production hosts


# Ingress
## Description: File Input, REST triggers, PyAirbyte connectors, and upstream data sources.
## url: https://fontana-ai.com/docs/data/ingress/

Ingress is how data enters a workflow: uploads on **File Input**, HTTP triggers on **REST Input**, or catalog sync through **Data Connector** nodes backed by PyAirbyte. Parsed tabular data is stored securely in your workspace (see **[Storage and processing](/docs/data/storage-and-processing/)**). Connector credentials are configured in node settings and resolved from Vault at run time, never stored in workflow config.

#### Upstream connector catalog

<DocsUpstreamConnectorsTable client:only="react" />

#### Ingress mechanisms

<DataReplicationDiagram />

| Mechanism                    | Description                                                                                                                                                                                                                                                                         |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **File Input**               | Upload or drag-and-drop. Tabular, FIX, SWIFT/MT940, and OFX/QFX files parse to row datasets on the workflow engine; PDF and office documents route through Docling on the platform cluster. See **[Supported formats](/docs/data/supported-formats/)** for the full extension list. |
| **REST Input**               | External systems POST JSON to trigger a run and populate the input node output port.                                                                                                                                                                                                |
| **Data Connector (Airbyte)** | Airbyte-catalog sources via PyAirbyte: Notion, Postgres, S3, and hundreds more. No separate Airbyte platform server.                                                                                                                                                                |

#### PyAirbyte execution path

<Aside type="note" title="What is PyAirbyte?">

Fontana runs the Airbyte connector catalog **inside your workspace** without a separate Airbyte platform server. Connectors fetch and deliver data as part of workflow execution on the workflow engine.

</Aside>

<PyairbyteDiagram />

- Python and YAML connectors run inside the workflow engine pod
- Docker and Java connectors run in isolated short-lived jobs with scanned, pinned images
- Connectors are baked into release images at build time (no runtime package installs on production hosts)

#### Related documentation

- **[Supported formats](/docs/data/supported-formats/)** - File Input extensions, FIX, SWIFT, and office formats
- **[Storage and processing](/docs/data/storage-and-processing/)** - where ingested data is stored and how runs persist
- **[Egress](/docs/data/egress/)** - export and downstream connectors
- **[Data](/docs/data/)** - connector secrets and section overview


# Supported formats
## Description: File Input extensions for tabular, financial, office, markup, and image uploads.
## url: https://fontana-ai.com/docs/data/supported-formats/

**File Input** nodes accept the extensions below by default. You can limit accepted types in the File Input node settings. Parsed tabular data becomes **Apache Arrow** datasets on the workflow engine (see **[Storage and processing](/docs/data/storage-and-processing/)**).

#### Extension reference

| Extension   | Category                   | Processing                                                                                                        |
| ----------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `.csv`      | Tabular and spreadsheets   | Parsed in-browser or on the workflow engine into Arrow datasets. Excel formats use server-side conversion.        |
| `.tsv`      | Tabular and spreadsheets   | Parsed in-browser or on the workflow engine into Arrow datasets. Excel formats use server-side conversion.        |
| `.psv`      | Tabular and spreadsheets   | Parsed in-browser or on the workflow engine into Arrow datasets. Excel formats use server-side conversion.        |
| `.xls`      | Tabular and spreadsheets   | Parsed in-browser or on the workflow engine into Arrow datasets. Excel formats use server-side conversion.        |
| `.xlsx`     | Tabular and spreadsheets   | Parsed in-browser or on the workflow engine into Arrow datasets. Excel formats use server-side conversion.        |
| `.json`     | Structured data            | JSON and XML load as structured records for downstream transforms.                                                |
| `.xml`      | Structured data            | JSON and XML load as structured records for downstream transforms.                                                |
| `.fix`      | Financial data             | FIX, SWIFT MT / MT940, and OFX/QFX parsed via the platform file-parser pipeline.                                  |
| `.swift`    | Financial data             | FIX, SWIFT MT / MT940, and OFX/QFX parsed via the platform file-parser pipeline.                                  |
| `.mt940`    | Financial data             | FIX, SWIFT MT / MT940, and OFX/QFX parsed via the platform file-parser pipeline.                                  |
| `.ofx`      | Financial data             | FIX, SWIFT MT / MT940, and OFX/QFX parsed via the platform file-parser pipeline.                                  |
| `.qfx`      | Financial data             | FIX, SWIFT MT / MT940, and OFX/QFX parsed via the platform file-parser pipeline.                                  |
| `.pdf`      | Office and PDF             | PDF and office documents route through Docling on the workflow engine for structure, tables, and text extraction. |
| `.docx`     | Office and PDF             | PDF and office documents route through Docling on the workflow engine for structure, tables, and text extraction. |
| `.pptx`     | Office and PDF             | PDF and office documents route through Docling on the workflow engine for structure, tables, and text extraction. |
| `.txt`      | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.md`       | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.markdown` | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.html`     | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.htm`      | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.adoc`     | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.asciidoc` | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.asc`      | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.tex`      | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.latex`    | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.vtt`      | Text, markup, and captions | Plain text and markup sources for structured extraction and downstream transforms.                                |
| `.png`      | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.jpg`      | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.jpeg`     | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.tif`      | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.tiff`     | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.bmp`      | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |
| `.webp`     | Images                     | Raster images for OCR and document-ingest workflows via Docling.                                                  |

#### FIX parsing

<Aside type="note" title="What is FIX?">

Industry messaging standard for trading and post-trade data. Fontana parses FIX log files to one row per message on File Input and supports round-trip export on Save File Export with recomputed checksum fields.

</Aside>

FIX (Financial Information eXchange) log files load through the same File Input path as other structured imports.

- **One row per message** - each FIX message becomes one workflow row.
- **Delimiters** - SOH (ASCII 1), pipe (`|`), and escaped `\x01` field separators are accepted in log files.
- **Field names** - standard FIX tags map to readable column names such as `MsgType` and `SenderCompID`. Unknown tags keep the numeric tag; duplicate tags can appear as arrays.
- **Message boundaries** - messages are recognised from `8=FIX.` through the `10=` checksum field.
- **Structured import** - FIX rows are treated as already structured (one row per message). The importer does not suggest subgrid flattening for FIX sources.

Related finance formats on File Input: **SWIFT** (`.swift`, `.mt940`) and **OFX/QFX** use separate parsers in the same package. FIX round-trip export is supported on **[Egress](/docs/data/egress/)** via Save File Export.

#### Related documentation

- **[Ingress](/docs/data/ingress/)** - File Input, REST Input, and PyAirbyte connectors
- **[Egress](/docs/data/egress/)** - Save File Export and FIX round-trip
- **[Storage and processing](/docs/data/storage-and-processing/)** - Arrow datasets and the workflow file store


# Storage and processing
## Description: Core Fontana data principles - immutable, durable, repeatable, auditable, transparent, and queryable workspace storage.
## url: https://fontana-ai.com/docs/data/storage-and-processing/

When a workflow runs on the server, the **workflow engine** stores every upload, connector read, transform output, and export on **encrypted storage inside your workspace**. Fontana is built around a small set of **data principles** so operators, auditors, and downstream systems can trust what was computed, what was changed, and what can be reproduced.

#### Core Fontana data principles

| Principle                       | In one sentence                                                                                             |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **[Immutable](#immutable)**     | Data input or engine output is never overwritten; corrections are explicit audited overlays.                |
| **[Durable](#durable)**         | Run data survives restarts and upgrades and upstream data changes - on encrypted, workspace-scoped storage. |
| **[Repeatable](#repeatable)**   | Each run is its own snapshot; selective re-runs refresh only what changed.                                  |
| **[Auditable](#auditable)**     | Validations, transforms, and manual edits leave structured audit evidence.                                  |
| **[Transparent](#transparent)** | Every node output is captured, inspectable in Flow, exportable, and explorable by you and your agents.      |
| **[Queryable](#queryable)**     | Datasets stay in Apache Arrow so you can inspect, filter, and query them in Flow.                           |
| **[Isolated](#isolated)**       | Your data stays in your workspace cluster unless you choose to export it.                                   |

#### Immutable

Fontana treats **processor output as immutable**. When a node executes, it produces a **new** dataset for that port. The engine does not mutate the raw output from a previous execution in place.

Manual **cell and header edits** in the Flow grid follow the same rule. Your corrections are stored as **separate overlays** and merged only when you view data or when downstream nodes consume a port. The underlying computed dataset remains available for audit and replay.

That separation matters for compliance: you can always distinguish **what the workflow calculated** from **what a person changed afterward**.

#### Durable

Workflow data lives on **dedicated encrypted storage** attached to your workspace as persistent volumes.

- **Encryption at rest** on production deployments protects uploads, run datasets, and exports at the infrastructure layer.
- **Survives restarts and upgrades** within your workspace so a platform update does not wipe active or historical runs.
- **Connector output** from PyAirbyte is written to the same durable store as File Input and transform results, keeping ingress and processing inside one boundary.

See **[Security](/docs/security/)** for the broader encryption and isolation control model.

#### Repeatable

Every workflow **run** is a **discrete snapshot**: uploads and connector reads for that execution, node outputs as the graph runs, audit and lineage metadata, and any export files generated along the way.

When inputs or node configuration change, the reactive engine **re-runs only the affected branches**. Unchanged branches can reuse prior results where hashes match, so you get predictable refreshes without recomputing the entire graph every time.

You can open an earlier run in Flow, compare outcomes, and download exports from that point in time without losing newer runs on the same workflow.

#### Auditable

Fontana records **audit items** throughout execution and review:

- **Validation and transform events** - warnings, errors, and operation summaries on retained rows
- **Manual edits** - row, column, previous value, new value, and timestamp for every grid correction
- **Data lineage** - pointer-based provenance from an output cell back through the workflow graph to sources (see **[Data lineage](/docs/data/data-lineage/)**)

Together, these signals support operational troubleshooting and compliance review. They describe **workflow data quality and provenance**. For sign-in, admin, and cluster events, see **[Immutable audit trail](/docs/compliance/immutable-audit-trail)**.

#### Transparent

Every step of a workflow is **inspectable and exportable**. Fontana captures the result at **each node** and surfaces it in the app, so you never work blind through a black-box pipeline.

On the canvas, open any node to review its output port: row counts, validation signals, and the full dataset behind that step. The same run data is available across Flow:

- **Data grid** - tabular view of each port, with your edits applied when viewing; edited cells show markers on the canvas
- **Audit panel** - filterable audit items for the active run
- **Data Lineage panel** - walk from a selected cell back through transforms, lookups, and sources (see **[Data lineage](/docs/data/data-lineage/)**)
- **Export** - download node or run outputs via Save File Export and related egress paths (see **[Egress](/docs/data/egress/)**)

**You and your governed AI agents** can work with that captured data directly: explore ports in the UI, **run queries** against run datasets in the workflow or grid, or use **AI chat and inference** to ask questions, summarize results, and analyze patterns on the data your workflow produced. Transparency means the platform exposes what ran at every step; you choose how to inspect it, query it, or reason over it.

See **[Governed AI](/docs/ai/)** for agents, tools, and model access on workflow context.

#### Queryable

Fontana stores run datasets in **Apache Arrow**, an open columnar format designed for analytics workloads. Columnar layout keeps large tables efficient for transforms, connectors, validation, and interactive review.

<Aside type="note" title="What is Apache Arrow?">

**Apache Arrow** is a standard columnar memory and storage format used across the analytics ecosystem. Because Fontana keeps workflow data in Arrow, the same datasets power the Flow grid, workflow **Query** nodes, and agent tooling without converting everything to JSON on every step.

</Aside>

In practice you can:

- **Browse and filter** port data in the Flow data grid
- **Run Query nodes** in a workflow using SQL and other supported query modes against run datasets
- **Let governed agents** query approved run data where your RBAC and workflow design allow it
- **Export** to CSV, Excel, JSON, and other formats for tools outside Flow (see **[Egress](/docs/data/egress/)**)

Queryable storage closes the loop: data is not only stored safely, it stays **usable** inside the platform and portable when you need it elsewhere.

#### Isolated

Your workspace is a **separate cluster** with its own storage, secrets, and identity plane. Workflow data is not co-mingled with other customers on the platform.

Data leaves that boundary only when **you** configure egress: Save File Export, REST or GraphQL API nodes, downstream connectors, or other delivery paths you approve. Platform operation itself does not require shipping your datasets to a shared Fontana-operated data pool.

#### Related documentation

- **[Ingress](/docs/data/ingress/)** - how data enters your workspace
- **[Egress](/docs/data/egress/)** - downloads and downstream delivery
- **[Data lineage](/docs/data/data-lineage/)** - atomic provenance and OpenLineage export
- **[Security](/docs/security/)** - encryption at rest and workspace isolation
- **[Compliance evidence](/docs/compliance/)** - data lineage and audit evidence
- **[Workflow orchestration](/docs/workflows/)** - reactive execution and document parsing


# Data lineage
## Description: Atomic provenance from ingress through export: how values were derived, what sources contributed, and which branches processed each row.
## url: https://fontana-ai.com/docs/data/data-lineage/

**Data lineage** is Fontana's record of **how every value in a workflow run came to be**: which files or connectors supplied it, which transforms and validations touched it, which lookups enriched it, and which nodes on the graph produced the result you see at export time.

Lineage is **atomic**: attached at row and column level, not only as a workflow summary. That gives auditors and operators a precise answer to _how_, _why_, and _from where_ data changed, including when **conditional logic** sent rows down different processing paths.

#### End-to-end coverage

Fontana captures lineage across the full data lifecycle:

| Stage                      | What lineage records                                                                                                                   |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Ingress**                | Files, connector reads, and API inputs (`loaded` provenance back to the source dataset)                                                |
| **Transform and validate** | Operations, lookups, merges, compute steps, schema mapping, and AI-assisted transforms                                                 |
| **Conditional processing** | Filters, validations, and condition operations that include or exclude rows, and branches of the graph that contributed to each output |
| **Egress**                 | Exported rows remain linked to the same provenance chain through Save File Export and downstream delivery                              |

Every processor step that derives data can emit pointer-based sources. Those sources may themselves have lineage, forming a **recursive tree** you can walk from an export cell back to original intake.

<Aside type="note" title="What is atomic lineage?">

**Atomic** means provenance is stored as **pointers** to upstream rows and columns, not as copied snapshots of entire datasets. You can trace one cell without duplicating millions of identical records, and each link names the **node** and, where relevant, the **operation** that produced the change.

</Aside>

#### How sources are described

Each lineage link carries a **usage type** that explains why that source contributed to the result:

| Usage              | Meaning                                                                                   |
| ------------------ | ----------------------------------------------------------------------------------------- |
| **loaded**         | Row or field came from an external source (file upload, connector, API)                   |
| **transformation** | Source value was transformed (for example string, date, or decimal operations)            |
| **lookup**         | Source was matched and enriched from a lookup dataset                                     |
| **merge**          | Source was combined with other inputs                                                     |
| **reference**      | Source passed through unchanged                                                           |
| **computed**       | Source used in a function or compute step                                                 |
| **generated**      | Source used to generate new values (for example schema mapping or AI output)              |
| **filtered**       | Source passed through filter or validation context (row included or excluded on a branch) |

Together, usage types and node references show **what data was used to calculate results** and **which conditional branches** were involved when filters, validations, or graph paths split processing.

#### View lineage in Flow

You do not need to leave the app to inspect provenance:

- **Data Lineage panel** - select a cell in the data grid; the side panel builds a **recursive tree** from that cell back through upstream nodes, with usage labels and operation context
- **Workflow data view** - review port-level datasets per node for the active run
- **Audit panel** - complementary **audit items** (validation outcomes, transform events, manual edits) alongside lineage for the same run

Lineage answers _provenance_; audit items answer _events_ on the row. Both support the **[Transparent](/docs/data/storage-and-processing/#transparent)** data principle.

#### Export lineage

Lineage travels with the run. You can **export provenance** for compliance archives, downstream metadata systems, or independent review:

- **OpenLineage** - Fontana lineage can be **viewed and exported in [OpenLineage](https://openlineage.io/) format**, the open standard for data pipeline metadata. Export supports integration with observability, catalog, and governance tools that consume OpenLineage events without custom adapters for Fontana's internal model.
- **Run context** - exports are scoped to a workflow **run**, so evidence matches the same snapshot as your datasets and Save File Export outputs (see **[Egress](/docs/data/egress/)**)

<Aside type="note" title="OpenLineage">

[OpenLineage](https://openlineage.io/) is an open framework for capturing **run**, **job**, and **dataset** metadata across data pipelines. Fontana maps atomic workflow lineage into that model so your organisation can consolidate pipeline provenance with other OpenLineage-compatible systems.

</Aside>

#### Why it matters

- **Regulatory and audit review** - demonstrate which inputs and transforms produced reported figures
- **Incident analysis** - isolate when a value changed and which step introduced an error or exception
- **Operational trust** - operators see the same provenance graph that compliance reviewers export
- **Downstream integration** - OpenLineage export lets metadata platforms ingest Fontana runs alongside warehouses, orchestrators, and BI tools

Lineage is part of Fontana's **[Auditable](/docs/data/storage-and-processing/#auditable)** and **[Transparent](/docs/data/storage-and-processing/#transparent)** data principles. It complements, but is **not the same as**, the platform **immutable security audit trail** (sign-in, admin, cluster events in ImmuDB).

<Aside type="caution" title="Lineage is not the WORM ledger">

**Data lineage** and **port audit items** live on the **workflow file store** (`.lineage.json`, `.audit.json` sidecars, canvas validation context). They answer _how a value was computed_. The **WORM ledger** in ImmuDB answers _who performed privileged platform activity_. Do not cite lineage exports as a substitute for ImmuDB audit evidence. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** and **[Data retention](/docs/security/data-retention/)**.

</Aside>

#### Related documentation

- **[Storage and processing](/docs/data/storage-and-processing/)** - Core Fontana data principles
- **[Ingress](/docs/data/ingress/)** - where loaded lineage begins
- **[Egress](/docs/data/egress/)** - exporting datasets with provenance context
- **[Compliance evidence](/docs/compliance/)** - workflow lineage overview
- **[Security](/docs/security/)** - SOC 2 control summary
- **[Workflow orchestration](/docs/workflows/)** - reactive execution and node families


# Egress
## Description: Save File Export, REST and GraphQL API nodes, PyAirbyte destinations, and downstream connectors.
## url: https://fontana-ai.com/docs/data/egress/

Egress is how processed data leaves a workflow: download files from **Save File Export**, call external APIs with **REST API** or **GraphQL API** nodes, or sync to catalog destinations through **Data Connector** nodes. Server-side runs keep export files on your workspace storage until you download them from the node or the **Files** UI.

#### Downstream connector catalog

<DocsDownstreamConnectorsTable client:only="react" />

#### Egress mechanisms

<DataExportDiagram />

| Mechanism                    | Description                                                                                                                     |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **Save File Export**         | CSV, TSV, PSV, XLS, JSON, XML, TXT, and FIX (`.fix` round-trip). Download from the node or the **Files** UI after a server run. |
| **REST API node**            | Call external HTTP APIs; map responses into the workflow dataset.                                                               |
| **GraphQL API node**         | Execute GraphQL queries against external endpoints.                                                                             |
| **Data Connector (Airbyte)** | Catalog destinations (and sources) through the same PyAirbyte path as ingress.                                                  |

#### FIX round-trip export

Save File Export can write `.fix` files with SOH delimiters and recomputed checksum fields (`BodyLength`, `CheckSum`). The canvas preview uses pipe delimiters for readability. Export requires mappable columns plus `BeginString` and `MsgType` on each row. See **[Supported formats](/docs/data/supported-formats/)** for FIX import behaviour.

#### Connector and API secrets

Outbound connectors and API nodes use the same Vault-backed secret model as ingress. Values resolve when a run executes; nothing sensitive is embedded in workflow config. See **[Data](/docs/data/)** for the secrets model.

#### Related documentation

- **[Ingress](/docs/data/ingress/)** - upstream sources and PyAirbyte execution
- **[Storage and processing](/docs/data/storage-and-processing/)** - secure workspace storage for export files
- **[Workflow orchestration](/docs/workflows/)** - canvas execution model


# Overview
## Description: Operational telemetry and security audit - two complementary planes for running and proving control of your Fontana workspace.
## url: https://fontana-ai.com/docs/observability/

Fontana includes **full observability** with every workspace deployment: operational telemetry for day-to-day platform operations, and a **tamper-evident security audit trail** for compliance evidence. Logs, metrics, dashboards, service health checks, and append-only audit history are built into the stack and provisioned with your environment. You do not need to install, license, or configure external telemetry or log aggregation products to gain visibility across your tenant.

On the **Flow** canvas, each node also exposes run status, row counts, and **data-quality audit items** as a workflow executes. Those signals are **workflow provenance on the file store**. See **[Data lineage](/docs/data/data-lineage/)** and **[Storage and processing](/docs/data/storage-and-processing/)**.

#### Topics

- **[Operational telemetry](/docs/observability/operational-telemetry/)** - HyperDX logs and dashboards, Gatus health checks, otel collector, and what Fontana collects for SRE and incident response
- **[Security audit (WORM)](/docs/observability/security-audit-worm/)** - append-only ImmuDB ledger, who emits security events, and how it differs from workflow lineage and chat audit

#### Planes at a glance

| Plane                                                                   | System of record                     | Typical use                                                                 |
| ----------------------------------------------------------------------- | ------------------------------------ | --------------------------------------------------------------------------- |
| **[Operational telemetry](/docs/observability/operational-telemetry/)** | **HyperDX** (search and dashboards)  | SRE dashboards, incident response, browser errors, pod logs, service health |
| **[Security audit (WORM)](/docs/observability/security-audit-worm/)**   | **ImmuDB** per workspace             | Compliance evidence, tamper-evident append-only platform activity           |
| **Workflow lineage and port audit**                                     | **Workflow file store** (not ImmuDB) | How a cell was computed; validation and edit context on a run               |

HyperDX may **mirror** security audit events for faster search. **ImmuDB** holds the compliance **system of record**. See **[Data retention](/docs/security/data-retention/)** for retention boundaries across HyperDX, ImmuDB, and the workflow file store.

<Aside type="note" title="Who configures this?">

Your **platform or Fontana deployment team** onboards HyperDX, wires log streams, and provisions WORM audit ingest during workspace install and upgrade (typically as part of `fontana apply`). Workspace users consume dashboards and search mirrors according to RBAC; they do not install observability agents themselves.

</Aside>

#### Related documentation

- **[Security](/docs/security/)** - workspace isolation, retention boundaries, SOC 2 control summary
- **[Data retention](/docs/security/data-retention/)** - HyperDX vs ImmuDB vs workflow file store
- **[Compliance evidence](/docs/compliance/)** - workflow lineage and immutable audit overview
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** - WORM producer map and operating status
- **[Identity and access](/docs/auth/)** - Zitadel identity events in the security audit stream
- **[Deployment](/docs/deployment/)** - cluster-per-workspace topology and upgrades


# Operational telemetry
## Description: HyperDX logs and dashboards, Gatus service health, and operational signals from Flow, Convex, and platform workloads.
## url: https://fontana-ai.com/docs/observability/operational-telemetry/

**Operational telemetry** is how your team **runs** Fontana day to day: search logs, trace errors, watch dashboards, and confirm services are up. Fontana centralises this at the **platform layer** so every workspace does not need its own observability stack.

Your operators use **HyperDX** on the platform cluster as the primary OTLP destination for logs, traces, and bundled dashboards. **Gatus** runs in-cluster (not inside the Flow app) and records **service health history** for platform and tenant workloads.

#### What gets collected

| Source                 | Examples                                | Typical use                             |
| ---------------------- | --------------------------------------- | --------------------------------------- |
| **Platform workloads** | Flow, workflow engine, Convex backend   | Pod logs, errors, run lifecycle         |
| **Flow browser**       | UI errors and product telemetry         | Front-end incidents and usage signals   |
| **Convex**             | Structured execution and console events | Backend behaviour and log stream topics |
| **Workflow engine**    | Run lifecycle, capacity samples         | Pipeline performance and failures       |
| **Connectors**         | Connector job output in the engine pod  | Ingress and egress troubleshooting      |
| **Node metrics**       | CPU, memory, load on the host           | Capacity and saturation                 |

Signals from each workspace are routed through **observability-api** in that workspace cluster before export to HyperDX, so tenancy boundaries stay intact.

<Aside type="note" title="Not in the Flow browser">

Fontana does **not** emit OpenTelemetry directly from the browser to third-party vendors. Browser telemetry goes through your workspace observability ingress so secrets and routing stay under your deployment's control.

</Aside>

#### HyperDX dashboards

During platform provisioning, your deployment team typically:

- Onboards a HyperDX team and OTLP credentials for the environment
- Loads bundled dashboards for platform and tenant services
- Points Convex log stream and browser telemetry webhooks at the workspace observability ingress

Ask your administrator for the **HyperDX UI URL** on your environment. On local k3d workspaces, run `fontana resources <tenant>` for the dedicated `observe-<tenant>` hostname (for example `https://observe-demo.localhost/`).

**Authentication:** the HyperDX UI requires a **native login** before showing any data. Provisioning registers the admin account through the HyperDX API during `fontana apply`, so no open welcome page is exposed after install. Admin credentials are stored in your workspace Vault and surfaced to operators by `fontana secrets <tenant>` under the **Operator surfaces** block.

#### HyperDX alerting

Fontana distinguishes **event delivery** (automatic) from **notification routing** (operator choice):

| Step                             | Automatic on `fontana apply`?         | What you get                                                                                                                                                                                                                         |
| -------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| OTLP ingestion key               | Yes (when HyperDX provisions cleanly) | Secret `observability-keys`; pods send logs to HyperDX                                                                                                                                                                               |
| Bundled dashboards               | Yes                                   | **Fontana — Logs**, **Fontana — Infrastructure**, **Fontana — WORM audit** (integrity scan outcomes)                                                                                                                                 |
| Integrity scan OTLP events       | Yes (daily CronJob)                   | One log per scan with `ServiceName:fontana-worm-integrity-scan` and `fontana.integrity_scan.outcome`                                                                                                                                 |
| Kubernetes audit security alerts | Yes                                   | Bundled K8s audit dashboard plus alert rules for authentication failures, secret access, destructive verbs, and ClusterRoleBinding changes; each alert delivers a webhook to observability-api, which appends a WORM evidence record |
| Email / Slack / PagerDuty alert  | No (one-time setup)                   | HyperDX alert rule on a dashboard tile or saved search; you pick the channel                                                                                                                                                         |

To add notifications for failed integrity scans: open **Fontana — WORM audit** in HyperDX, use the **Integrity scan failures** tile menu → **Create alert**, choose threshold `above 0`, and attach your Slack or email channel. Fontana cannot provision that channel without your webhook or recipient list.

If apply logs `team exists but Secret observability-keys has no stored credentials`, HyperDX is in a stuck state: restore the Secret from backup or reset the HyperDX data volume and re-run `fontana apply` before expecting scan events or alerts to work.

#### Gatus service health

**Gatus** probes HTTP endpoints for Flow, workflow engine, Convex, Vault, Postgres, Docling, and related services. It provides a **health history page** separate from HyperDX log search.

Gatus records **availability and uptime** for platform and tenant HTTP endpoints. Your administrator can share the in-cluster URL and port for your box.

**Authentication:** the Gatus status page enforces **HTTP basic auth**. Credentials are generated at deploy time, stored in your workspace Vault, and surfaced by `fontana secrets <tenant>` under the **Operator surfaces** block. In-cluster health probes are unaffected by the UI login.

#### HyperDX mirror vs WORM system of record

HyperDX is the **operational search plane**: fast log query, dashboards, and alerting during incidents. When security audit events are mirrored into HyperDX, they appear alongside pod logs and browser telemetry for convenience.

**ImmuDB is the compliance system of record** for platform security audit. HyperDX retention TTLs, index rebuilds, or search deletes do **not** change the append-only WORM ledger. For diligence, cite ImmuDB integrity procedures and **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**.

See **[Data retention](/docs/security/data-retention/)** for retention boundaries across workflow data, HyperDX, and ImmuDB.

#### Node log and metrics collector (otel-collector)

Each workspace cluster runs an **OpenTelemetry collector** as a **DaemonSet** on every node. It ships container logs and node CPU, memory, and load metrics to that cluster's HyperDX OTLP receiver.

| Property           | Posture                                                                                                                                     |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **Scope**          | **Per workspace cluster** only; no cross-tenant telemetry surface                                                                           |
| **Host access**    | Runs as root with **read-only** mounts of host log directories and host filesystem paths needed to read `/proc` and `/sys` for node metrics |
| **Kubernetes API** | Pod metadata is derived from log file paths on the node                                                                                     |
| **Egress**         | Allowed only to the in-cluster HyperDX OTLP endpoint (plus DNS), enforced by NetworkPolicy                                                  |
| **Redaction**      | A transform processor runs in every log pipeline before export, replacing bearer tokens, JWTs, API keys, and Vault tokens with `[REDACTED]` |

Within a single-tenant cluster, the node already runs only that workspace's pods, so host-level log collection does not read another tenant's workloads. This is an **operational telemetry** control, distinct from WORM audit evidence.

#### Workflow canvas signals (operational context)

While a workflow runs, the Flow canvas shows **per-node status**, row counts, and **port audit items** (validation warnings, transform notes, user edits). These help operators and builders debug a run in context.

That canvas audit is **workflow data quality and provenance on the workflow file store**. It is **not** operational telemetry in HyperDX and **not** the WORM security ledger. Port audit items and lineage sidecars answer _how a value was derived_; ImmuDB answers _who acted on the platform_. See **[Storage and processing](/docs/data/storage-and-processing/)**, **[Data lineage](/docs/data/data-lineage/)**, and **[Security audit (WORM)](/docs/observability/security-audit-worm/)**.

#### Related documentation

- **[Observability overview](/docs/observability/)** - operational vs security audit planes
- **[Security audit (WORM)](/docs/observability/security-audit-worm/)** - ImmuDB compliance ledger (distinct from HyperDX search)
- **[Data retention](/docs/security/data-retention/)** - HyperDX TTL vs WORM retention
- **[Deployment](/docs/deployment/)** - HyperDX onboarding during install and upgrade


# Security audit (WORM)
## Description: Append-only ImmuDB security ledger per workspace - identity, admin, and platform events for compliance evidence.
## url: https://fontana-ai.com/docs/observability/security-audit-worm/

**Security audit (WORM)** is Fontana's **compliance-grade event history**: who did what, when, and with what outcome, stored so later edits cannot silently rewrite the past. **WORM** means write once, read many: new records append; existing entries are not updated or removed through normal application paths.

Each workspace has its own **ImmuDB** ledger. **observability-api** in that workspace is the **only writer** to the ledger. Application components do not call ImmuDB directly; they emit events that observability-api normalises, redacts, and appends.

HyperDX may **mirror** the same events for operator search. For diligence and evidence, treat **ImmuDB as the system of record**, not HyperDX. HyperDX is a **search plane** with its own retention TTL; expiry or reindexing there does not alter the append-only WORM ledger. See **[Data retention](/docs/security/data-retention/)** and **[Operational telemetry](/docs/observability/operational-telemetry/)**.

<Aside type="note" title="What is WORM?">

**Write once, read many (WORM)** storage appends new audit records and detects tampering through cryptographic chaining. Fontana uses ImmuDB per workspace for security audit; operational log search in HyperDX is a separate plane.

</Aside>

#### What gets recorded

Security audit events share a **normalised schema**: workspace, timestamp, actor, action, resource, outcome, and correlation id. Sensitive material is **redacted before storage** (no API keys, bearer tokens, or full JWT bodies).

| Category                              | Examples                                            | Typical source                                                                                                      |
| ------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Identity and access**               | Sign-in, MFA, role and provisioning changes         | Zitadel (automated poll)                                                                                            |
| **Platform and cluster**              | Failed auth, secret access, privileged API calls    | Kubernetes API audit (bundled HyperDX alert rules fire on these events and append a WORM evidence record per alert) |
| **Application and admin**             | RBAC, BYOK, admin configuration                     | Convex structured admin logs (`auditable: true`) on the log stream                                                  |
| **Workflow and AI (security-tagged)** | Privileged runs, sensitive tool or connector access | Application audit producers (via Convex)                                                                            |

Identity events from **Zitadel** (login, MFA, federation, deprovision) reach the ledger **automatically** without custom Flow code.

#### Fail closed

If append to the WORM ledger fails, ingest paths return errors rather than dropping security events quietly. Your deployment team monitors observability-api and ImmuDB health as part of platform operations.

#### Review and integrity

Each workspace runs a daily **integrity scan** CronJob (`immu-audit-scan`, schedule `0 3 * * *` UTC):

1. **Scan (init container)** — `immuclient login` + `immuclient scan` against the ImmuDB ledger.
2. **Evidence (observability-api container)** — writes a JSON manifest and raw scan log to the workspace evidence volume (`worm-audit-evidence` PVC at `/evidence` on Kubernetes), optionally uploads both files to **S3 Object Lock** when your platform team configures `wormAudit.s3`, and posts a structured OTLP log to HyperDX.

| Artifact                       | Location                                            | Purpose                                                         |
| ------------------------------ | --------------------------------------------------- | --------------------------------------------------------------- |
| `immuclient-scan-<scanId>.log` | Evidence volume                                     | Raw scan output for auditors                                    |
| `manifest-<scanId>.json`       | Evidence volume                                     | Tenant, outcome, SHA-256 of log, S3 keys, HyperDX reported flag |
| OTLP log                       | HyperDX (`ServiceName:fontana-worm-integrity-scan`) | Operator search and optional alert rules                        |

**HyperDX alerting:** Fontana **automatically emits** a success or failure log on every scan when HyperDX is provisioned (`observability-keys` Secret present). The bundled **Fontana — WORM audit** dashboard (injected on `fontana apply`) shows recent scan outcomes. **Email, Slack, or PagerDuty notifications** are a one-time HyperDX alert rule you attach to the dashboard tile or saved search (Fontana does not know your notification channel until you configure it). See **[Operational telemetry](/docs/observability/operational-telemetry/)** § HyperDX alerting.

If the scan fails, or HyperDX OTLP delivery fails when configured, the CronJob exits non-zero so Kubernetes retains a failed Job for investigation (fail-closed).

Operators and auditors also use:

- **HyperDX** for interactive search during incidents (security audit mirror at `ServiceName:fontana-worm-audit`)
- **In-cluster REST** (`http://immu-audit:8080`) for smoke tests and break-glass verification (see operator runbook)

Read-only auditor access can be provisioned separately from the writer credential used by automated append.

#### Verify integrity scan (operators)

After deploy, trigger a one-off Job from the CronJob and confirm evidence plus HyperDX delivery:

```bash
export KUBECONFIG="$FONTANA_HOME/kubeconfig/tenant-<id>.yaml"   # e.g. tenant-demo
NS=fontana
JOB="immu-audit-scan-manual-$(date +%s)"
kubectl create job -n "$NS" "$JOB" --from=cronjob/immu-audit-scan
kubectl wait -n "$NS" --for=condition=complete "job/$JOB" --timeout=300s
kubectl logs -n "$NS" "job/$JOB" -c immuclient-scan
kubectl logs -n "$NS" "job/$JOB" -c evidence
```

Expect the evidence container log line `outcome=success`. In HyperDX, open the **Fontana — WORM audit** dashboard or search `ServiceName:fontana-worm-integrity-scan`. If `fontana apply` logged `observability-keys` missing, fix HyperDX provisioning before testing OTLP (see runbook below).

<Aside type="caution" title="HyperDX must be provisioned">

Integrity scan evidence **requires** Secret `observability-keys` (ingestion API key). When HyperDX MongoDB already has a team but the Secret was lost, `fontana apply` warns and skips re-onboarding. Restore the Secret from backup or reset the HyperDX PVC and re-run apply. Until then, the evidence Job fails on OTLP delivery by design.

</Aside>

<Aside type="caution" title="Names that sound like audit but are not WORM">

| Name                                        | What it is                                                                              |
| ------------------------------------------- | --------------------------------------------------------------------------------------- |
| **Audit logs → Chat** in Flow               | LLM debug inspection for authorised operators; stored in Convex                         |
| **Port audit items** on the canvas          | Data-quality warnings and errors on workflow outputs; **workflow file store**, not WORM |
| **Data lineage** (`.lineage.json` sidecars) | Calculation provenance; **workflow file store**, not WORM                               |
| **Workflow document history**               | Edit history in Convex tables; mutable application data                                 |

For calculation provenance, see **[Data lineage](/docs/data/data-lineage/)** and **[Compliance evidence](/docs/compliance/)**. For retention boundaries, see **[Data retention](/docs/security/data-retention/)**.

</Aside>

See **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** for schema detail, producer map, and workspace isolation.

#### Related documentation

- **[Observability overview](/docs/observability/)** - operational telemetry vs security audit
- **[Operational telemetry](/docs/observability/operational-telemetry/)** - HyperDX and Gatus
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail)** - full WORM architecture and producer map
- **[Identity and access](/docs/auth/)** - Zitadel and identity events in the ledger


# Overview
## Description: Deployment models, multi-tenancy, upgrades, and where Fontana runs.
## url: https://fontana-ai.com/docs/deployment/

Fontana runs on **Kubernetes** and ships the same governed stack whether Fontana operates the host or you run it in your own environment. Production releases are **pinned and immutable** so upgrades are reproducible and audit-friendly. Every **workspace** lives in a **separate cluster** with its own data, secrets, and identity, regardless of deployment model.

<Aside type="note" title="Cluster-per-tenant">

Multiple workspaces can share one physical host, each in its own Kubernetes cluster. That separation is the primary data and secrets boundary. A **dedicated host** (one customer per machine) is available when you need a smaller blast radius. See **[Architecture](/docs/deployment/architecture/)** for the platform vs workspace layout.

</Aside>

#### Topics

- **[Architecture](/docs/deployment/architecture/)** - platform and workspace clusters, shared services, isolation
- **[Cloud deployment](/docs/deployment/cloud-deployment/)** - Fontana-managed cloud and customer VPC on AWS, Azure, or GCP
- **[Self-hosted deployment](/docs/deployment/self-hosted-deployment/)** - on-premises Kubernetes, private registry mirrors, and customer-operated hosts
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - platform snapshots, rollback, and Admin configuration backup
- **[Fontana CLI](/docs/deployment/fontana-cli/)** - install, configure, apply, and tenant lifecycle on the host

#### Where Fontana runs

<DeployTargetsDiagram />

| Model                                                                  | Summary                                                                             |
| ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **[Cloud deployment](/docs/deployment/cloud-deployment/)**             | Managed cloud (Fontana operates the host) or **customer VPC** in your cloud account |
| **[Self-hosted deployment](/docs/deployment/self-hosted-deployment/)** | **On-premises** or customer-operated Kubernetes in your facility or private cloud   |
| **Dedicated host**                                                     | One customer per physical box; available in cloud or self-hosted arrangements       |

#### Upgrades and releases

Workspace upgrades use **immutable release tags** and automatic **pre-upgrade snapshots** so you can roll back if needed. Host operators reconcile changes with the **[Fontana CLI](/docs/deployment/fontana-cli/)**; application configuration can be exported from **Admin → Deployment → Backup/Restore**. See **[Backup and restore](/docs/deployment/backup-and-restore/)** for both paths.

#### Related documentation

- **[Security](/docs/security/)** - isolation, encryption, supply chain, SOC 2 control summary
- **[Compliance evidence](/docs/compliance/)** - workflow lineage and immutable audit trail
- **[Observability](/docs/observability/)** - HyperDX onboarding during install and upgrade
- **[Identity and access](/docs/auth/)** - Zitadel provisioned per workspace cluster
- **[Platform overview](/docs/welcome/platform-overview/)** - runtime components and platform capabilities


# Architecture
## Description: Platform and workspace clusters, shared services, and cluster-per-tenant isolation.
## url: https://fontana-ai.com/docs/deployment/architecture/

Fontana deploys as **two cluster layers** on each host: one **platform cluster** for shared edge and stateless services, and one **workspace cluster per tenant** for the full application stack. That layout is the same on managed cloud, customer VPC, and self-hosted installs.

<Aside type="note" title="Cluster-per-tenant">

Multiple workspaces can share one physical host, each in its own Kubernetes cluster. That separation is the primary **data and secrets boundary**. A **dedicated host** (one customer per machine) reduces shared-kernel exposure while keeping the same topology.

</Aside>

#### Topology diagram

<TechnicalDocArchitectureTopology client:only="react" />

#### Platform cluster

Shared across all workspaces on the host:

- **Edge ingress** and the welcome gateway (routing to workspace URLs)
- **Observability** - HyperDX/OTLP collection and operator dashboards
- **Docling** - shared document parsing for PDF and office uploads
- **OpenSandbox** - isolated code execution for agent MCP tooling
- **Platform health checks** - service uptime probes for operators

The platform cluster does **not** hold workspace databases, Vault secrets, or workflow datasets.

#### Workspace cluster

Each workspace is a **complete Fontana environment**:

| Area              | Components                                                           |
| ----------------- | -------------------------------------------------------------------- |
| **Application**   | Flow, Convex backend, workflow engine, PyAirbyte connector runtime   |
| **Data**          | Postgres, encrypted workflow file store, Convex persistence          |
| **Security**      | HashiCorp Vault (BYOK, connector secrets), Zitadel identity          |
| **Observability** | observability-api (operational telemetry and security audit ingress) |
| **Health**        | In-cluster probes for Flow, engine, Convex, Vault, and dependencies  |

There is **no shared database or secret store** between workspace clusters. SSO, RBAC, AI configuration, and run data stay inside the workspace boundary unless you configure **[egress](/docs/data/egress/)**.

#### What this means for you

- **Blast radius** - a compromise or misconfiguration in one workspace does not grant access to another cluster's data plane.
- **Upgrades** - platform and workspace releases can be reconciled independently on the host while preserving per-workspace volumes (see **[Backup and restore](/docs/deployment/backup-and-restore/)**).
- **Compliance narrative** - isolation, encryption at rest, and audit routing map cleanly to this two-layer model (see **[Security](/docs/security/)** and **[Compliance evidence](/docs/compliance/)**).

#### Related documentation

- **[Deployment overview](/docs/deployment/)** - cloud vs self-hosted models and where Fontana runs
- **[Cloud deployment](/docs/deployment/cloud-deployment/)** - managed cloud and customer VPC
- **[Self-hosted deployment](/docs/deployment/self-hosted-deployment/)** - on-premises and private registry
- **[Platform overview](/docs/welcome/platform-overview/)** - product capabilities and technology stack


# Cloud deployment
## Description: Fontana-managed cloud and customer VPC deployment on AWS, Azure, or GCP.
## url: https://fontana-ai.com/docs/deployment/cloud-deployment/

**Cloud deployment** covers production Fontana on public cloud infrastructure: Fontana can **operate the host for you**, or you can run the same stack in **your own VPC** on AWS, Azure, or GCP. In both cases you get pinned releases, cluster-per-workspace isolation, and the same Flow, AI, and compliance surface as every other deployment model.

#### Where Fontana runs

<DeployTargetsDiagram />

| Model                     | Summary                                                                                        |
| ------------------------- | ---------------------------------------------------------------------------------------------- |
| **Fontana-managed cloud** | Fontana operates the host; you consume workspaces and set policy (**preferred**)               |
| **Customer VPC**          | Your AWS, Azure, or GCP account; Fontana supplies pinned releases and the install bundle       |
| **Dedicated cloud host**  | One customer per physical box in cloud or colo; same stack, smaller shared-kernel blast radius |

For on-premises and private-registry installs, see **[Self-hosted deployment](/docs/deployment/self-hosted-deployment/)**.

#### Fontana-managed cloud

**Preferred for most teams.** Fontana provisions and operates the Kubernetes host: platform cluster, workspace clusters, edge TLS, upgrades, backups, and monitoring. You focus on workspaces, policy, and consumption.

| You provide                                                          | Fontana provides                                                |
| -------------------------------------------------------------------- | --------------------------------------------------------------- |
| Business requirements, access policy, approved models and connectors | Host provisioning, Kubernetes lifecycle, pinned upgrades        |
| Identity federation decisions (SSO, MFA) in your IdP                 | Workspace clusters with Zitadel, Vault, and the full app stack  |
| BYOK and connector secrets for your integrations                     | Snapshots before upgrades, rollback path, operational telemetry |
|                                                                       | Cloud account audit perimeter: CloudTrail, Config, GuardDuty + Security Hub, daily volume snapshots (see **[Network and hardening](/docs/security/network-and-hardening/)**) |

Contact Fontana for managed cloud onboarding, host sizing, and diligence materials.

#### Customer VPC

Run Fontana **inside your cloud account** when you need direct control over networking, residency, backup policy, or cloud commit consumption.

| You operate                                                                          | Fontana supplies                                                             |
| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| Cloud account, VPC, subnets, security groups, DNS where applicable                   | Pinned OCI images and verified install bundle from GHCR                      |
| Host backups, change windows, and facility-adjacent cloud governance                 | Same cluster-per-workspace architecture and release cadence as managed cloud |
| **`fontana apply`** on the box (your platform team or Fontana professional services) | Upgrade bundles, snapshot/rollback semantics, observability hooks            |

Your workloads and persistent volumes stay in **your** account boundary. Fontana does not require a shared multi-tenant SaaS data plane for this model.

<Aside type="note" title="Dedicated cloud host">

Some regulated customers choose a **dedicated physical host** in cloud or colo (one customer per box) with the same k3d cluster-per-workspace layout. That reduces shared-kernel exposure compared with multi-tenant hosts while keeping the standard Fontana stack.

</Aside>

#### Releases and supply chain

Cloud production hosts pull **immutable release tags** from GHCR (never floating `latest`). Bundle checksums and image scanning gates apply before a release is promoted. Your operators pin a tag in host config and reconcile with the **[Fontana CLI](/docs/deployment/fontana-cli/)**.

#### Getting started

1. Choose **managed cloud** or **customer VPC** with Fontana sales or your platform team.
2. Provision or receive the host and install the CLI from the published bundle.
3. Configure **`fontana.yaml`** (domain, tenants, release source) and run **`fontana apply`**.
4. Wire SSO in Zitadel, BYOK in Vault, and observability webhooks per **[Observability](/docs/observability/)**.

#### Related documentation

- **[Architecture](/docs/deployment/architecture/)** - platform and workspace topology
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - snapshots and Admin configuration backup
- **[Fontana CLI](/docs/deployment/fontana-cli/)** - apply, tenants, snapshots, rollback
- **[Security](/docs/security/)** - encryption at rest and workspace isolation
- **[Compliance evidence](/docs/compliance/)** - audit and lineage evidence


# Self-hosted deployment
## Description: On-premises Kubernetes, private registry mirrors, and customer-operated Fontana hosts.
## url: https://fontana-ai.com/docs/deployment/self-hosted-deployment/

**Self-hosted deployment** means **you** operate the machine, network, and change control for a Fontana host: on-premises datacenter, private cloud, or an isolated environment without Fontana managing the box day to day. You still run the **same** governed stack, cluster-per-workspace layout, and pinned releases as cloud customers.

#### When to choose self-hosted

- **Data residency or air-gap** requirements that keep the host inside your facility
- **Existing Kubernetes operations** teams who want Fontana as a standard Helm/k3d workload
- **Private registry** policy: mirror Fontana images internally instead of pulling from GHCR at runtime
- **Customer-operated change control**: your CAB, backup, and DR procedures own the host lifecycle

#### What you operate

| Area                | Your responsibility                                                                               |
| ------------------- | ------------------------------------------------------------------------------------------------- |
| **Hardware or VM**  | Sizing, OS patching, disk encryption policy on the underlying volumes                             |
| **Network**         | Ingress (443), SSH for break-glass, firewall rules, corporate DNS                                 |
| **Kubernetes host** | k3d clusters for platform + each workspace; CLI-driven lifecycle                                  |
| **Backups and DR**  | Snapshot retention policy, off-host backup of volume archives where required                      |
| **Upgrades**        | Pin release tags, run **`fontana apply`** in approved windows, validate with **`fontana status`** |

#### What Fontana supplies

- **Pinned release bundles** and container images (GHCR or mirrored into your registry)
- **Fontana CLI** to reconcile platform and tenant clusters from **`fontana.yaml`**
- **Same architecture** as cloud: platform cluster for shared services, one cluster per workspace
- Documentation for observability, identity, compliance, and upgrade/rollback semantics

<Aside type="note" title="Private registry mirror">

Production hosts pull **digest-pinned OCI images** from **GHCR** or from a **private registry mirror** you maintain. Your mirror syncs the same immutable tags promoted by Fontana releases.

</Aside>

#### Dedicated vs shared host

On a **shared self-hosted host**, multiple workspaces use separate clusters on one Linux machine (soft multi-tenancy at the kernel). A **dedicated host** gives one customer the entire box for a smaller blast radius. The cluster-per-workspace isolation model is identical in both cases.

#### Local and lab installs

Developers often run a **local `*.localhost`** install on a laptop or build server for integration testing. That uses the same CLI and topology at smaller scale. Production self-hosted runbooks differ mainly in TLS, DNS, and backup policy, not in application architecture.

#### Getting started

1. Size a host (CPU, RAM, encrypted data disk) per Fontana guidance for your tenant count.
2. Install the **[Fontana CLI](/docs/deployment/fontana-cli/)** from the GHCR release bundle (or your private registry mirror).
3. Author **`fontana.yaml`** with **`baseDomain`**, tenants, and pinned **`source.tag`** for production.
4. Run **`fontana apply`**, then configure SSO, BYOK, and observability in the workspace.

#### Related documentation

- **[Architecture](/docs/deployment/architecture/)** - platform and workspace topology
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - snapshots and Admin configuration backup
- **[Cloud deployment](/docs/deployment/cloud-deployment/)** - managed cloud and customer VPC alternatives
- **[Fontana CLI](/docs/deployment/fontana-cli/)** - install, apply, snapshots, rollback
- **[Security](/docs/security/)** - encryption at rest, isolation, supply chain
- **[Compliance evidence](/docs/compliance/)** - lineage and immutable audit trail


# Backup and restore
## Description: Platform snapshots, upgrade rollback, and Admin configuration backup for workspaces.
## url: https://fontana-ai.com/docs/deployment/backup-and-restore/

Fontana gives you **two complementary recovery paths**: **platform snapshots** for host-level upgrade rollback, and **application configuration backup** in Flow for migrating or restoring workspace settings. Together they cover release safety, operator change control, and configuration portability without mixing secrets into export files.

#### Platform snapshots and rollback

Before a workspace **upgrade**, the host takes an automatic **full-stack snapshot**: the pinned release metadata plus the workspace data volume. That gives you a reversible change-control point when applying a new Fontana version.

| Capability               | What it does for you                                                                   |
| ------------------------ | -------------------------------------------------------------------------------------- |
| **Pre-upgrade snapshot** | Captured automatically when the bundle release changes during upgrade                  |
| **Manual snapshot**      | Operators can create a snapshot on demand before risky changes                         |
| **Rollback**             | Restores the data volume **and** redeploys the pinned release from the chosen snapshot |
| **Retention**            | Configurable how many snapshots to keep per workspace on the host                      |

Rollback is an **operator action** on the host via the **[Fontana CLI](/docs/deployment/fontana-cli/)** (`fontana rollback`). It is the primary path when an upgrade must be undone quickly while keeping workspace data intact.

<Aside type="note" title="What snapshots include">

Platform snapshots capture the **workspace stack state** on the host: persistent volumes for databases, workflow storage, Vault, and related in-cluster data, together with the release version that was running. They support **upgrade safety and disaster recovery at the workspace level**.

</Aside>

#### Application configuration backup (Admin)

In Flow, **Admin → Deployment → Backup/Restore** exports and imports a **composite configuration backup** for your workspace. Use it to clone settings across environments, recover after a configuration mistake, or document approved platform state.

**Export includes** (non-secret application configuration):

- Roles, teams, and users (users merge by email on restore)
- Workflows and workflow documents
- AI agents, skills, skill assignments, and approved model lists
- MCP and external agent interop connections (connection definitions in the backup file)
- Knowledge Graph sources and documents
- Non-secret platform configuration keys

**Export does not include:**

- **BYOK and other Vault secrets** - LLM keys and sensitive credentials stay in Vault; restore them through your secret-management procedures
- **Workflow run datasets** - tabular run data and exports live in workspace storage (see **[Storage and processing](/docs/data/storage-and-processing/)**); use platform snapshots or your DR policy for volume-level recovery

Restore shows a **preview** with per-section choices: skip duplicates, overwrite, or import as duplicate. You control which sections apply before committing.

Requires **`platform:write`** permission (typically administrators).

#### When to use which path

| Scenario                                                 | Recommended path                                                                        |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Undo a failed **platform upgrade**                       | Platform snapshot **rollback** via Fontana CLI                                          |
| Migrate **workflows and AI config** to another workspace | Admin **Backup/Restore** composite export/import                                        |
| Recover **secrets and BYOK**                             | Vault backup procedures (operator runbook); composite export excludes secrets by design |
| Recover **workflow run history and uploads**             | Platform snapshot rollback or volume-level DR per your host policy                      |
| Change control before a **host maintenance window**      | Manual platform snapshot + optional Admin export                                        |

#### Upgrades and change control

Production deploys use **immutable release tags** so every upgrade is reproducible and auditable. The automatic pre-upgrade snapshot is part of that story: you can always roll back to the prior release **and** data state if validation fails after apply.

For day-to-day reconciliation on the host, see **[Fontana CLI](/docs/deployment/fontana-cli/)** (`fontana apply`, `fontana tenant upgrade`, snapshot commands).

#### Related documentation

- **[Fontana CLI](/docs/deployment/fontana-cli/)** - snapshot create, list, and rollback commands
- **[Deployment overview](/docs/deployment/)** - deployment models and release pinning
- **[Architecture](/docs/deployment/architecture/)** - what lives in each workspace cluster
- **[Security](/docs/security/)** - encryption at rest and backup control summary
- **[Compliance evidence](/docs/compliance/)** - immutable audit trail
- **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** - why secrets are outside the composite backup


# Fontana CLI
## Description: Install, configure, and operate the unified fontana CLI; apply, tenants, snapshots, rollback, and recovery after Docker cleanup.
## url: https://fontana-ai.com/docs/deployment/fontana-cli/

The **`fontana` CLI** is the single operator entry point for a Fontana host. You declare desired state in **`fontana.yaml`** and reconcile the box with **`fontana apply`**. The CLI manages the platform k3d cluster (welcome gateway, shared services) and one k3d cluster per workspace tenant (the full Fontana app stack). It ships inside the verified deploy bundle that your host pulls from GHCR (or from your private registry mirror).

<Aside type="note" title="Supported platforms">

Run **`fontana` on the host that runs Docker and your Fontana clusters**. Edge TLS, DNS, and cluster lifecycle reconcile on that same machine.

| Platform                                          | Typical use                                                               |
| ------------------------------------------------- | ------------------------------------------------------------------------- |
| **Linux** (x86_64 or arm64)                       | Production self-hosted servers, private cloud VMs, and customer VPC hosts |
| **macOS** (Apple Silicon or Intel, with Homebrew) | Local and lab **`*.localhost`** installs on a laptop or workstation       |

On Linux production hosts you usually install with administrator privileges; on macOS you install as your user (see **Install** below). **Windows is not supported** as a CLI host today.

If Fontana operates the host for you, you may not need the CLI on your own machines. See **[Cloud deployment](/docs/deployment/cloud-deployment/)** for managed cloud and customer VPC, or **[Self-hosted deployment](/docs/deployment/self-hosted-deployment/)** when your team runs the box.

</Aside>

#### Install

```bash
export FONTANA_GHCR_PAT=…   # read:packages token from Fontana
curl -fsSL <pages-host>/cli/install.sh | sudo -E bash   # Linux
curl -fsSL <pages-host>/cli/install.sh | bash           # macOS (no sudo)
```

The installer links `fontana` into your PATH and seeds **`fontana.yaml`** from the bundle template. All CLI-owned state (releases, snapshots, kubeconfigs) lives under one **`FONTANA_HOME`** directory (`/opt/fontana` on Linux production hosts, `~/.fontana` on macOS).

<Aside type="caution" title="macOS: never sudo fontana">

Run `fontana` as your user. Only the managed `/etc/hosts` block prompts for an admin password during `fontana apply`.

</Aside>

#### Configuration

**`fontana.yaml`** is the sole declarative install config for the host. It is per-host and not checked into your own git (like `.env.local`). The installer copies a starter template on first install.

| Key                    | Purpose                                                                                                                     |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **`baseDomain`**       | Platform hostname (`localhost` for lab installs; your production domain otherwise). Tenants serve at `<name>.<baseDomain>`. |
| **`source.kind`**      | `ghcr` (pull bundle + images from GHCR or your private registry mirror).                                                    |
| **`source.tag`**       | Release tag; pin an immutable `sha-…` tag in production.                                                                    |
| **`snapshots.retain`** | How many full-stack snapshots to keep per tenant.                                                                           |
| **`tenants`**          | Desired workspace list. `fontana apply` creates missing tenants and upgrades existing ones.                                 |

Secrets never belong in `fontana.yaml`. At apply time, export:

- **`FONTANA_GHCR_PAT`** (and optionally **`FONTANA_GHCR_USER`**) so the CLI can pull the release bundle and images
- Convex instance secrets are auto-generated in-cluster by the vault-seed hook

Override the config path with **`--config <path>`** or **`FONTANA_CONFIG`**.

Example production config:

```yaml
version: 1
baseDomain: flow.example.com
source:
  kind: ghcr
  tag: sha-abc123def456
snapshots:
  retain: 3
tenants:
  demo: {}
```

#### `fontana apply`

**`fontana apply`** is the day-to-day reconcile command. It:

1. Ensures the **platform** k3d cluster is running and deploys the gateway + shared charts (Docling, OpenSandbox edge, and related platform services).
2. For each tenant in **`fontana.yaml`**: creates or upgrades the tenant cluster and deploys the Fontana platform Helm release.
3. Refreshes **Caddy** edge routes and the managed **`/etc/hosts`** block once at the end (not mid-tenant, so routes stay stable during maintenance).
4. Runs **fail-closed** post-apply health checks (clusters, workloads, HTTPS probes).

Tenants that exist on the box but are **absent from YAML** are warned only. Removal always requires an explicit **`fontana tenant destroy <name>`**.

##### Upgrade pipeline (per tenant)

When the bundle release version differs from what is deployed in-cluster, the CLI takes a **pre-upgrade snapshot** (stops the cluster briefly, archives the data volume), then redeploys. While upgrading, Caddy serves a static maintenance page for that tenant's hostnames; auth hostnames stay proxied so OIDC discovery keeps working.

Gates you will see in logs:

| Gate                     | Why                                                                                                           |
| ------------------------ | ------------------------------------------------------------------------------------------------------------- |
| **`cluster_wait_ready`** | Blocks kubectl/helm until server-0, serverlb, and API `/readyz` succeed (required after snapshot stop/start). |
| **Vault unseal**         | Vault seals on cluster restart; deploy fails closed until unsealed.                                           |
| **Image import verify**  | After `k3d image import`, the CLI confirms the image ref is present on the node.                              |
| **`apply_status_check`** | Exits non-zero if any component or HTTPS route is unhealthy.                                                  |

Skip pre-upgrade snapshots when re-applying the same release with **`FONTANA_SKIP_SNAPSHOT=1`**.

#### Command reference

##### Top-level

| Command                          | Description                                                                                                                                                                            |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`fontana apply`**              | Reconcile platform + all YAML tenants (create or upgrade).                                                                                                                             |
| **`fontana status`**             | Component health checks; exits non-zero on failure.                                                                                                                                    |
| **`fontana audit`**              | Compare live k3d clusters and edge hostnames to `fontana.yaml`.                                                                                                                        |
| **`fontana resources <tenant>`** | List HTTPS URLs and operator login hints (Flow, Gatus, Convex dashboard, ImmuDB, HyperDX).                                                                                             |
| **`fontana secrets <tenant>`**    | Print deploy credentials for `.env.local` (Convex, Zitadel, Vault keys, ImmuDB WORM audit, automation PAT), plus an **Operator surfaces** block with Gatus and HyperDX browser logins. |

##### Platform

| Command                                                   | Description                                        |
| --------------------------------------------------------- | -------------------------------------------------- |
| **`fontana platform create`**                             | Install platform cluster + gateway (alias: `add`). |
| **`fontana platform upgrade`**                            | Redeploy platform charts from the current release. |
| **`fontana platform destroy`**                            | Delete the platform cluster (tenants untouched).   |
| **`fontana platform start` / `stop` / `status` / `list`** | Cluster lifecycle and listing.                     |

##### Tenant

| Command                                               | Description                                                             |
| ----------------------------------------------------- | ----------------------------------------------------------------------- |
| **`fontana tenant add <name>`**                       | Create tenant cluster + deploy the app stack.                           |
| **`fontana tenant upgrade <name>`**                   | Snapshot (when release changed), then redeploy.                         |
| **`fontana tenant destroy <name>`**                   | Final snapshot (unless skipped), delete cluster + data volume + routes. |
| **`fontana tenant start` / `stop` / `status <name>`** | Cluster lifecycle.                                                      |
| **`fontana tenant list`**                             | List tenants with app and auth hostnames.                               |

##### Snapshots and rollback

| Command                                  | Description                                                               |
| ---------------------------------------- | ------------------------------------------------------------------------- |
| **`fontana snapshot create <name>`**     | Full-stack snapshot (release metadata + data volume tarball).             |
| **`fontana snapshot list [name]`**       | List snapshots for a tenant.                                              |
| **`fontana rollback <name> [snapshot]`** | Restore data volume + redeploy pinned release (default: newest snapshot). |

##### Credentials and URLs

| Command                        | Description                                                                                                                                                                                                           |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`fontana resources <name>`** | HTTPS URLs for Flow, APIs, Gatus, Convex dashboard, HyperDX (`observe-<tenant>`), plus operator login hints (ImmuDB, Gatus, HyperDX).                                                                                 |
| **`fontana secrets <name>`**    | Full `.env.local` credential block for devcontainers (Convex, Zitadel client id, ImmuDB, service keys), followed by the **Operator surfaces** block (Gatus basic-auth and HyperDX admin logins from workspace Vault). |

Use **`fontana resources`** when you need browser URLs and sign-in details for operator consoles. Use **`fontana secrets`** when wiring a local dev loop or copying keys into `.env.local`.

#### State on disk

All CLI-owned paths sit under **`FONTANA_HOME`**:

| Path                                 | Contents                                                          |
| ------------------------------------ | ----------------------------------------------------------------- |
| **`releases/`** / **`current`**      | Downloaded GHCR release bundles for the pinned tag.               |
| **`snapshots/<tenant>/`**            | Snapshot tarballs + `meta.json` (release version, image pins).    |
| **`kubeconfigs/`**                   | Per-cluster kubeconfig files (not `~/.config/k3d`).               |
| **`cluster-ports/<cluster>`**        | Recorded API + HTTP port pairs for recreate after Docker cleanup. |
| **`image-import-stamps/<cluster>/`** | Content-hash stamps for skipped tarball imports.                  |

Each k3d cluster mounts a named Docker volume **`fontana-data-<cluster>`** at the k3s storage path. That volume is the unit for snapshots and rollback.

Host ports are discovered from the live serverlb container (no separate `clusters.json` registry). During snapshot **stop**, edge routing uses **`HostConfig.PortBindings`** so Caddy keeps stable `:808x` mappings while containers are stopped.

#### Recovery after manual Docker cleanup

If you remove k3d containers or images outside **`fontana`** (Docker Desktop prune, manual `docker rm`, and similar), k3d can lose its node records while the **data volume** still holds PVC data. Symptoms include:

- **`No nodes found for given cluster`** during `fontana apply` or after a snapshot restart
- Docling or tarball import errors such as **`content digest … not found`**
- Edge HTTPS failing after a partial apply

The CLI is designed to recover automatically when the data volume remains:

1. Snapshots can proceed even when k3d node records are missing.
2. When a cluster fails to start but **`fontana-data-<cluster>`** still exists, the CLI **recreates** the k3d cluster on the same host ports and re-attaches the data volume.
3. Port mappings are read from the live cluster or from **`${FONTANA_HOME}/cluster-ports/<cluster>`** if containers are gone.
4. **`fontana apply`** treats a tenant with a data volume but no k3d cluster as an **upgrade** (not a fresh create), so you do not lose tenant data.
5. **Image imports**: after recreate, the CLI re-pulls images from your registry and verifies each ref is present on the node before deploy continues.

**What you should do:**

```bash
export FONTANA_GHCR_PAT=…   # required if Docker pruned cached images
fontana apply
```

Vault seals when a cluster is recreated; the deploy pipeline unseals it before hook jobs run. If apply still fails, check **`fontana status`** and logs. A full reset is **`fontana tenant destroy <name>`** then **`fontana apply`** (destroys in-cluster state unless you restore from a snapshot).

<Aside type="note" title="Port assignment after a full wipe">

If both k3d containers **and** the **`cluster-ports/`** file are gone, the CLI may allocate new HTTP ports (you will see a warning in logs). Edge routes and `/etc/hosts` update on the next successful apply. To preserve URLs, restore from snapshot or avoid deleting **`FONTANA_HOME/cluster-ports/`**.

</Aside>

#### Environment variables

| Variable                                         | Purpose                                                  |
| ------------------------------------------------ | -------------------------------------------------------- |
| **`FONTANA_CONFIG`**                             | Path to `fontana.yaml` (alternative to `--config`).      |
| **`FONTANA_HOME`**                               | Root for releases, snapshots, kubeconfigs, port records. |
| **`FONTANA_GHCR_PAT`** / **`FONTANA_GHCR_USER`** | GHCR pull credentials at apply time.                     |
| **`FONTANA_SKIP_SNAPSHOT=1`**                    | Skip pre-upgrade and pre-destroy snapshots.              |
| **`FONTANA_SKIP_MAINTENANCE=1`**                 | Skip maintenance-page Caddy routes during upgrade.       |
| **`FONTANA_LOG_FORMAT=json`**                    | Structured JSON logs (OTel-friendly).                    |
| **`FONTANA_K8S_CLUSTER_READY_TIMEOUT`**          | Seconds to wait for API `/readyz` (default 180).         |

#### Logging

Install and deploy logs use a unified format:

```
2026-06-30T06:11:48Z INFO [tenant-demo] apply: tenant 'demo': exists - upgrading
2026-06-30T06:12:38Z INFO [tenant-demo] cluster: recreating k3d cluster (api=127.0.0.1:6446 http=8082, data volume preserved)
```

Cluster context appears in brackets (`platform`, `tenant-demo`). Components include `apply`, `cluster`, `snapshot`, `source`, `deploy`, and `edge`.

#### Related documentation

- [**Architecture**](/docs/deployment/architecture/): platform and workspace clusters
- [**Backup and restore**](/docs/deployment/backup-and-restore/): snapshots, rollback, Admin configuration backup
- [**Cloud deployment**](/docs/deployment/cloud-deployment/): managed cloud and customer VPC
- [**Self-hosted deployment**](/docs/deployment/self-hosted-deployment/): on-premises and private registry mirrors
- [**Deployment overview**](/docs/deployment/): deployment models and release pinning
- [**Observability**](/docs/observability/): HyperDX onboarding during `fontana apply`


# Security
## Description: Platform security for SOC 2-aligned diligence - workspace isolation, secrets, encryption, network hardening, and supply chain integrity.
## url: https://fontana-ai.com/docs/security/

Fontana is built for organisations that need **provable control** over who can access what, where data lives, and how change is recorded. Your workspace runs inside **your** Kubernetes boundary with dedicated identity, secrets, data stores, and audit paths. The pages below describe the security model you can review during procurement and SOC 2-aligned diligence.

<Aside type="note" title="Certification status">

Fontana is **designed for SOC 2 diligence** with SOC 2-aligned controls across identity, segregation, secrets, audit, and supply chain. Formal certification posture, control narratives, and evidence packs are published on **[Security and governance](/technology/security)** and available under NDA on request.

</Aside>

#### Topics

- **[Workspace isolation](/docs/security/workspace-isolation/)** - cluster-per-tenant boundaries, shared platform services, and soft vs dedicated-host tenancy
- **[Secrets and encryption](/docs/security/secrets-and-encryption/)** - Vault write-only secrets, BYOK, secret rotation, encryption in transit and at rest
- **[Network and hardening](/docs/security/network-and-hardening/)** - TLS transmission boundary, operator host access, NetworkPolicy, pod security
- **[Supply chain](/docs/security/supply-chain/)** - pinned OCI images, bundle integrity verification, and vulnerability scanning
- **[Data retention](/docs/security/data-retention/)** - workflow data, HyperDX operational logs, and WORM audit retention boundaries
- **[Deployment environments](/docs/security/deployment-environments/)** - production-class vs integration/local SOC 2 scope
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - mapped control areas for security questionnaires and audit review
- **[Vendor diligence FAQ](/docs/security/vendor-diligence-faq/)** - answers to recurring procurement and audit questions with links to deep dives

#### Three evidence planes

Security and compliance review spans **platform controls** (this section) and **runtime evidence** (audit and lineage). Keep these separate in questionnaires:

| Plane                               | What it proves                                                        | System of record                                                             | Where to read more                                                    |
| ----------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **Platform security**               | Isolation, secrets, encryption, network, supply chain                 | Host and cluster configuration                                               | This section                                                          |
| **Security audit (WORM)**           | Who did what on the platform; privileged activity                     | **ImmuDB** per workspace (append-only)                                       | **[Security audit (WORM)](/docs/observability/security-audit-worm/)** |
| **Workflow lineage and port audit** | How an output value was derived; validation and edit context on a run | **Workflow file store** (lineage sidecars, port audit items; **not** ImmuDB) | **[Data lineage](/docs/data/data-lineage/)**                          |

**HyperDX** is an **operational search plane**. It may mirror WORM events for faster incident search, but it is **not** the compliance system of record. Retention, reindexing, or TTL expiry in HyperDX does not replace ImmuDB evidence. See **[Data retention](/docs/security/data-retention/)** and **[Operational telemetry](/docs/observability/operational-telemetry/)**.

#### Related documentation

- **[Identity and access](/docs/auth/)** - Zitadel SSO, MFA, SCIM, and application RBAC
- **[Compliance evidence](/docs/compliance/)** - workflow lineage and immutable audit trail deep dive
- **[Observability](/docs/observability/)** - operational telemetry vs security audit routing
- **[Deployment architecture](/docs/deployment/architecture/)** - platform and workspace cluster topology
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - snapshots, rollback, and configuration export


# Workspace isolation
## Description: Cluster-per-tenant segregation, shared platform services, and when to use a dedicated host for hard isolation.
## url: https://fontana-ai.com/docs/security/workspace-isolation/

Each **workspace** (tenant) is a separate Kubernetes cluster with its own Postgres, Convex backend, HashiCorp Vault, Zitadel identity plane, workflow engine file store, and persistent volumes. There is **no shared database or secret store** across workspaces on a host. Cluster-per-tenant is the primary segregation boundary for data, credentials, and compute.

#### Isolation by layer

| Layer                            | Isolation primitive                       | Per workspace? | Notes                                                     |
| -------------------------------- | ----------------------------------------- | -------------- | --------------------------------------------------------- |
| **Control plane**                | Separate k3d (k3s) cluster and API server | Yes            | Kubernetes API bound to localhost; not publicly reachable |
| **Compute**                      | Separate Deployments and Pods             | Yes            | Flow, platform-api, workflow-engine, convex-backend       |
| **Data**                         | Separate Postgres, Convex backend, PVCs   | Yes            | No shared database; volumes on encrypted host storage     |
| **Secrets**                      | Separate in-cluster Vault (KV `fontana`)  | Yes            | Seeded in-cluster; no shared secrets across tenants       |
| **Network**                      | Per-cluster NetworkPolicy default-deny    | Yes            | Scoped egress per workload                                |
| **Identity**                     | Self-hosted Zitadel per workspace         | Yes            | SSO, MFA, and RBAC scoped to the tenant                   |
| **Ingress and TLS**              | Host edge terminates TLS per hostname     | Yes            | Certificate per workspace subdomain (ACME)                |
| **Kernel and container runtime** | Shared host kernel and Docker daemon      | **Shared**     | Soft multi-tenancy boundary (see below)                   |

#### Soft vs hard tenancy

Cluster-per-tenant delivers strong **control-plane, RBAC, secret, and network** isolation. Workspaces on the same physical host still share **one Linux kernel and one container runtime**, so a container escape or kernel vulnerability could theoretically affect other tenants on that box. Fontana treats this as **soft** multi-tenancy.

Mitigations on shared hosts include NetworkPolicy default-deny, Pod Security baseline, per-workspace Vault paths, image vulnerability scanning, and no cross-tenant secret access.

When your risk model requires **hard** isolation, deploy a **dedicated host** with a single workspace (or fewer tenants). The mechanism is the same; the blast radius is smaller because no other tenant workloads share the kernel.

<Aside type="note" title="Dedicated host">

Managed cloud and customer VPC deployments can pin one customer to one box. Self-hosted operators choose host sizing and tenant count the same way. See **[Deployment](/docs/deployment/)** for deployment models.

</Aside>

#### Platform cluster (shared, stateless)

A **platform cluster** on each host runs shared edge and stateless services: welcome gateway routing, HyperDX/OTLP collection, Gatus health probes, Docling document parsing, and OpenSandbox code execution for agent MCP tooling.

The platform cluster does **not** hold workspace databases, Vault secrets, or workflow datasets. Workspace clusters remain the data and secrets boundary.

#### Documented cross-tenant exceptions

Two platform services are intentionally shared across tenants. Fontana documents compensating controls for each:

| Service             | Why shared                         | Primary controls                                                                                                                               |
| ------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **OpenSandbox MCP** | One sandbox control plane per host | Per-tenant bearer auth (ForwardAuth), proxy enforces sandbox ownership fail-closed, digest-pinned base image, sandbox egress restricted to DNS |
| **Docling**         | Stateless PDF and office parsing   | No public hostname; only workflow-engine may call the internal bridge; no durable cross-tenant store                                           |

These exceptions affect **sandbox execution** and **in-flight document parsing** only. They do not expose one workspace's database, Vault, or workflow file store to another.

#### What this means for you

- **Blast radius** - a compromise or misconfiguration in one workspace does not grant access to another cluster's data plane through normal application paths.
- **Evidence** - segregation claims map to separate clusters, Vault instances, and PVCs you can inspect during diligence.
- **Graduation path** - regulated workloads that outgrow soft multi-tenancy move to a dedicated host without changing the product topology.

#### Related documentation

- **[Deployment architecture](/docs/deployment/architecture/)** - platform vs workspace cluster diagram
- **[Secrets and encryption](/docs/security/secrets-and-encryption/)** - per-workspace Vault and encrypted volumes
- **[Network and hardening](/docs/security/network-and-hardening/)** - NetworkPolicy and TLS boundaries
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - tenant isolation row in the control matrix


# Secrets and encryption
## Description: Per-workspace Vault, write-only secret handling, BYOK, and encryption in transit and at rest.
## url: https://fontana-ai.com/docs/security/secrets-and-encryption/

Fontana keeps credentials **out of workflow config and browser responses**. Connector secrets, MCP tokens, LLM API keys (BYOK), and service credentials resolve from **HashiCorp Vault inside your workspace** at execution time. Encryption protects data on the wire and on disk through infrastructure and TLS controls described below.

#### Secrets model

| Secret type                                  | Storage                               | When resolved                    | Browser exposure                          |
| -------------------------------------------- | ------------------------------------- | -------------------------------- | ----------------------------------------- |
| **BYOK (LLM keys)**                          | Vault KV `fontana`                    | Server actions at inference time | Never returned; Admin shows metadata only |
| **Connector credentials**                    | Vault placeholders in workflow config | Workflow engine at run time      | Never embedded in exported config         |
| **MCP OAuth tokens**                         | Vault                                 | Server-side MCP calls            | Never returned to Flow UI                 |
| **Service auth (engine, observability-api)** | Vault-backed OAuth clients            | Backend token introspection      | Not used by browser sessions              |

Vault is **seeded in-cluster** during workspace provisioning. Production deployments do not rely on plaintext seed files on the host or in ConfigMaps as the system of record.

<Aside type="note" title="Write-only from Admin">

When you save a BYOK or connector secret in Admin, Fontana stores the value in Vault and confirms success without echoing the secret back. Rotation replaces the stored value; prior runs do not embed historical keys in workflow JSON.

</Aside>

#### BYOK scopes

LLM credentials support **org**, **team**, and **person** scopes so you can align keys with procurement and chargeback. Resolution is fail-closed: if no key exists at the required scope, the call does not silently fall back to another tenant's or Fontana-owned credentials.

See **[Bring Your Own Key (BYOK)](/docs/ai/byok/)** for Admin paths and gateway approval.

#### Secret rotation

Fontana stores secret **values** in Vault; rotation replaces the stored value without embedding secrets in workflow JSON or returning them to the browser.

| Secret class | Who rotates | Typical procedure |
| ------------ | --------- | ----------------- |
| **BYOK (LLM keys)** | Your admin in **Admin → Models → API Keys** | Save a new key at the same scope; old value is overwritten in Vault |
| **Connector and MCP credentials** | Your admin in Flow Admin or node settings | Update the Vault-backed secret; next run resolves the new value |
| **SCIM bearer token** | Your identity team in **Zitadel Management Console** | Rotate SCIM token in Zitadel and update the corporate IdP SCIM app configuration |
| **Service OAuth clients** (engine, observability-api) | Platform operator during upgrade or runbook | Rotated through Vault-backed provisioning as part of controlled deploys |
| **Vault root / unseal material** | Platform operator | Runbook-driven; not exposed to workspace users |

After BYOK or connector rotation, **historical workflow config does not retain prior secret values**. Runs that already completed used the key effective at execution time; audit metadata may record that a key was updated without storing the secret itself.

Composite configuration backup **does not** include BYOK values. After disaster recovery, restore secrets from a **Vault Raft snapshot** in addition to application backup. See **[Backup and restore](/docs/deployment/backup-and-restore/)**.

#### Encryption in transit

For the full hop-by-hop transmission table (public TLS vs in-cluster HTTP), see **[Network and hardening](/docs/security/network-and-hardening/#transmission-boundary-tls)**.

| Hop                                    | Protection                                                               |
| -------------------------------------- | ------------------------------------------------------------------------ |
| Internet to your workspace hostname    | **TLS** at the host edge (ACME-managed certificates)                     |
| Edge to in-cluster ingress             | HTTP on localhost or bridge (same host)                                  |
| In-cluster service to service          | HTTP on cluster network; **NetworkPolicy** restricts which pods may talk |
| Pods to external LLM or connector APIs | **HTTPS** to provider endpoints (egress allow on port 443)               |

Compensating controls for plain HTTP inside the cluster: NetworkPolicy default-deny, cluster-per-tenant separation, and no multi-tenant pod sharing on the data plane. Full table: **[Transmission boundary](/docs/security/network-and-hardening/#transmission-boundary-tls)**.

#### Encryption at rest

Production hosts use **encrypted block storage** for the dedicated data volume that backs all workspace PVCs: Postgres, Convex backend data, workflow-engine file store, and Vault Raft.

Fontana does not apply a separate application-layer field encryption on workflow datasets. Protection relies on **infrastructure encryption** plus **workspace isolation** so only your cluster's workloads mount your volumes.

Workflow run data (uploads, Arrow datasets, audit and lineage sidecars, exports) persists on a **dedicated ReadWriteOnce PVC** per workspace, retained across Helm upgrades.

#### Backup and disaster recovery

- **Composite configuration backup** (Admin) exports Convex configuration; it **does not** include BYOK secret values.
- **Vault Raft snapshot** is the authoritative restore path for secrets and BYOK after disaster recovery.
- **Volume-level procedures** cover Postgres, Convex, and workflow-engine PVCs; see **[Backup and restore](/docs/deployment/backup-and-restore/)**.

#### Related documentation

- **[Workspace isolation](/docs/security/workspace-isolation/)** - separate Vault per workspace
- **[Identity and access](/docs/auth/)** - Zitadel and RBAC (distinct from BYOK)
- **[Secret rotation](/docs/security/secrets-and-encryption/#secret-rotation)** - BYOK, connectors, SCIM tokens
- **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** - security events when secrets and admin settings change
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - secrets and encryption rows


# Network and hardening
## Description: Perimeter exposure, TLS, NetworkPolicy, pod security, and backend service authentication.
## url: https://fontana-ai.com/docs/security/network-and-hardening/

Fontana reduces attack surface through **narrow perimeter exposure**, **default-deny networking inside each workspace cluster**, and **workload hardening** (dedicated service accounts, resource limits, and Vault Agent injection). Backend services validate opaque tokens through **RFC 7662 introspection** using dedicated OAuth clients.

#### Perimeter and exposure

| Surface                | Typical production posture                                                                                                                                         |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Public HTTPS**       | Host edge accepts **80 and 443** (ACME + redirect) only; no standing SSH ingress on Fontana-managed cloud                                                          |
| **Kubernetes API**     | Bound to **127.0.0.1** on the host; remote administration reaches the API through an authenticated host session (AWS SSM Session Manager on Fontana-managed cloud) |
| **Workspace backends** | Same-origin path routing behind one hostname per workspace; no cross-service browser CORS in tenant mode                                                           |
| **Shared sandbox MCP** | Public HTTPS with **ForwardAuth**; unauthenticated requests fail closed before reaching the sandbox server                                                         |

Inside a workspace, the browser talks to Flow, Convex, platform-api, and the workflow engine through **one origin**. Traefik strips `/_*` path prefixes to the correct in-cluster service.

#### Transmission boundary (TLS)

Fontana terminates **TLS at the public edge**. Traffic inside the host and inside each workspace cluster is plain HTTP on restricted network paths. Compensating controls are **cluster-per-tenant separation**, **NetworkPolicy default-deny**, and **no multi-tenant pod sharing** on the data plane.

| Hop                              | Protocol | TLS termination            | Notes                                                               |
| -------------------------------- | -------- | -------------------------- | ------------------------------------------------------------------- |
| Internet to host edge            | HTTPS    | Host edge (ACME)           | Public Flow, Convex, engine, and API paths                          |
| Edge to in-cluster ingress       | HTTP     | None (localhost or bridge) | Same physical host                                                  |
| Ingress to application pod       | HTTP     | None                       | Cluster network; NetworkPolicy scoped                               |
| Pod to in-cluster Vault          | HTTP     | None                       | NetworkPolicy scoped to Vault clients                               |
| Pod to external LLM or connector | HTTPS    | Provider                   | Egress allow on port 443 from approved workloads                    |
| Internet to shared sandbox MCP   | HTTPS    | Host edge                  | ForwardAuth before MCP server; unauthenticated requests fail closed |

Single-origin routing means the browser never needs cross-service CORS in tenant mode: every call stays on one workspace hostname with path prefixes such as `/_convex`, `/_platform`, and `/_engine`.

#### Operator and host access

Platform **operators** (your Fontana deployment team or managed-service administrators) reach the host through controlled paths:

| Access type                  | Typical production posture                                                                                                                                                                                                               |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Public Internet**          | HTTPS **80 and 443** only for application traffic; the security group carries no SSH rule on Fontana-managed cloud                                                                                                                       |
| **Shell access**             | **AWS SSM Session Manager** on Fontana-managed cloud (authenticated, audited sessions); SSH is a documented break-glass path enabled only per change window                                                                              |
| **Kubernetes API**           | Bound to **127.0.0.1** on the host; remote administration reaches the API through the same authenticated host session                                                                                                                    |
| **Durable platform changes** | Reconciled through the **Fontana CLI** (`fontana apply`, pinned tenant upgrades) and **Terraform** host provisioning automation                                                                                                          |
| **Break-glass diagnostics**  | Read-only host diagnostics where your runbook allows; production durable changes still flow through CLI and infrastructure as code                                                                                                       |
| **Cluster RBAC**             | Namespaced operator roles (read-only, deploy, secret-admin) are versioned in git and applied at bootstrap; drift is checked with `kubectl diff` against the committed manifests                                                          |
| **Operator dashboards**      | Gatus (HTTP basic auth), HyperDX (native login provisioned via API), and the Convex dashboard (admin key) each require authentication before any data; credentials live in workspace Vault and surface through `fontana secrets <tenant>` |

Workspace **end users** never receive host or cluster credentials. They authenticate through **Zitadel OIDC** in the browser and are authorised by **application RBAC** in Flow.

See **[Deployment environments](/docs/security/deployment-environments/)** for which host classes are in SOC 2 scope.

#### Cloud account perimeter (Fontana-managed AWS)

On Fontana-managed cloud, the AWS account surrounding the production host carries its own Terraform-managed control set, complementing the in-cluster stack (Vault, NetworkPolicy, ImmuDB WORM audit):

| Control                        | What it provides                                                                                                       |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| **Organization CloudTrail**    | Every operator sign-in and cloud API call is recorded to a dedicated audit bucket with log-file validation             |
| **AWS Config rules**           | Continuous drift detection: security-group ports, volume encryption, public bucket exposure, trail status              |
| **GuardDuty and Security Hub** | Threat detection plus FSBP and CIS benchmark checks; high and critical findings route to the security mailbox          |
| **Daily volume snapshots**     | The persistent data volume is snapshotted daily with a rolling retention window (complements platform snapshots)       |
| **SSO-only operator identity** | Operators authenticate through IAM Identity Center with least-privilege permission sets; no long-lived IAM access keys |
| **WORM evidence bucket**       | S3 Object Lock in compliance mode archives integrity-scan evidence off-box for the retention period                    |

Customer VPC deployments can apply the same pattern in their own account; the audit surface (CloudTrail, Config, GuardDuty) is standard AWS tooling driven by infrastructure as code.

#### NetworkPolicy (default deny)

Each workspace cluster applies **default-deny** NetworkPolicy in the application namespace, then **scoped allow** rules per workload:

- DNS resolution for allowed pods
- Postgres, Vault, and Convex only from designated clients
- Egress to external HTTPS for approved connector and LLM traffic from engine and Convex paths
- Observability and health probes on documented ports

The goal is **least privilege between pods**: a compromised workload cannot reach arbitrary cluster services or the Internet without an explicit allow rule.

#### Pod and workload hardening

| Control                       | Approach                                                                                              |
| ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Service accounts**          | Per-workload Kubernetes ServiceAccounts; no reliance on the default SA                                |
| **Pod Security**              | Baseline profile enforced on tenant workloads                                                         |
| **Resources**                 | CPU and memory limits on Deployments                                                                  |
| **Secret injection**          | Vault Agent sidecars where required; no long-lived secrets in plain env ConfigMaps                    |
| **Single-replica file store** | Workflow engine uses Recreate strategy on ReadWriteOnce PVC so only one pod mounts run data at a time |

#### Service authentication

Human users authenticate through **Zitadel OIDC** in the browser. Backend services use **dedicated OAuth clients** and **token introspection**:

- **Workflow engine** validates opaque tokens before executing privileged paths
- **observability-api** gates security audit ingest and operational routing
- Misconfigured introspection or discovery causes **fail-closed** errors rather than anonymous access

See **[Identity and access](/docs/auth/)** for SSO, MFA, SCIM, and application RBAC.

#### Identity events in audit

Sign-in, MFA challenges, role changes, and deprovisioning feed the **immutable security audit trail** through observability-api. See **[Security audit (WORM)](/docs/observability/security-audit-worm/)**.

#### Related documentation

- **[Workspace isolation](/docs/security/workspace-isolation/)** - per-cluster network boundary
- **[Secrets and encryption](/docs/security/secrets-and-encryption/)** - TLS and in-cluster HTTP compensating controls
- **[Deployment architecture](/docs/deployment/architecture/)** - single-origin routing table
- **[Deployment environments](/docs/security/deployment-environments/)** - production-class SOC 2 scope
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - network and authentication rows


# Supply chain
## Description: Immutable OCI images, deploy bundle integrity, and vulnerability scanning for SOC 2-aligned change control.
## url: https://fontana-ai.com/docs/security/supply-chain/

Production Fontana deployments consume **pre-built OCI images and a verified deploy bundle** from your registry (typically GHCR). Application source code is **compiled inside images** in CI; production hosts pull artifacts only. That boundary supports SOC 2-aligned **change management** and **vulnerability management** without running build toolchains on customer data planes.

#### Artifact flow

```text
CI (GitHub Actions, release branch only)
  → OCI images (immutable sha-<git> tags)
  → Deploy bundle (Helm chart, fontana CLI, rendered values, SHA256SUMS)
  → Trivy CRITICAL gate
  → Host pulls bundle + images via fontana CLI
```

Release artifacts are built **only from the protected release branch** after pull-request review and required status checks pass; feature-branch pushes run tests without publishing images or bundles.

#### Integrity controls

| Control                            | What you get                                                                                                      |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Immutable image tags**           | Deploys reference `sha-<git>` tags; branch tags are convenience only                                              |
| **Bundle checksums**               | `SHA256SUMS` in the deploy bundle; installer verifies before activating a release                                 |
| **Pinned upgrades**                | `fontana tenant upgrade` applies a specific bundle version; automatic snapshot before upgrade                     |
| **Rollback**                       | `fontana rollback <tenant> <snapshot-id>` restores data and release state                                         |
| **Artifact-only production hosts** | Production hosts pull digest-pinned OCI images and the deploy bundle; no git or build toolchain on the data plane |

<Aside type="note" title="Connector images">

PyAirbyte connector images are mirror-pinned and digest-pinned on GHCR with the same **Trivy CRITICAL** scanning gate as platform images. See **[Data ingress](/docs/data/ingress/)** for connector boundaries.

</Aside>

#### Vulnerability management

CI runs **Trivy** against built images. **CRITICAL** findings fail the pipeline until remediated or explicitly waived under your change process. That gate is part of the SOC 2-aligned supply-chain control set documented for diligence.

Source dependencies are gated on the same pipeline:

- **`pnpm audit`** fails the build on any high or critical advisory in the dependency tree
- **CodeQL** static analysis runs on every push and pull request to the release branch
- **Dependabot** keeps lockfile and GitHub Actions dependencies current
- A **public-function inventory diff** fails the build when the backend API surface changes without a reviewed baseline update

#### Change control and audit trail

- **Manual pinned releases** on production hosts reference immutable `sha-<git>` tags
- **Automatic full-stack snapshot** before each tenant upgrade (release + data volumes)
- **Terraform** for host provisioning; durable platform changes reconcile through `fontana apply` and infrastructure as code

Snapshot history on the host plus immutable GHCR tags form the **deploy audit trail** alongside application security audit in ImmuDB.

#### Related documentation

- **[Fontana CLI](/docs/deployment/fontana-cli/)** - install, apply, upgrade, rollback commands
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - snapshots and recovery
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - supply chain and change control rows
- **[Security and governance](/technology/security)** - diligence pack and certification posture


# SOC 2 control summary
## Description: Mapped control areas for SOC 2-aligned diligence - identity, segregation, secrets, audit, supply chain, and availability.
## url: https://fontana-ai.com/docs/security/soc2-controls/

Production deployments are **designed for SOC 2 diligence** with controls across identity, segregation, secrets, audit, supply chain, and availability. Use this page as a **questionnaire index** (control matrix and Trust Service Criteria map). For **question-and-answer** copy you can paste into vendor forms, see **[Vendor diligence FAQ](/docs/security/vendor-diligence-faq/)**. Detailed narratives and evidence exports are available on **[Security and governance](/technology/security)** and under NDA from Fontana.

<Aside type="caution" title="Not a certification claim">

This summary describes **SOC 2-aligned controls** in the product and deployment model. It is not a substitute for Fontana's formal certification status or your auditor's test procedures. Contact Fontana for control narratives, subprocessors, and evidence under NDA.

</Aside>

#### Control matrix

| Topic                             | Approach                                                                                                                                                                                                        | Deep dive                                                                                                                     |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Authentication**                | Self-hosted Zitadel per workspace: OIDC, SSO federation, MFA (TOTP + WebAuthn)                                                                                                                                  | **[Identity providers](/docs/auth/identity-providers/)**                                                                      |
| **Provisioning**                  | Inbound SCIM v2 through workspace Zitadel; role assignment via Zitadel project roles                                                                                                                            | **[Identity providers](/docs/auth/identity-providers/)**, **[Roles and RBAC](/docs/auth/roles-and-rbac/)**                    |
| **Authorization**                 | Application RBAC in Convex with team scoping and permission registry                                                                                                                                            | **[Roles and RBAC](/docs/auth/roles-and-rbac/)**                                                                              |
| **Service authentication**        | Dedicated OAuth clients for workflow-engine and observability-api; RFC 7662 token introspection                                                                                                                 | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Secrets**                       | Per-workspace in-cluster Vault (KV `fontana`); BYOK for AI; connector secrets via Vault at runtime                                                                                                              | **[Secrets and encryption](/docs/security/secrets-and-encryption/)**                                                          |
| **Encryption in transit**         | TLS at public edge; documented transmission boundary; in-cluster HTTP with NetworkPolicy compensating controls                                                                                                  | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Encryption at rest**            | Encrypted data volume backing all workspace PVCs                                                                                                                                                                | **[Secrets and encryption](/docs/security/secrets-and-encryption/)**                                                          |
| **Workspace isolation**           | Cluster-per-tenant control plane, data, secrets, and identity                                                                                                                                                   | **[Workspace isolation](/docs/security/workspace-isolation/)**                                                                |
| **Network**                       | Host security group: 80 + 443 only (shell via SSM Session Manager); k8s API localhost-only; NetworkPolicy default-deny                                                                                          | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Cloud account perimeter**       | Organization CloudTrail, AWS Config rules, GuardDuty + Security Hub with findings routing, SSO-only operator identity                                                                                           | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Pod and workload hardening**    | Per-workload ServiceAccounts; Pod Security baseline; limits; Vault Agent sidecars                                                                                                                               | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Audit (identity and cluster)**  | WORM ImmuDB: Zitadel identity events and Kubernetes API audit (automated ingest)                                                                                                                                | **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**                                                          |
| **Security alerting**             | Bundled HyperDX alert rules on Kubernetes audit events (auth failures, secret access, destructive verbs, ClusterRoleBinding changes) with WORM evidence per alert                                               | **[Operational telemetry](/docs/observability/operational-telemetry/)**                                                       |
| **Audit (application admin)**     | WORM via Convex `audit_log` log stream (ingest shipped; mutation producers rolling out by release)                                                                                                              | **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**                                                          |
| **Audit (workflow, WORM target)** | Privileged runs and security-tagged connector access → Convex `audit_log` (target path)                                                                                                                         | **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**                                                          |
| **Audit (workflow lineage)**      | Lineage sidecars and port audit items on **workflow file store only**; answers calculation provenance, **not** platform WORM                                                                                    | **[Data lineage](/docs/data/data-lineage/)**, **[Compliance evidence](/docs/compliance/)**                                    |
| **Operational telemetry**         | HyperDX search and dashboards; **mirror only** for audit events; ImmuDB is WORM system of record; collector-side redaction of credentials in log bodies                                                         | **[Operational telemetry](/docs/observability/operational-telemetry/)**, **[Data retention](/docs/security/data-retention/)** |
| **Backup and recovery**           | Automatic snapshot before tenant upgrade; `fontana rollback`; daily volume snapshots on managed cloud; Vault Raft for secrets DR                                                                                | **[Backup and restore](/docs/deployment/backup-and-restore/)**                                                                |
| **Supply chain**                  | Immutable GHCR `sha-<git>` tags, bundle SHA256SUMS, Trivy CRITICAL gate, dependency audit + CodeQL gates in CI                                                                                                  | **[Supply chain](/docs/security/supply-chain/)**                                                                              |
| **Change control**                | Release artifacts built from the protected release branch only; manual pinned releases; Terraform for host provisioning; SSM read/debug only on prod                                                            | **[Supply chain](/docs/security/supply-chain/)**, **[Fontana CLI](/docs/deployment/fontana-cli/)**                            |
| **Availability monitoring**       | In-cluster Gatus probes for platform and tenant services; status page behind HTTP basic auth with Vault-stored credentials                                                                                      | **[Operational telemetry](/docs/observability/operational-telemetry/)**                                                       |
| **AI governance**                 | BYOK, approved gateways/models, ZDR where supported, tool-call audit                                                                                                                                            | **[BYOK](/docs/ai/byok/)**, **[Zero Data Retention](/docs/ai/zdr/)**                                                          |
| **Data retention**                | Separate retention for workflow file store, HyperDX operational logs, and ImmuDB WORM ledger                                                                                                                    | **[Data retention](/docs/security/data-retention/)**                                                                          |
| **Deployment environments**       | Production-class hosts in SOC 2 scope; integration/local for engineering only                                                                                                                                   | **[Deployment environments](/docs/security/deployment-environments/)**                                                        |
| **Operator access**               | Narrow host perimeter; localhost-bound cluster API; CLI-driven durable changes; Kubernetes operator roles versioned in git and applied at bootstrap; authenticated operator dashboards (Gatus, HyperDX, Convex) | **[Network and hardening](/docs/security/network-and-hardening/)**                                                            |
| **Secret rotation**               | Vault-backed rotation for BYOK, connectors, SCIM tokens, and service credentials                                                                                                                                | **[Secrets and encryption](/docs/security/secrets-and-encryption/)**                                                          |

#### Trust Service Criteria mapping (high level)

Auditors often map evidence to AICPA Trust Service Criteria. Fontana's deployment model supports review across common areas:

| TSC area                      | How Fontana addresses it                                                                                                                         |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Security (CC)**             | Identity, RBAC, network segmentation, secrets, vulnerability scanning                                                                            |
| **Availability (A)**          | Gatus health checks, snapshots, rollback, durable PVCs                                                                                           |
| **Processing integrity (PI)** | Deterministic workflow execution, lineage, validation audit items                                                                                |
| **Confidentiality (C)**       | Tenant isolation, encryption, write-only secrets, BYOK                                                                                           |
| **Privacy (P)**               | Customer-controlled retention and egress; separate workflow, HyperDX, and WORM stores. See **[Data retention](/docs/security/data-retention/)**. |

Your auditor may require **operating effectiveness** testing on **your** deployment instance, not only design documentation.

#### Audit operating status

The control matrix uses separate rows because **not every audit path is at the same maturity**. Questionnaires often ask whether controls **operate** in production, not only whether the architecture supports them.

| Evidence type                                                                 | WORM ledger?                 | Status                                                                                                           |
| ----------------------------------------------------------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| **Zitadel identity** (sign-in, MFA, role changes, SCIM lifecycle)             | Yes                          | **Automated** - observability-api polls Zitadel Events API                                                       |
| **Kubernetes API** (auth failures, secret access, destructive verbs)          | Yes                          | **Automated** - cluster audit log → audit-ingest                                                                 |
| **Convex admin mutations** (RBAC, BYOK, team membership, deployment settings) | Yes                          | **Ingest shipped**; individual mutation producers **roll out by release**                                        |
| **Workflow engine** (privileged runs, sensitive connector access)             | Target                       | **Target path** - engine → Convex `audit_log` (not direct ImmuDB)                                                |
| **Workflow lineage and port audit items**                                     | **No (workflow file store)** | **Not WORM** - `.lineage.json` / `.audit.json` sidecars and canvas port audit items; calculation provenance only |
| **HyperDX (including audit mirror)**                                          | **No (search plane)**        | **Not system of record** - operational search and shorter TTL; **ImmuDB** holds compliance evidence              |
| **Audit logs → Chat** (LLM debug in Flow)                                     | No                           | Product feature; RBAC-gated, not compliance ledger                                                               |

Full producer map: **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**.

#### Environment scope

SOC 2 evidence collection targets **production-class hosts** only. Integration and local developer environments are **out of scope** for regulated customer data and formal change control.

See **[Deployment environments](/docs/security/deployment-environments/)** for environment classes, what production-class means, and why integration/local boxes must not host regulated tenants.

#### Requesting evidence

For architecture diagrams, subprocessors, DPA terms, penetration test summaries, and exported control snapshots, use **[Security and governance](/technology/security)** or contact Fontana for the diligence pack under NDA.

Fontana maintains a structured **auditor evidence index** that maps each control row above to dated artifacts: control snapshots (`fontana soc2 snapshot`), committed operator RBAC manifests, Kubernetes audit dashboards and alert history, WORM integrity-scan records, and backup drill results. Auditors receive this index with the diligence pack.

#### Related documentation

- **[Vendor diligence FAQ](/docs/security/vendor-diligence-faq/)** - recurring procurement and audit Q&A
- **[Security overview](/docs/security/)** - topics index and three evidence planes
- **[Compliance evidence](/docs/compliance/)** - workflow lineage and immutable audit trail narrative
- **[Deployment](/docs/deployment/)** - where Fontana runs and upgrade mechanics


# Vendor diligence FAQ
## Description: Answers to recurring security and SOC 2-aligned procurement questions, with links to detailed platform documentation.
## url: https://fontana-ai.com/docs/security/vendor-diligence-faq/

Use this page when you map a **vendor security questionnaire**, SIG-style review, or customer audit request to Fontana. Each answer is short and points to the canonical doc for depth. For the **control matrix** and Trust Service Criteria map, start with **[SOC 2 control summary](/docs/security/soc2-controls/)**.

<Aside type="caution" title="Public summary only">

These answers describe **SOC 2-aligned controls** in the product and deployment model. They are not a certification claim, penetration test report, or substitute for your auditor's test procedures on **your** production instance. Subprocessor registers, DPA terms, architecture exports, and formal evidence packs are available on **[Security and governance](/technology/security)** under NDA.

</Aside>

#### How to use this page

| If your questionnaire asks about… | Start here                                                     | Full control map                                                     |
| --------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------- |
| Access, SSO, MFA, RBAC            | [Identity and access](#identity-and-access)                    | **[SOC 2 control summary](/docs/security/soc2-controls/)**           |
| Encryption, secrets, keys         | [Encryption and secrets](#encryption-and-secrets)              | **[Secrets and encryption](/docs/security/secrets-and-encryption/)** |
| Multi-tenancy, isolation          | [Tenant isolation](#tenant-isolation-and-network-security)     | **[Workspace isolation](/docs/security/workspace-isolation/)**       |
| Audit logs, SIEM, retention       | [Logging and audit](#logging-audit-and-monitoring)             | **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** |
| Backups, DR, uptime               | [Backup and availability](#backup-recovery-and-availability)   | **[Backup and restore](/docs/deployment/backup-and-restore/)**       |
| SDLC, vuln scan, releases         | [Change and supply chain](#change-management-and-supply-chain) | **[Supply chain](/docs/security/supply-chain/)**                     |
| LLM, BYOK, AI data use            | [AI governance](#ai-governance-and-data-handling)              | **[BYOK](/docs/ai/byok/)**, **[Zero Data Retention](/docs/ai/zdr/)** |

Questionnaires rarely match this outline exactly. Use the **[SOC 2 control summary](/docs/security/soc2-controls/)** matrix when you need a row-per-control mapping.

#### Certification and program status

**Is Fontana SOC 2 certified?**

Fontana is **designed for SOC 2 diligence** with mapped controls across identity, segregation, secrets, audit, supply chain, and availability. Formal certification posture and programme status are published on **[Security and governance](/technology/security)**. Contact Fontana for control narratives and evidence under NDA.

**Does Fontana support ISO 27001 or GDPR review?**

Security management controls and GDPR-aligned data-processing safeguards are part of the diligence programme described on **[Security and governance](/technology/security)**. This docs site focuses on **technical controls** you can verify on your deployment; legal terms (DPA, subprocessors) are provided under NDA.

**Can our auditor test controls on our instance?**

Yes. Your auditor may require **operating effectiveness** testing on **your** production-class deployment, not only design documentation. Environment scope, pinned releases, and evidence boundaries are documented in **[Deployment environments](/docs/security/deployment-environments/)**.

#### Deployment models and data residency

**Where does Fontana run?**

Fontana runs on **Kubernetes (k3d) on a host you control or Fontana operates**: managed cloud, dedicated single-tenant host, customer VPC, or self-hosted production. Each workspace is a separate cluster with its own data stores. See **[Deployment](/docs/deployment/)** and **[Deployment architecture](/docs/deployment/architecture/)**.

**Who holds customer data?**

Customer workflow data, configuration, secrets, and audit evidence live **inside your workspace cluster** on encrypted persistent volumes. Fontana does not commingle workspace databases or Vault stores across tenants on a host. See **[Workspace isolation](/docs/security/workspace-isolation/)**.

**Which environments are in scope for SOC 2 evidence?**

**Production-class** hosts where regulated customer data lives are in scope. Integration and local developer environments are **out of scope** for regulated workloads and formal evidence collection. See **[Deployment environments](/docs/security/deployment-environments/)**.

#### Identity and access

**Do you support SSO and MFA?**

Each workspace runs **self-hosted Zitadel** with OIDC, enterprise SSO federation, and MFA (TOTP and WebAuthn). End-user sign-in flows through your configured identity provider where applicable. See **[Identity providers](/docs/auth/identity-providers/)**.

**Can we provision and deprovision users automatically?**

Yes. **Inbound SCIM v2** through workspace Zitadel supports automated user lifecycle. Role assignment uses Zitadel project roles mapped to application permissions. See **[Identity providers](/docs/auth/identity-providers/)** and **[Roles and RBAC](/docs/auth/roles-and-rbac/)**.

**How is authorization enforced in the application?**

**Application RBAC** in Convex uses a permission registry with **team scoping**. Admin actions require explicit permissions (for example `platform:write` for deployment backup). See **[Roles and RBAC](/docs/auth/roles-and-rbac/)**.

**How do backend services authenticate?**

Dedicated OAuth clients for **workflow-engine** and **observability-api** use **RFC 7662 token introspection**. Service credentials resolve from Vault; they are not embedded in browser sessions. See **[Network and hardening](/docs/security/network-and-hardening/)**.

**How is operator access to production controlled?**

Production hosts expose **HTTP and HTTPS** only; shell access uses **AWS SSM Session Manager** on Fontana-managed cloud (no standing SSH ingress). The Kubernetes API binds to **localhost**. Durable platform changes reconcile through the **Fontana CLI** and **Terraform** for host provisioning. Kubernetes operator roles (read-only, deploy, secret-admin) are **versioned in git** and applied at bootstrap, with drift checked against the committed manifests. Operator dashboards (Gatus, HyperDX, Convex) require **authentication** with credentials held in workspace Vault. See **[Network and hardening](/docs/security/network-and-hardening/)**.

#### Encryption and secrets

**How are secrets stored?**

Connector credentials, MCP tokens, BYOK LLM keys, and service credentials live in **per-workspace HashiCorp Vault** (KV `fontana`). Admin saves secrets **write-only**: values are never echoed back to the browser. See **[Secrets and encryption](/docs/security/secrets-and-encryption/)**.

**Is data encrypted in transit?**

**TLS** terminates at the public edge (host Caddy). In-cluster HTTP is bounded by documented transmission boundaries and **NetworkPolicy** compensating controls. See **[Network and hardening](/docs/security/network-and-hardening/)**.

**Is data encrypted at rest?**

Workspace persistent volumes sit on **encrypted host storage** backing Postgres, Convex, workflow file store, Vault, and ImmuDB data. See **[Secrets and encryption](/docs/security/secrets-and-encryption/)**.

**Do you support customer-managed keys for AI?**

Yes. **BYOK** stores LLM API keys in Vault at **org, team, or person** scope. Resolution is fail-closed: missing keys do not silently fall back to shared credentials. See **[BYOK](/docs/ai/byok/)**.

**How does secret rotation work?**

Rotation replaces values in Vault without embedding secrets in workflow JSON or export files. BYOK, connectors, SCIM tokens, and service credentials follow Vault-backed rotation patterns documented in **[Secrets and encryption](/docs/security/secrets-and-encryption/)**.

#### Tenant isolation and network security

**How are tenants isolated?**

**Cluster-per-tenant**: separate k3d cluster, Postgres, Convex, Vault, Zitadel, workflow storage, and NetworkPolicy per workspace. There is no shared database or secret store across workspaces. See **[Workspace isolation](/docs/security/workspace-isolation/)**.

**Is isolation physical or logical?**

Default isolation is **logical** (separate clusters on a shared host kernel). For **hard** isolation, deploy a **dedicated host** with one workspace (or fewer tenants). The mechanism is the same; blast radius is smaller. See **[Workspace isolation](/docs/security/workspace-isolation/)**.

**What network controls apply inside the cluster?**

**NetworkPolicy default-deny** in the workspace namespace, per-workload ServiceAccounts, Pod Security baseline, resource limits, and Vault Agent sidecars. See **[Network and hardening](/docs/security/network-and-hardening/)**.

**Are there shared platform services?**

A **platform cluster** on each host runs stateless edge services (welcome gateway, shared OpenSandbox MCP, document parsing). It does **not** hold workspace databases or secrets. Documented cross-tenant exceptions include compensating controls in **[Workspace isolation](/docs/security/workspace-isolation/)**.

#### Logging, audit, and monitoring

**Do you maintain an immutable audit trail?**

Yes. Security-relevant events append to **ImmuDB** per workspace (WORM: write once, read many). **observability-api** is the **only** writer. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**.

**What events are in the WORM ledger today?**

| Source                                                                 | Status                                                       |
| ---------------------------------------------------------------------- | ------------------------------------------------------------ |
| Zitadel identity (sign-in, MFA, SCIM lifecycle)                        | **Automated**                                                |
| Kubernetes API audit (auth failures, secret access, destructive verbs) | **Automated**                                                |
| Convex admin mutations (RBAC, BYOK, teams, deployment settings)        | **Ingest shipped**; individual producers roll out by release |
| Workflow engine (privileged runs, sensitive connector access)          | **Target path** via Convex `audit_log`                       |

Full producer map: **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** and **[SOC 2 control summary](/docs/security/soc2-controls/)** (audit operating status).

**Is HyperDX the compliance audit system of record?**

**No.** HyperDX is the **operational search plane** (logs, dashboards, shorter TTL). It may **mirror** WORM events for incident search, but **ImmuDB** holds compliance evidence. See **[Operational telemetry](/docs/observability/operational-telemetry/)** and **[Data retention](/docs/security/data-retention/)**.

**What is workflow lineage vs security audit?**

**Workflow lineage** and port audit items on the **workflow file store** answer **how a value was calculated**. The **immutable security audit trail** answers **who changed platform settings, signed in, or accessed the cluster**. Do not substitute lineage for WORM evidence. See **[Data lineage](/docs/data/data-lineage/)** and **[Compliance evidence](/docs/compliance/)**.

**How do you monitor availability?**

In-cluster **Gatus** probes platform and tenant services; the status page sits behind HTTP basic auth. Operational telemetry flows to HyperDX via OTLP, with bundled alert rules on Kubernetes audit events that record WORM evidence per alert. See **[Operational telemetry](/docs/observability/operational-telemetry/)**.

#### Backup, recovery, and availability

**Do you take backups before upgrades?**

Yes. An **automatic full-stack snapshot** runs before each workspace upgrade (data volume plus pinned release metadata). Operators can also create manual snapshots. See **[Backup and restore](/docs/deployment/backup-and-restore/)**.

**Can we roll back a failed upgrade?**

Yes. **`fontana rollback`** restores the data volume and redeploys the pinned release from a chosen snapshot. See **[Fontana CLI](/docs/deployment/fontana-cli/)**.

**Can we export workspace configuration without secrets?**

**Admin → Deployment → Backup/Restore** exports roles, teams, workflows, agents, and non-secret configuration. **Vault secrets and workflow run datasets** are excluded by design. See **[Backup and restore](/docs/deployment/backup-and-restore/)**.

**How is Vault recovered?**

Vault uses **Raft storage** on the workspace cluster as part of the snapshot boundary. Treat platform snapshots and your DR policy as the recovery path for secrets infrastructure. See **[Backup and restore](/docs/deployment/backup-and-restore/)**.

#### Change management and supply chain

**How are releases built and deployed?**

CI builds **immutable OCI images** and a verified deploy bundle. Production hosts pull **GHCR only** with **`sha-<git>`** tags; bundle **SHA256SUMS** verifies integrity. Upgrades are **manual pinned releases** via the Fontana CLI. See **[Supply chain](/docs/security/supply-chain/)** and **[Fontana CLI](/docs/deployment/fontana-cli/)**.

**Do you scan container images for vulnerabilities?**

Yes. **Trivy** runs in CI with a **CRITICAL** gate on platform images. See **[Supply chain](/docs/security/supply-chain/)**.

**Can production hosts build from source?**

**No** on production-class hosts. Immutable artifacts only; no monorepo checkout required to run tenants. Integration hosts may differ and are out of SOC 2 scope. See **[Deployment environments](/docs/security/deployment-environments/)**.

**How is infrastructure provisioned?**

Host provisioning uses **Terraform** (`infra-k8s/`). Platform state is declared in **`fontana.yaml`** and reconciled by **`fontana apply`**. See **[Deployment architecture](/docs/deployment/architecture/)**.

#### AI governance and data handling

**Can we bring our own LLM keys and gateways?**

Yes. **BYOK** and **approved gateways/models** in Admin enforce which providers and models your workspace may call. See **[BYOK](/docs/ai/byok/)** and **[Gateways](/docs/ai/gateways/)**.

**Does Fontana train on our data?**

Fontana routes inference to **your configured providers** under BYOK. **Zero Data Retention (ZDR)** applies where the upstream gateway supports it. Provider-specific behaviour is documented in **[Zero Data Retention](/docs/ai/zdr/)**.

**Are AI tool calls audited?**

Tool invocations write structured records with protocol and latency metadata. Correlation with the **WORM ledger** depends on enabled audit producers. Human approval gates (askHuman, canvas suggestions, NL compile HITL) are documented in **[Human in the loop](/docs/ai/human-in-the-loop/)**.

**How is sandboxed agent code execution isolated?**

Shared **OpenSandbox** runs in the platform cluster with **per-tenant bearer auth** and workload isolation via **fontana-sandbox-proxy** (fail-closed). See **[Tools and MCP](/docs/ai/tools-and-mcp/)** and **[Workspace isolation](/docs/security/workspace-isolation/)**.

#### Privacy, retention, and subprocessors

**What are your data retention boundaries?**

Three stores with **separate retention**: workflow file store (run data and lineage sidecars), HyperDX operational logs (search plane, shorter TTL), and **ImmuDB WORM** (compliance audit, append-only). See **[Data retention](/docs/security/data-retention/)**.

**Can we control egress and exports?**

Workflow **egress** nodes and export paths are part of your configured workflows. Platform configuration and legal terms for data processing are available under NDA. See **[Data egress](/docs/data/egress/)** and **[Security and governance](/technology/security)**.

**Who are your subprocessors?**

A current subprocessor list, DPA, and vendor register are provided in the **diligence pack under NDA**. Self-hosted deployments use **your chosen LLM and connector providers** under BYOK.

#### Incident response and evidence requests

**What is your incident response process?**

Escalation paths, customer communication, and IR runbooks are diligence topics covered in the **NDA pack** and **[Security and governance](/technology/security)**. Operational detection uses Gatus, HyperDX alerts, and WORM audit review.

**Can you provide penetration test results?**

Penetration test summaries and formal test reports are available **under NDA** on request via **[Security and governance](/technology/security)**.

**Can you export control snapshots for our auditor?**

Operators can run evidence collection on production-class hosts before audits. Architecture diagrams, RBAC exports, and extended snapshot fields are described in internal runbooks; contact Fontana for **exported control snapshots** and the full diligence pack.

**Where do I request the full diligence pack?**

Use **[Security and governance](/technology/security)** or contact Fontana for architecture diagrams, subprocessors, DPA terms, certification programme status, and evidence exports under NDA.

#### Related documentation

- **[SOC 2 control summary](/docs/security/soc2-controls/)** - control matrix and Trust Service Criteria map
- **[Security overview](/docs/security/)** - topics index and three evidence planes
- **[Compliance evidence](/docs/compliance/)** - immutable audit trail and workflow lineage
- **[Deployment](/docs/deployment/)** - cloud, self-hosted, backup, and CLI


# Data retention
## Description: Retention boundaries for workflow data, operational logs, WORM audit, and customer-controlled egress.
## url: https://fontana-ai.com/docs/security/data-retention/

Retention questions in SOC 2 and privacy reviews usually span **three stores** with different purposes: **workflow run data** on your workspace file store, **operational telemetry** in HyperDX, and the **immutable security audit ledger** (ImmuDB). Fontana separates these so you can apply the right retention policy to each plane.

#### Retention by store

| Store                     | What it holds                                                         | Typical retention posture                                                                                      | Compliance role                                                |
| ------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| **Workflow file store**   | Uploads, Arrow run datasets, lineage and port audit sidecars, exports | Persists on encrypted workspace PVC until you delete runs or reclaim space through your operational procedures | Calculation provenance; **not** the WORM platform ledger       |
| **HyperDX (operational)** | Pod logs, browser errors, dashboards, mirrored audit search           | **Shorter TTL** configurable per environment; optimised for search and incident response                       | **Search plane only**; not the authoritative compliance record |
| **ImmuDB (WORM audit)**   | Append-only security audit events per workspace                       | **Longer retention targets** on production-class hosts; sized for diligence and investigation                  | **System of record** for platform security audit               |

<Aside type="note" title="HyperDX is not the compliance ledger">

HyperDX may **mirror** security audit events so operators can search quickly during incidents. If HyperDX retention expires or an index is rebuilt, **ImmuDB remains the tamper-evident system of record**. See **[Security audit (WORM)](/docs/observability/security-audit-worm/)**.

</Aside>

#### Workflow data and lineage

Workflow run data follows the **[Core Fontana data principles](/docs/data/storage-and-processing/)**: durable on encrypted workspace storage, immutable processor outputs, explicit edit overlays. Retention of historical runs is an **operational choice** within your workspace boundary:

- Run directories, lineage sidecars, and port audit items live alongside datasets on the workflow engine file store
- Deleting or archiving runs removes that **workflow provenance** evidence; it does not delete WORM platform audit entries that already recorded privileged activity

Lineage and port audit items answer **how a value was computed**. They are stored on the **workflow file store**, not in ImmuDB. See **[Data lineage](/docs/data/data-lineage/)** and **[Compliance evidence](/docs/compliance/)**.

#### Operational logs (HyperDX)

Your deployment team configures HyperDX during install and upgrade: OTLP credentials, bundled dashboards, and **retention TTL** appropriate to the environment. Production-class hosts use longer retention than integration or local boxes where applicable.

Operational logs support **availability and incident response** (TSC Availability). They are not a substitute for WORM audit evidence in a compliance review.

#### WORM security audit (ImmuDB)

The immutable ledger retains **platform security events** (identity, cluster, and application admin paths documented in **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)**). Integrity scans and read-only auditor access support evidence that records were not altered after append.

Contact Fontana under NDA for environment-specific retention targets and diligence pack detail.

#### Egress and external processors

When you export workflow data or call external LLM and connector APIs, **retention and subprocessors follow your configuration and provider contracts**:

- **[Egress](/docs/data/egress/)** - where workflow outputs leave the workspace
- **[BYOK](/docs/ai/byok/)** and **[Zero Data Retention](/docs/ai/zdr/)** - model routing under your keys and gateway policies
- DPA and subprocessor lists in the **[Security and governance](/technology/security)** diligence pack

Fontana does not operate a shared production LLM pool for your workspace; inference uses credentials and gateways you approve.

#### Related documentation

- **[Compliance evidence](/docs/compliance/)** - lineage vs immutable audit trail
- **[Operational telemetry](/docs/observability/operational-telemetry/)** - HyperDX, Gatus, and the otel collector
- **[Deployment environments](/docs/security/deployment-environments/)** - which hosts are in SOC 2 scope
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - Privacy and Availability TSC rows


# Deployment environments
## Description: Production-class hosts in SOC 2 scope vs integration and local environments for engineering only.
## url: https://fontana-ai.com/docs/security/deployment-environments/

Fontana runs on **more than one class of host**. SOC 2-aligned control narratives and evidence collection apply to **production-class** deployments where regulated customer data lives. Integration and local environments support **engineering velocity** on hosts outside that scope.

#### Environment classes

| Class                | Typical use                                                              | Customer / regulated data                | Change control                                                                              |
| -------------------- | ------------------------------------------------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------- |
| **Production-class** | Managed cloud production, customer VPC, dedicated self-hosted production | **In scope**                             | Pinned immutable releases, automatic pre-upgrade snapshots, GHCR-only artifacts on the host |
| **Integration**      | Shared engineering box, pre-release validation                           | **Out of scope** for regulated workloads | Pre-release validation; may build from source on the host                                   |
| **Local**            | Developer laptop, `*.localhost` installs                                 | **Out of scope**                         | Local integration testing and hybrid dev workflows                                          |

<Aside type="caution" title="Environment separation">

Point production DNS and regulated customer tenants at **production-class** hosts. Use integration and local environments for feature testing and development workflows.

</Aside>

#### What production-class means

Production-class hosts share these properties:

- **Immutable deploy artifacts only** on the box: pre-built OCI images and a verified deploy bundle from your registry (typically GHCR). No monorepo checkout or build toolchain required to run tenants.
- **Pinned upgrades** via the Fontana CLI using immutable `sha-<git>` release tags.
- **Automatic full-stack snapshot** before each tenant upgrade, with rollback support.
- **Cluster-per-tenant** workspace isolation, per-workspace Vault, and WORM audit as documented in **[Security](/docs/security/)**.

Managed cloud production and customer VPC deployments Fontana operates follow this model. Self-hosted customers can adopt the same pinned-release pattern on their own production hosts.

#### Integration and local (out of scope)

Integration hosts may keep a source checkout for on-box builds and faster iteration. Local development may combine containerised platform services with live application dev servers.

Those environments help your team ship features. They do **not** replace production controls for:

- Supply chain integrity (pinned digest tags, bundle checksum verification)
- Formal change records (snapshot history, pinned release trail)
- Regulated data residency and audit evidence export

#### Related documentation

- **[Supply chain](/docs/security/supply-chain/)** - production artifact and upgrade controls
- **[SOC 2 control summary](/docs/security/soc2-controls/)** - questionnaire index and environment scope
- **[Vendor diligence FAQ](/docs/security/vendor-diligence-faq/)** - recurring procurement Q&A
- **[Deployment](/docs/deployment/)** - cloud, self-hosted, and dedicated host models
- **[Fontana CLI](/docs/deployment/fontana-cli/)** - apply, upgrade, and rollback on the host


# Compliance evidence
## Description: Workflow data lineage, immutable security audit trail, and how compliance evidence differs from platform security controls.
## url: https://fontana-ai.com/docs/compliance/

Fontana gives you **two complementary evidence types** for compliance review: **workflow data lineage** (how a value was computed) and the **immutable security audit trail** (who did what on the platform). Platform controls such as isolation, encryption, secrets, and supply chain are documented under **[Security](/docs/security/)**.

<Aside type="note" title="Which section to read?">

| Your question                                         | Start here                                                           |
| ----------------------------------------------------- | -------------------------------------------------------------------- |
| Who signed in, changed RBAC, or accessed the cluster? | **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** |
| How was this cell derived from upstream data?         | **[Data lineage](/docs/data/data-lineage/)** (below)                 |
| Is my workspace isolated and encrypted?               | **[Security](/docs/security/)**                                      |

</Aside>

#### Topics

- **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** - append-only WORM ledger (ImmuDB), event producers, HyperDX mirror vs system of record, integrity scans
- **[Data lineage](/docs/data/data-lineage/)** - atomic provenance from ingress through export, OpenLineage export, canvas inspection

#### Workflow data lineage

Fontana records **atomic data lineage** across workflow execution: every transformation, lookup, merge, filter, compute step, and AI operation that derives a cell can attach pointer-based provenance back to upstream rows and columns. Lineage is stored as structured metadata alongside port datasets, not as copied row snapshots, so auditors can traverse a recursive source chain from any output cell to its origins.

- **Per-operation lineage** - workflow processors emit lineage entries with usage types such as `transformation`, `lookup`, `merge`, `reference`, `computed`, `generated`, and `filtered`
- **Per-port artifacts** - server runs persist lineage and audit sidecars under each node port in the run directory; validation failures surface as audit items on retained rows
- **UI inspection** - the Data Lineage panel on the canvas walks the provenance tree for the selected cell
- **Durable evidence** - run-scoped lineage and port audit files persist on encrypted workspace storage across pod restarts and upgrades within your tenant

Lineage answers **calculation provenance**. It is not the same as the **immutable security audit trail**, which answers **privileged platform activity**. See **[Immutable audit trail](/docs/compliance/immutable-audit-trail/)** for the comparison table.

#### Workflow run audit (canvas and file store)

During execution, the workflow engine records **data-quality audit items** on port outputs: validation results, transform summaries, and manual edit overlays. Lineage sidecars (`.lineage.json`) and port audit artifacts (`.audit.json`) persist on the **workflow file store** alongside run datasets.

<Aside type="caution" title="Not the WORM ledger">

Port audit items and lineage answer **calculation provenance** (_how_ a value was derived). They are **not** stored in ImmuDB and are **not** a substitute for the **immutable security audit trail** (_who_ changed RBAC, signed in, or accessed the cluster). See **[Data lineage](/docs/data/data-lineage/)** and **[Data retention](/docs/security/data-retention/)**.

</Aside>

- **AI tool calls** - invocations write structured records with protocol and latency metadata; correlation with WORM audit depends on enabled producers
- **Admin changes in Flow** - privileged configuration changes should appear in admin audit logs and, when producers are enabled, the immutable audit trail via Convex

#### Platform security and SOC 2 summary

Isolation, Vault secrets, encryption in transit and at rest, network hardening, supply chain integrity, and the questionnaire-oriented control matrix live under **[Security](/docs/security/)**, including **[SOC 2 control summary](/docs/security/soc2-controls/)** and **[Vendor diligence FAQ](/docs/security/vendor-diligence-faq/)**. Certification posture and diligence packs: **[Security and governance](/technology/security)**.

#### Related documentation

- **[Security](/docs/security/)** - platform security controls and SOC 2 control summary
- **[Data storage and processing](/docs/data/storage-and-processing/)** - Core Fontana data principles (immutable, durable, auditable)
- **[Data retention](/docs/security/data-retention/)** - workflow file store vs HyperDX vs ImmuDB
- **[Identity and access](/docs/auth/)** - Zitadel SSO, MFA, SCIM, RBAC
- **[Backup and restore](/docs/deployment/backup-and-restore/)** - snapshots, rollback, configuration export


# Immutable audit trail
## Description: How Fontana captures identity, application, and platform security events in a tamper-evident append-only ledger.
## url: https://fontana-ai.com/docs/compliance/immutable-audit-trail/

Fontana maintains a **security audit trail** for every workspace: a single, normalized stream of who did what, when, and with what outcome. Events are written to an **append-only, tamper-evident ledger** (WORM: write once, read many) so later edits or deletes cannot silently rewrite history. That ledger is the **system of record** for security and compliance review; **HyperDX** mirrors the same events for day-to-day operator search.

This trail is separate from **workflow data lineage** (pointer-based provenance from output cells back through transforms). Lineage explains how a number was derived; the immutable audit trail explains **privileged and security-relevant activity** across the platform. See **[Compliance evidence](/docs/compliance/)** for lineage and **[Security](/docs/security/)** for platform controls.

<Aside type="note" title="What is WORM?">

**Write once, read many (WORM)** storage means new audit records can be appended, but existing entries are not updated or removed through normal application paths. Cryptographic verification on each append makes tampering detectable. Fontana uses **ImmuDB** per workspace as the WORM store; **observability-api** is the **only** writer.

</Aside>

#### Single writer: observability-api

All security audit paths converge on **observability-api** inside the workspace cluster. Application code (Flow, workflow engine, Airbyte sidecar) **never** calls ImmuDB directly. Components either:

- emit events that observability-api ingests automatically (**Zitadel**, **Kubernetes API audit**), or
- record events in **Convex** with structured **`console`** logs (`auditable: true` for admin writes), which the webhook forwards to observability-api for ImmuDB append when eligible.

```text
Flow · Convex · workflow engine · Zitadel · cluster APIs
  → observability-api (normalize, redact, dedupe)
  → append-only ImmuDB ledger (tamper-evident)
  → mirror to HyperDX for search and dashboards
```

#### Where events come from

##### Flow App UI

| Signal                                 | Immutable ledger?                                                                                 |
| -------------------------------------- | ------------------------------------------------------------------------------------------------- |
| Browser errors and product telemetry   | **No** (HyperDX only)                                                                             |
| **Audit logs → Chat** (LLM debug)      | **No** (Convex product feature; RBAC-gated)                                                       |
| Admin and security-relevant UI actions | **Yes**, via **Convex** structured logs (`auditable: true`) on the log stream → observability-api |

Flow has no direct WORM writer. Privileged UI changes are recorded in Convex as structured JSON on the **`console`** log stream topic; observability-api maps them to HyperDX and ImmuDB when **`auditable: true`**.

##### Convex

| Signal                                        | Immutable ledger?                 |
| --------------------------------------------- | --------------------------------- |
| General log stream events                     | HyperDX mirror only               |
| Structured admin logs (`auditable: true`)     | **Yes** → ImmuDB + HyperDX        |
| Log stream topic **`audit_log`** (deployment) | **Yes** → ImmuDB                  |
| Tables such as workflow document edit history | **No** (mutable application data) |

Convex is the primary **application** path for security audit: RBAC changes, team membership, BYOK configuration, admin deployment settings, and similar mutations emit **`FontanaObservabilityLog` v1** JSON via `console.log` after successful writes. Flow passes a **`correlationId`** per admin action so related logs share one chain id in HyperDX and ImmuDB.

##### Workflow engine

| Signal                                         | Immutable ledger?                                 |
| ---------------------------------------------- | ------------------------------------------------- |
| Lifecycle and capacity logs                    | **No** (HyperDX via pod logs)                     |
| Port **audit items** on run outputs            | **No** (data-quality semantics on the file store) |
| Privileged runs and sensitive connector access | **Target:** engine → Convex **`audit_log`**       |

The engine does not POST to observability-api for WORM. Operational logs support SRE; security-relevant engine actions should route through Convex.

##### Zitadel (automated)

**Fully automated.** observability-api polls the Zitadel Events API on a fixed interval using a service credential from Vault. Identity events (sign-in, MFA, federation, role assignment, deprovision, and related types) normalize into the shared schema and append to ImmuDB. No Flow or Convex code is required for this path.

##### Airbyte (PyAirbyte sidecar)

| Signal                                               | Immutable ledger?                                    |
| ---------------------------------------------------- | ---------------------------------------------------- |
| Connector `/read`, `/check`, job protocol logs       | **No** (HyperDX via shared pod logs)                 |
| External data access classified as security-relevant | **Target:** workflow engine → Convex **`audit_log`** |

There is no dedicated Airbyte → ImmuDB pipeline. The sidecar runs beside the workflow engine on `localhost` only.

##### Kubernetes API (platform)

**Shipped.** The cluster audit log is tailed by an OpenTelemetry collector pipeline. Events export to HyperDX and to observability-api **audit-ingest**, which appends to ImmuDB. Examples: authentication failures, `secrets` access, destructive verbs in the workspace namespace.

Bundled HyperDX **alert rules** watch these events (authentication failures, secret access, destructive verbs, ClusterRoleBinding changes). Every alert transition posts a webhook back to observability-api, which appends an `alert.fired` or `alert.resolved` record to ImmuDB, so the ledger also evidences that detection operated.

#### Normalized schema

Security audit events share: workspace id, timestamp, actor, action, resource, outcome, correlation id, source, and dedupe id. Sensitive material is **redacted before storage**. If ImmuDB append fails, webhook and audit-ingest paths return errors (**fail closed**) rather than dropping events quietly.

| Category                              | Examples                                            | Source (v1)                                                                              |
| ------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Identity and access**               | Sign-in, MFA, role and provisioning changes         | Zitadel Events API (automated)                                                           |
| **Platform and cluster**              | Failed auth, secret access, privileged API calls    | Kubernetes API audit                                                                     |
| **Security alerting**                 | Alert fired / resolved on Kubernetes audit events   | HyperDX alert webhook → observability-api                                                |
| **Application and admin**             | RBAC, BYOK, admin configuration                     | Convex log stream structured admin logs (`auditable: true`) + deployment **`audit_log`** |
| **Workflow and AI (security-tagged)** | Privileged runs, sensitive tool or connector access | Target: Convex **`audit_log`** via engine or backend                                     |

Together, these sources give risk and compliance teams a **cross-cutting view**: identity, cluster, and application activity in one tamper-evident stream.

<Aside type="caution" title="Search vs system of record">

Use **HyperDX** for interactive search and alerting during operations. Use the **immutable ledger** (and integrity procedures below) when you need evidence that records were not altered after they were written. Retention differs: the mirror may use shorter TTL; the ledger is sized for longer retention targets on production workspaces.

</Aside>

#### Tamper evidence and integrity

Each append to the ledger is **cryptographically chained** with prior state. Fontana runs a daily **integrity scan** CronJob per workspace:

| Stage              | Component                                   | Output                                                             |
| ------------------ | ------------------------------------------- | ------------------------------------------------------------------ |
| Scan               | `immuclient scan`                           | Cryptographic consistency proof against the full ledger            |
| Archive            | Evidence volume (`worm-audit-evidence` PVC) | Timestamped scan log + JSON manifest (SHA-256, tenant, outcome)    |
| Off-box (optional) | S3 Object Lock                              | Same files when `wormAudit.s3.bucket` is configured                |
| Operations         | HyperDX OTLP                                | Success or failure log (`ServiceName:fontana-worm-integrity-scan`) |

Operators can:

- Search HyperDX (**Fontana — WORM audit** dashboard or `ServiceName:fontana-worm-integrity-scan`) for scan history and optional alert rules
- Read manifests on the evidence volume for audit packs (see operator runbook)
- Use **HyperDX** (`ServiceName:fontana-worm-audit`) for interactive search of mirrored security events
- Use **in-cluster REST** or **`immuclient`** for direct ledger verification when HyperDX is not enough

Read-only **auditor** access can be provisioned separately from the writer credential so reviewers do not share the same secret as the automated append pipeline.

#### Workspace isolation

Each workspace is a **separate Kubernetes cluster** with its own ImmuDB instance, Vault secrets, and PVC-backed storage. Audit data does not share a database or secret store with other workspaces on the same host. Network policy restricts which in-cluster services may write or reach the ledger.

#### Relationship to other compliance topics

| Topic                   | Immutable audit trail                                                            | Workflow lineage                                               |
| ----------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| **Question it answers** | Who did what on the platform (security and operations)?                          | How was this output value computed from inputs?                |
| **Storage**             | Append-only ImmuDB per workspace                                                 | Per-run `.lineage.json` / `.audit.json` on workflow file store |
| **Primary audience**    | Security, risk, compliance, platform ops                                         | Operations, auditors reviewing calculation provenance          |
| **UI today**            | Operator tools (HyperDX mirror, integrity scan Job); not embedded in Flow canvas | Data Lineage panel on the canvas                               |

For credential and operator access to the ledger, your Fontana deployment team uses workspace-scoped Vault secrets and the WORM audit runbook (`k8s/soc2/worm-audit-runbook.md`). For diligence packs and control narratives, contact Fontana under NDA.

#### Related documentation

- **[Observability overview](/docs/observability/)** - operational telemetry vs security audit planes
- **[Security audit (WORM)](/docs/observability/security-audit-worm/)** - ImmuDB ledger and event categories
- **[Security](/docs/security/)** - isolation, encryption, and SOC 2 control summary
- **[Compliance evidence](/docs/compliance/)** - workflow lineage overview
- **[Identity and access](/docs/auth/)** - Zitadel, SSO, MFA, and application RBAC