Immutable audit trail

Fontana maintains a security audit trail for every workspace: a single, normalized stream of who did what, when, and with what outcome. Events are written to an append-only, tamper-evident ledger (WORM: write once, read many) so later edits or deletes cannot silently rewrite history. That ledger is the system of record for security and compliance review; HyperDX mirrors the same events for day-to-day operator search.

This trail is separate from workflow data lineage (pointer-based provenance from output cells back through transforms). Lineage explains how a number was derived; the immutable audit trail explains privileged and security-relevant activity across the platform. See Compliance evidence for lineage and Security for platform controls.

What is WORM?

Write once, read many (WORM) storage means new audit records can be appended, but existing entries are not updated or removed through normal application paths. Cryptographic verification on each append makes tampering detectable. Fontana uses ImmuDB per workspace as the WORM store; observability-api is the only writer.

Single writer: observability-api

All security audit paths converge on observability-api inside the workspace cluster. Application code (Flow, workflow engine, Airbyte sidecar) never calls ImmuDB directly. Components either:

• emit events that observability-api ingests automatically (Zitadel, Kubernetes API audit), or
• record events in Convex with structured console logs (auditable: true for admin writes), which the webhook forwards to observability-api for ImmuDB append when eligible.

Flow · Convex · workflow engine · Zitadel · cluster APIs
  → observability-api (normalize, redact, dedupe)
  → append-only ImmuDB ledger (tamper-evident)
  → mirror to HyperDX for search and dashboards

Where events come from

Flow App UI

Signal	Immutable ledger?
Browser errors and product telemetry	No (HyperDX only)
Audit logs → Chat (LLM debug)	No (Convex product feature; RBAC-gated)
Admin and security-relevant UI actions	Yes, via Convex structured logs (auditable: true) on the log stream → observability-api

Flow has no direct WORM writer. Privileged UI changes are recorded in Convex as structured JSON on the console log stream topic; observability-api maps them to HyperDX and ImmuDB when auditable: true.

Convex

Signal	Immutable ledger?
General log stream events	HyperDX mirror only
Structured admin logs (auditable: true)	Yes → ImmuDB + HyperDX
Log stream topic audit_log (deployment)	Yes → ImmuDB
Tables such as workflow document edit history	No (mutable application data)

Convex is the primary application path for security audit: RBAC changes, team membership, BYOK configuration, admin deployment settings, and similar mutations emit FontanaObservabilityLog v1 JSON via console.log after successful writes. Flow passes a correlationId per admin action so related logs share one chain id in HyperDX and ImmuDB.

Workflow engine

Signal	Immutable ledger?
Lifecycle and capacity logs	No (HyperDX via pod logs)
Port audit items on run outputs	No (data-quality semantics on the file store)
Privileged runs and sensitive connector access	Target: engine → Convex audit_log

The engine does not POST to observability-api for WORM. Operational logs support SRE; security-relevant engine actions should route through Convex.

Zitadel (automated)

Fully automated. observability-api polls the Zitadel Events API on a fixed interval using a service credential from Vault. Identity events (sign-in, MFA, federation, role assignment, deprovision, and related types) normalize into the shared schema and append to ImmuDB. No Flow or Convex code is required for this path.

Airbyte (PyAirbyte sidecar)

Signal	Immutable ledger?
Connector /read, /check, job protocol logs	No (HyperDX via shared pod logs)
External data access classified as security-relevant	Target: workflow engine → Convex audit_log

There is no dedicated Airbyte → ImmuDB pipeline. The sidecar runs beside the workflow engine on localhost only.

Kubernetes API (platform)

Shipped. The cluster audit log is tailed by an OpenTelemetry collector pipeline. Events export to HyperDX and to observability-api audit-ingest, which appends to ImmuDB. Examples: authentication failures, secrets access, destructive verbs in the workspace namespace.

Bundled HyperDX alert rules watch these events (authentication failures, secret access, destructive verbs, ClusterRoleBinding changes). Every alert transition posts a webhook back to observability-api, which appends an alert.fired or alert.resolved record to ImmuDB, so the ledger also evidences that detection operated.

Normalized schema

Security audit events share: workspace id, timestamp, actor, action, resource, outcome, correlation id, source, and dedupe id. Sensitive material is redacted before storage. If ImmuDB append fails, webhook and audit-ingest paths return errors (fail closed) rather than dropping events quietly.

Category	Examples	Source (v1)
Identity and access	Sign-in, MFA, role and provisioning changes	Zitadel Events API (automated)
Platform and cluster	Failed auth, secret access, privileged API calls	Kubernetes API audit
Security alerting	Alert fired / resolved on Kubernetes audit events	HyperDX alert webhook → observability-api
Application and admin	RBAC, BYOK, admin configuration	Convex log stream structured admin logs (auditable: true) + deployment audit_log
Workflow and AI (security-tagged)	Privileged runs, sensitive tool or connector access	Target: Convex audit_log via engine or backend

Together, these sources give risk and compliance teams a cross-cutting view: identity, cluster, and application activity in one tamper-evident stream.

Search vs system of record

Use HyperDX for interactive search and alerting during operations. Use the immutable ledger (and integrity procedures below) when you need evidence that records were not altered after they were written. Retention differs: the mirror may use shorter TTL; the ledger is sized for longer retention targets on production workspaces.

Tamper evidence and integrity

Each append to the ledger is cryptographically chained with prior state. Fontana runs a daily integrity scan CronJob per workspace:

Stage	Component	Output
Scan	immuclient scan	Cryptographic consistency proof against the full ledger
Archive	Evidence volume (worm-audit-evidence PVC)	Timestamped scan log + JSON manifest (SHA-256, tenant, outcome)
Off-box (optional)	S3 Object Lock	Same files when wormAudit.s3.bucket is configured
Operations	HyperDX OTLP	Success or failure log (ServiceName:fontana-worm-integrity-scan)

Operators can:

• Search HyperDX (Fontana — WORM audit dashboard or ServiceName:fontana-worm-integrity-scan) for scan history and optional alert rules
• Read manifests on the evidence volume for audit packs (see operator runbook)
• Use HyperDX (ServiceName:fontana-worm-audit) for interactive search of mirrored security events
• Use in-cluster REST or immuclient for direct ledger verification when HyperDX is not enough

Read-only auditor access can be provisioned separately from the writer credential so reviewers do not share the same secret as the automated append pipeline.

Workspace isolation

Each workspace is a separate Kubernetes cluster with its own ImmuDB instance, Vault secrets, and PVC-backed storage. Audit data does not share a database or secret store with other workspaces on the same host. Network policy restricts which in-cluster services may write or reach the ledger.

Relationship to other compliance topics

Topic	Immutable audit trail	Workflow lineage
Question it answers	Who did what on the platform (security and operations)?	How was this output value computed from inputs?
Storage	Append-only ImmuDB per workspace	Per-run .lineage.json / .audit.json on workflow file store
Primary audience	Security, risk, compliance, platform ops	Operations, auditors reviewing calculation provenance
UI today	Operator tools (HyperDX mirror, integrity scan Job); not embedded in Flow canvas	Data Lineage panel on the canvas

For credential and operator access to the ledger, your Fontana deployment team uses workspace-scoped Vault secrets and the WORM audit runbook (k8s/soc2/worm-audit-runbook.md). For diligence packs and control narratives, contact Fontana under NDA.

Related documentation

• Observability overview - operational telemetry vs security audit planes
• Security audit (WORM) - ImmuDB ledger and event categories
• Security - isolation, encryption, and SOC 2 control summary
• Compliance evidence - workflow lineage overview
• Identity and access - Zitadel, SSO, MFA, and application RBAC