Skip to content
Back to

Security audit (WORM)

Security audit (WORM) is Fontana’s compliance-grade event history: who did what, when, and with what outcome, stored so later edits cannot silently rewrite the past. WORM means write once, read many: new records append; existing entries are not updated or removed through normal application paths.

Each workspace has its own ImmuDB ledger. observability-api in that workspace is the only writer to the ledger. Application components do not call ImmuDB directly; they emit events that observability-api normalises, redacts, and appends.

Platform Loki may mirror the same events for operator search. For diligence and evidence, treat ImmuDB as the system of record, not the operational log store. Loki is a search plane with its own retention; expiry or compaction there does not alter the append-only WORM ledger. See Data retention and Operational telemetry.

What gets recorded

Section titled “What gets recorded”

Security audit events share a normalised schema: workspace, timestamp, actor, action, resource, outcome, and correlation id. Sensitive material is redacted before storage (no API keys, bearer tokens, or full JWT bodies). When resource holds a JSON-encoded event payload, observability-api deep-redacts nested sensitive keys (for example token, password, and *_secret) before append and Loki mirror.

Category Examples Typical source
Identity and access Sign-in, MFA, role and provisioning changes Zitadel (automated poll)
Platform and cluster Failed auth, secret access, privileged API calls Kubernetes API audit (bundled Grafana alert rules fire on these events and append a WORM evidence record per alert)
Application and admin RBAC, BYOK, admin configuration Convex structured admin logs (auditable: true) on the log stream
Workflow and AI (security-tagged) Privileged runs, sensitive tool or connector access Application audit producers (via Convex)

Identity events from Zitadel (login, MFA, federation, deprovision) reach the ledger automatically without custom Flow code.

Fail closed

Section titled “Fail closed”

If append to the WORM ledger fails, ingest paths return errors rather than dropping security events quietly. Your deployment team monitors observability-api and ImmuDB health as part of platform operations.

Review and integrity

Section titled “Review and integrity”

Each workspace runs a daily integrity scan CronJob (immu-audit-scan, schedule 0 3 * * * UTC):

  1. Scan (init container) — immuclient login + immuclient scan against the ImmuDB ledger.
  2. Evidence (observability-api container) — writes a JSON manifest and raw scan log to the workspace evidence volume (worm-audit-evidence PVC at /evidence on Kubernetes), optionally uploads both files to S3 Object Lock when your platform team configures wormAudit.s3, and posts a structured OTLP log to platform Loki.
Artifact Location Purpose
immuclient-scan-<scanId>.log Evidence volume Raw scan output for auditors
manifest-<scanId>.json Evidence volume Tenant, outcome, SHA-256 of log, S3 keys, OTLP reported flag
OTLP log Loki (ServiceName:fontana-worm-integrity-scan) Operator search and optional alert rules

Grafana alerting: Fontana automatically emits a success or failure log on every scan when platform OTLP is configured. The bundled Fontana — WORM audit dashboard shows recent scan outcomes. Email, Slack, or PagerDuty notifications are a one-time Grafana alert rule you attach to the dashboard tile or saved query. See Operational telemetry.

If the scan fails, or OTLP delivery fails when configured, the CronJob exits non-zero so Kubernetes retains a failed Job for investigation (fail-closed).

Operators and auditors also use:

  • Grafana for interactive search during incidents (security audit mirror at ServiceName:fontana-worm-audit)
  • In-cluster REST (http://immu-audit:8080) for smoke tests and break-glass verification (see operator runbook)

Read-only auditor access can be provisioned separately from the writer credential used by automated append.

Verify integrity scan (operators)

Section titled “Verify integrity scan (operators)”

After deploy, trigger a one-off Job from the CronJob and confirm evidence plus OTLP delivery:

Terminal window
export KUBECONFIG="$FONTANA_HOME/kubeconfig/tenant-<id>.yaml"   # e.g. tenant-demo
NS=fontana
JOB="immu-audit-scan-manual-$(date +%s)"
kubectl create job -n "$NS" "$JOB" --from=cronjob/immu-audit-scan
kubectl wait -n "$NS" --for=condition=complete "job/$JOB" --timeout=300s
kubectl logs -n "$NS" "job/$JOB" -c immuclient-scan
kubectl logs -n "$NS" "job/$JOB" -c evidence

Expect the evidence container log line outcome=success. In Grafana, search ServiceName:fontana-worm-integrity-scan or open the Fontana — WORM audit dashboard.

See Immutable audit trail for schema detail, producer map, and workspace isolation.

Section titled “Related documentation”