Skip to content
Back to

Operational telemetry

Operational telemetry is how your team runs Fontana day to day: search logs, trace errors, watch dashboards, and confirm services are up. Fontana centralises this at the platform layer so every workspace does not need its own observability stack.

Your operators use Grafana on the shared platform cluster (EKS in cloud environments, platform k3d on local *.localhost installs) with Loki as the log store. Gatus runs in-cluster (not inside the Flow app) and records service health history for platform and tenant workloads.

What gets collected

Section titled “What gets collected”
Source Examples Typical use
Platform workloads Flow, workflow engine, Convex backend Pod logs, errors, run lifecycle
Flow browser UI errors and product telemetry Front-end incidents and usage signals
Convex Structured execution and console events Backend behaviour and log stream topics
Workflow engine Run lifecycle, capacity samples Pipeline performance and failures
Connectors Connector job output in the engine pod Ingress and egress troubleshooting
Node metrics CPU, memory, load on the host Capacity and saturation

Signals from each workspace pass through observability-api in that workspace cluster before export to the platform OTLP gateway, so tenancy boundaries stay intact.

Platform Grafana and Loki

Section titled “Platform Grafana and Loki”

Each install stamps fontana.environment (install label, for example aws-dev or zoltan-laptop) and fontana_tenant_id (workspace key, for example demo) on every OTLP batch. Grafana dashboards and Explore filter on both dimensions so shared platform backends stay tenant-scoped.

Property Behavior
Remote path platform.observability.url (OTLP) + platform.observability.uiUrl (Grafana) in fontana.yaml
OTLP auth Shared install Bearer (FONTANA_PLATFORM_BEARER), same pattern as Docling and Infinity remotes
Local path platform.observability.disabled: false on platform k3d; UI at https://status.<baseDomain>/observe
WORM audit Unchanged — observability-api and ImmuDB stay per workspace

Example LogQL in Grafana Explore:

{fontana_environment="zoltan-laptop", fontana_tenant_id="demo"}

Your deployment team provisions the Fontana tenant logs home dashboard, Loki datasource, and bundled alert packs on platform deploy.

Grafana operator access

Section titled “Grafana operator access”
  • Cloud dev / staging: https://observe.platform.<domain>/ (Grafana admin login; no platform bearer on the UI)
  • Local platform k3d: https://status.<baseDomain>/observe
  • Credentials: Grafana admin password from your platform Terraform output or operator env file (GRAFANA_ADMIN_PASSWORD in .env.dev for AWS dev EKS)

Use the Environment and Tenant variables at the top of the home dashboard, or Explore with the LogQL pattern above.

Gatus service health

Section titled “Gatus service health”

Gatus runs in each tenant cluster and records a health history page separate from Grafana log search. Tenant Gatus probes in-cluster apps (workflow engine, Convex, Chat Channels, Collab, Vault, Zitadel), backing databases over TCP (Convex Postgres, Zitadel Postgres, FalkorDB, ImmuDB), and shared platform-cluster services (Traefik, Docling, TEI, Kokoro TTS) via the host bridge. A separate platform-cluster Gatus probes Docling, TEI, Kokoro, Traefik, and OpenSandbox MCP.

Tenant Gatus and kubelet probe the workflow engine at in-cluster GET /healthz. That handler stays on the HTTP thread as an unauthenticated ok. Arrow work for Flow grid pages and sample-data runs on a dedicated worker thread so a large port decode does not starve the probe.

Gatus records availability and uptime for those endpoints. Your administrator can share the in-cluster URL and port for your box.

Authentication: the Gatus status page enforces HTTP basic auth. Credentials are generated at deploy time, stored in your workspace Vault, and surfaced by fontana resources <tenant> under operator console logins.

Operational logs vs WORM system of record

Section titled “Operational logs vs WORM system of record”

Grafana and Loki are the operational search plane: fast log query, dashboards, and alerting during incidents. Security audit events may be mirrored into the same Loki store for operator search.

ImmuDB is the compliance system of record for platform security audit. Loki retention, compaction, or index rotation does not change the append-only WORM ledger. For diligence, cite ImmuDB integrity procedures and Immutable audit trail.

See Data retention for retention boundaries across workflow data, operational logs, and ImmuDB.

Node log and metrics collector (otel-collector)

Section titled “Node log and metrics collector (otel-collector)”

Each workspace cluster runs an OpenTelemetry collector as a DaemonSet on every node. It ships container logs and node CPU, memory, and load metrics to the platform OTLP gateway (or local platform bridge /otlp).

Property Posture
Scope Per workspace cluster only; stamped with fontana.tenant_id and fontana.environment
Host access Runs as root with read-only mounts of host log directories and host filesystem paths
Kubernetes API Pod metadata is derived from log file paths on the node
Egress Allowed only to the configured OTLP endpoint (plus DNS), enforced by NetworkPolicy
Redaction Bearer tokens, JWTs, API keys, and Vault tokens replaced with [REDACTED] before export

Workflow canvas signals (operational context)

Section titled “Workflow canvas signals (operational context)”

While a workflow runs, the Flow canvas shows per-node status, row counts, and port audit items (validation warnings, transform notes, user edits). These help operators and builders debug a run in context.

That canvas audit is workflow data quality and provenance on the workflow file store. It is not operational telemetry in Loki and not the WORM security ledger.

Section titled “Related documentation”