Observability on GENIE.AI

Stack Overview

Mon, 01 Jan 0001 00:00:00 +0000

The observability stack has four layers: instrumentation (in the apps), collection (the OpenTelemetry Collector), storage (the Victoria trio), and presentation (Grafana).

Prerequisites: enable the stack first — see Configuration for the environment variables and the ENABLE_OBSERVABILITY=1 toggle. Everything below assumes observability is already running.

Glossary. OTLP = OpenTelemetry wire protocol (HTTP/protobuf). BFF = Backend-for-Frontend, the Node.js API layer. Prometheus metrics = the standard scrape-and-store metric format Grafana and VictoriaMetrics both speak. fluentd = a Docker log-forwarding driver that ships container stdout/stderr to a TCP collector.

Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Grafana dashboards are auto-provisioned from configs/grafana/provisioning/dashboards/, so a deployment gets a working set the moment the stack starts — no manual import. The Grafana file provider has foldersFromFilesStructure: true, so dashboards placed in a subdirectory land in a folder named after that subdirectory.

The 9 dashboards are grouped into two folders:

General folder — 5 service dashboards at the root of dashboards/. (The file provider creates no application/ subdirectory, so these land in the default General folder.)
Observability folder — 4 stack-health dashboards in dashboards/observability/.

First time opening Grafana

You do this once after deploy.

Alerting

Mon, 01 Jan 0001 00:00:00 +0000

Alerting is auto-provisioned alongside the dashboards, so a fresh deployment already watches the failures that matter most: the stack going silent, storage filling up, the log pipeline breaking, and trace export failing.

How Grafana alerting is organised

Three layers, one job each:

Alert rules (alert-rules.yml) — what triggers an alert: a query, a threshold, and a for: window (the alert must hold true for that long before firing — short blips don’t wake anyone up).
Notification policies (notification-policies.yml) — how alerts are grouped, escalated, and routed between contact points. The shipped default groups by alertname + component and routes everything to a single contact point.
Contact points (contact-points.yml) — where notifications are sent (a webhook URL, Slack, SMTP, PagerDuty). The shipped default is a single webhook pointing at http://localhost:9999/alerts — replace per environment.

Built-in alert rules

The rules target the failure modes that would otherwise make the observability stack itself lie to you. All rules query the VictoriaMetrics datasource and live in the Observability folder.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

The observability stack has two layers with different defaults. VictoriaLogs and the OTel Collector are always-on (no environment opt-out) — the admin logs UI queries VictoriaLogs directly and every container ships logs via the fluentd driver, so VL is not gated. The in-app OTel SDK is opt-in via ENABLE_OBSERVABILITY=0|1 (default 0). Profile-gated services (Grafana, VictoriaMetrics, VictoriaTraces, tempo-proxy) come up with the observability profile in Compose, or ENABLE_OBSERVABILITY=1 in Swarm.

Prerequisites: an existing GENIE.AI deploy with an editable .env file. See the deployment guides for setup — Docker Swarm Setup or Docker Compose Setup. Also: ensure enough disk for the retention windows below (defaults are 30 days of metrics, logs, and traces).