Configuration

The observability stack has two layers with different defaults. VictoriaLogs and the OTel Collector are always-on (no environment opt-out) — the admin logs UI queries VictoriaLogs directly and every container ships logs via the fluentd driver, so VL is not gated. The in-app OTel SDK is opt-in via ENABLE_OBSERVABILITY=0|1 (default 0). Profile-gated services (Grafana, VictoriaMetrics, VictoriaTraces, tempo-proxy) come up with the observability profile in Compose, or ENABLE_OBSERVABILITY=1 in Swarm.

Prerequisites: an existing GENIE.AI deploy with an editable .env file. See the deployment guides for setup — Docker Swarm Setup or Docker Compose Setup. Also: ensure enough disk for the retention windows below (defaults are 30 days of metrics, logs, and traces).

Telemetry pipeline

flowchart LR
    subgraph Apps[App + AI services]
        BE[Backend<br/>Node.js OTel SDK]
        DR[Document Repository]
        OPEA[OPEA Python<br/>FastAPI OTel]
        KONG[Kong<br/>OTel plugin]
    end
    subgraph Ingest[Container stdout / fluentd driver]
        FL[fluentd]
    end
    OC[OTel Collector<br/>4318 OTLP, 24224 fluent_forward]
    VM[(VictoriaMetrics<br/>8428)]
    VL[(VictoriaLogs<br/>9428)]
    VT[(VictoriaTraces<br/>10428)]
    GF[Grafana<br/>Kong /grafana/]

    BE -->|OTLP traces + metrics| OC
    DR -->|OTLP traces + metrics| OC
    OPEA -->|OTLP traces + metrics| OC
    KONG -->|OTLP traces| OC
    Apps --> FL --> OC
    OC --> VM
    OC --> VL
    OC --> VT
    VM --> GF
    VL --> GF
    VT --> GF

Three signal paths converge on the OTel Collector: (1) app and OPEA SDKs push traces + metrics over OTLP, (2) Kong pushes traces via its OTel plugin, (3) every container’s stdout/stderr is tailed by the fluentd log driver and forwarded over fluent_forward (port 24224, localhost-only). The Collector fans out to the three Victoria* stores; Grafana queries all three behind /grafana/.

Enabling

How it is enabled per deployment mode

• Docker Compose — docker compose --profile observability up -d. The profile gates victoriametrics, victoriatraces, tempo-proxy, and grafana. The OTel Collector and VictoriaLogs are always on (the Collector is required for the fluentd logging driver to ship container logs, and VictoriaLogs is required by the admin log endpoints).
• Docker Swarm — ENABLE_OBSERVABILITY=1 in .env. The same four services scale from replicas: 0 to replicas: 1; the Collector and VL are always on.
• Ansible — enable_observability: "1" in group_vars/all.yml (the playbook provisions the Grafana dashboards and provisioning only when the flag is 1).

Sampling

Retention

Each Victoria store keeps data for a configurable window; lower it to save disk, raise it for longer historical analysis.

Grafana access

Grafana is served behind the Kong gateway at /grafana/ (no direct host port exposed) with Keycloak OIDC SSO.

SSO is the intended access path. The local admin is a break-glass account; day-to-day access goes through Keycloak, so access is governed by the same identity provider as the rest of the platform.

Verify SSO works

1. Browse to https://<your-domain>/grafana/.
2. You should be redirected to the Keycloak realm login — sign in with a realm user.
3. After login you land in Grafana with your Keycloak username.
4. If the local admin login form appears instead of the Keycloak redirect, KC_GRAFANA_CLIENT_SECRET is wrong (or KC_GRAFANA_CLIENT_ID does not match what keycloak-config-cli created).

Collector placement

The OTel Collector runs in Docker Swarm mode: global with no node placement constraint — one collector on every node (gateway, genieai, gpu). You do not need to do anything for collector placement — the stack already runs one collector per Swarm node, so container logs are captured regardless of where the scheduler lands a service.

The collector’s fluent_forward receiver (port 24224) is bound to 0.0.0.0:24224 inside the container, but the Swarm port mapping is mode: host, host_ip: 127.0.0.1, so only the local Docker daemon can push container logs into it. External log injection is blocked by design.

Verify on the Observability stack health dashboard — otelcol_* metrics are non-zero from every Swarm node.

Disabling

Set ENABLE_OBSERVABILITY=0 (Swarm) or omit the observability profile (Compose). What stops and what does not:

• Always-on regardless of the flag — otel-collector (the fluentd logging driver needs it) and victorialogs (admin endpoints depend on it).
• Gated by the flag — victoriametrics, victoriatraces, tempo-proxy, and grafana scale to replicas: 0 and are not started under the observability profile.

The Collector continues to receive OTLP, but every exporter is configured with sending_queue and retry_on_failure, so traces/logs/metrics are buffered on disk (file-backed for traces) and retried — telemetry is not lost unless the queue is full. Application services continue to run unaffected.

Practical consequences of disabling. Grafana dashboards stop refreshing (Grafana is off); the OTel Collector Down alert will fire (no VictoriaMetrics to scrape — silence the rule while observability is off, or the on-call channel gets woken up nightly). Backend log records continue to reach VictoriaLogs (always-on), so the Admin Logs UI keeps working.

Identity model (service.namespace / service.name / service.version)

Telemetry emitted by every instrumented service carries a hard-coded identity triple (no environment override, per the 2026-09-18 ops decision). Dashboards and alerting query against it.

The service.namespace is the natural filter for cross-service queries (everything in genieai participates in the RAG pipeline; genie-core participates in admin/UI/ingest). Cross-link: observe/dashboards.md where the namespace appears in the common-tag set for the pre-built dashboards.

Config file locations

Next steps

Once enabled, open Grafana → Dashboards to confirm the 9 dashboards are present (see Dashboards), then visit Alerting to wire a contact point and confirm the alert chain is live.