Observability on GENIE.AI

Stack Overview

Mon, 01 Jan 0001 00:00:00 +0000

The observability stack has four layers: instrumentation (in the apps), collection (the OpenTelemetry Collector), storage (the Victoria trio), and presentation (Grafana).

Layer 1 — Instrumentation

Each application service initialises the OpenTelemetry SDK and emits telemetry through the standard OTLP protocol.

Service	Language	What it emits
Backend (BFF)	Node.js	Express request spans, ArangoDB query spans, custom business spans, Prometheus metrics.
OPEA services	Python	FastAPI spans, per-RAG-stage spans (retrieval, reranking, labelling…), service logs.
Kong gateway	Lua	OTel plugin forwards request traces.

Instrumentation helpers enforce a single way to create spans (so attributes and PII filtering stay consistent):

Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Tracing is what makes a GENIE.AI user query legible. Because every service propagates a single W3C traceparent header and every RAG stage emits a span, one user question produces one contiguous trace that can be read end-to-end in Grafana — revealing exactly where time went and which chunks were retrieved.

Trace propagation

The trace starts at the edge. Kong creates the root span and injects a traceparent header into the request. Each downstream hop reads that header, joins the trace as a child span, and re-injects the header for the next hop:

Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Grafana dashboards are auto-provisioned from configs/grafana/provisioning/, so a deployment gets a working set of dashboards the moment the stack starts — no manual import. They are grouped into two folders.

Application folder

Operational views of the GENIE.AI services themselves.

Dashboard	What it shows
Service health	Per-service up/down, replica count, error rate, request rate. The landing view for “is anything broken?”.
Application metrics	Business and HTTP metrics: request rates, latency distributions, error rates, custom counters.
Service logs	Centralised log search across all services (VictoriaLogs). Pivot from a metric spike to the matching logs.
Trace explorer	Distributed-trace search (VictoriaTraces via the Jaeger datasource): filter by service, operation, duration, attribute.
RAG pipeline trace waterfall	A waterfall charting per-service latency across the RAG pipeline (backend, chatqna, retriever, reranker, dataprep), so the slow stage is immediately visible.

Observability folder

Health of the observability stack itself — so the monitoring does not fail silently.

Alerting

Mon, 01 Jan 0001 00:00:00 +0000

Alerting is auto-provisioned alongside the dashboards, so a fresh deployment already watches the failures that matter most: the stack going silent, storage filling up, the log pipeline breaking, and trace export failing.

Three files under configs/grafana/provisioning/alerting/ define the setup:

File	Purpose
`alert-rules.yml`	The alert definitions (condition + for-duration).
`notification-policies.yml`	Which alerts route to which contact points, and grouping/escalation.
`contact-points.yml`	Where notifications are sent. The shipped default is a single webhook contact point; Slack/SMTP/PagerDuty are added per environment.

Built-in alert rules

The rules target the failure modes that would otherwise make the observability stack itself lie to you.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

The observability stack is disabled by default. Enabling it is a single environment variable, but a handful of related variables control access, retention, and sampling.

Enabling

Variable	Default	Notes
`ENABLE_OBSERVABILITY`	`0`	Must be `0` or `1` (not `true`/`false`). Swarm mode reads this to place the stack.
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://otel-collector:4318`	OTLP base URL all instrumented services export to. Override only for an external collector.

How it is enabled per deployment mode:

Docker Compose — docker compose --profile observability up -d
Docker Swarm — ENABLE_OBSERVABILITY=1 in .env
Ansible — enable_observability: "1" in group_vars/all.yml

Sampling

Variable	Default	Notes
`OTEL_TRACES_SAMPLER_RATE`	100.0	Trace sampling percentage. Lower for high-volume prod; raise when diagnosing. Metrics and logs are not sampled.
`KONG_TRACING_SAMPLING_RATE`	1.0	Kong trace sampling rate (0–1), aligned with `OTEL_TRACES_SAMPLER_RATE`.
`KONG_TRACING_INSTRUMENTATIONS`	`request`	Kong tracing instrumentations. Negligible overhead when the OTel plugin is disabled.

Retention

Each Victoria store keeps data for a configurable window; lower it to save disk, raise it for longer historical analysis.