Admin Logs

The Logs tab in the admin dashboard (reachable at https://<NGINX_PUBLIC_DOMAIN>/admin → “Logs”) is the operator-facing surface for searching container logs across all GENIE.AI services. It is backed by VictoriaLogs via the backend’s MELT seam (MELT_PROVIDER=victorialogs, an abstraction layer in components/shared/lib/melt/ named for Metrics / Events / Logs / Traces — a future-proof seam that lets the storage backend swapped without changing call sites), with graceful degradation when VL is unreachable.

This page covers what the Logs tab does, the five endpoints behind it, the env vars that govern behaviour, and how to debug common VL failures.

Architecture

flowchart LR
  subgraph services[Services with logging: *fluent-logging]
    S1[backend / document-repository<br/>+ all OPEA services]
  end
  S1 -- stdout/stderr<br/>tag=genie.<container_name> --> FC[OTel Collector<br/>:24224 fluent_forward]
  FC -- OTLP HTTP<br/>logs pipeline --> VL[VictoriaLogs<br/>:9428]
  subgraph backend[Node.js backend]
    API[/api/admin/logs/*/]
    LogsService --> MELT[shared/lib/melt<br/>victorialogs-client.js]
    MELT -- /select/logsql/{query,hits} --> VL
  end
  API -- GET search/summary --> LogsService
  VL -. VlUnavailableError .-> API

The Logs tab never talks to VL directly — every read goes through the backend’s MELT seam (components/shared/lib/melt/victorialogs-client.js). Ingestion also goes through the OTel Collector (no direct jsonline POST from the app code). When VL is unreachable the read path throws a typed VlUnavailableError that surfaces to the API as HTTP 503 — see Failure modes.

Prerequisites

• Admin role in Keycloak (the JWT must contain realm_access.roles including admin; see Configuration → Keycloak admin guide).
• Observability stack on (or VL reachable on VICTORIALOGS_URL). When the observability profile is off, VL is still started as a core service (see the env file’s Section 12D: ADMIN LOGS MIGRATION) — but Grafana / dashboards are not provisioned.
• For non-admin operators: see the read-only Service Logs Grafana dashboard under General → Service Logs in Observability → Dashboards.

What the Logs tab is for

The Logs tab gives an operator a single search box for the structured Winston JSON envelopes emitted by every Node.js and Python service, plus the OTel Collector (services with logging: *fluent-logging in docker-compose.yaml). Three concrete workflows:

1. “Did my last re-ingest finish cleanly?” — pick dataprep-arango-service in the service dropdown (the Logs tab shows the actual VL service name — genie.<stack>_dataprep-arango-service_<n> for a Compose deployment), choose a date range, look for failed entries.
2. “Why did the login fail for user X?” — pick backend in the service dropdown (VL name: genie.<stack>_backend_<n>), search for the user’s Keycloak sub claim.
3. “Was VictoriaLogs itself healthy during yesterday’s outage?” — pick victorialogs in the service dropdown and use the “Yesterday” preset. Note: the OTel Collector container does not use the fluentd logging driver and does not export its own logs via OTLP, so its records do not land in VictoriaLogs.

Where service names come from: every container that runs with logging: *fluent-logging ships stdout/stderr to the OTel Collector under the tag genie.<container_name> (Compose container name format: <project>_<service>_<index>, e.g. genieai_backend_1). The Collector forwards to VictoriaLogs, which stores the tag as the service.name resource attribute. The Logs tab’s service dropdown is populated from those _stream.service values.

Workflow decision tree

Map your question to the right dropdown + filter combination:

flowchart TD
  Q{What do you need?} -->|Did ingest finish?| A[service=dataprep-arango-service<br/>dateRange=today or yesterday<br/>term=failed]
  Q -->|Why did login fail?| B[service=backend<br/>dateRange=today<br/>term=Keycloak sub claim]
  Q -->|Was VL healthy?| C[service=victorialogs<br/>dateRange=yesterday<br/>no term filter]
  Q -->|Generic error hunt?| D[level=ERROR<br/>dateRange=week<br/>leave service=all]
  Q -->|Specific term across all services| E[term=substring<br/>dateRange=week<br/>leave service=all]

Endpoints

All five endpoints live under /api/admin/logs and require a Keycloak JWT with the admin realm role.

Example: search for errors in the backend over the last week

ADMIN_TOKEN=...  # Get via the Keycloak master admin recipe

# `service=backend` matches the VL service.name `genie.<stack>_backend_<n>`
# (the Logs tab translates the dropdown label back to that exact value).
# Use `level=error` (lowercase is fine — the API uppercases via _normalizeLevelFilter).
curl -sk -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://<NGINX_PUBLIC_DOMAIN>/api/admin/logs/search?level=error&service=backend&dateRange=week&term=ECONNREFUSED" \
  | jq '.logs[] | {ts: .timestamp, level: .level, service: .service, msg: .message}'

Wire-level contract (VictoriaLogs HTTP API)

The backend’s shared/lib/melt/victorialogs-client.js is the read-side client for the Logs tab. It speaks VL’s /select/logsql/query and /select/logsql/hits endpoints. Ingestion goes through the OTel Collector (via logger.emit(...) in components/shared/lib/victorialogs-transport.js), not via a direct jsonline POST.

• VL base URL — VICTORIALOGS_URL (Ansible default http://victorialogs:9428).
• VL image — victoriametrics/victoria-logs:v1.50.0 (docker-compose.yaml:1834). The 1.50+ matters because that release introduced the canonical AccountID + ProjectID tenant headers; older VL builds ignore the multi-tenant header.
• Tenant header — VICTORIALOGS_TENANT_ID (default 0:0; format AccountID:ProjectID per VL 1.50+ canonical tenant header).
• Query timeout — VL_QUERY_TIMEOUT_MS (default 30000 ms; raised for 7-day security scans that aggregate more rows).
• Time field — _time. Every log line carries an ISO-8601 timestamp.
• Envelope parsing gotcha — for Node.js services, VL’s _msg field is the raw Winston JSON envelope. VictoriaLogsAdapter._normalizeRows (shared/lib/melt/victorialogs-client.js:237) projects _msg into the row-level message field. The Logs tab reads message, not _msg, so the TYPE column shows the human-readable text — never the literal JSON envelope {"level":"info","message":...}.

VictoriaLogs field reference

The fields below are VL-internal — VictoriaLogsAdapter rewrites most of them before the row reaches the Logs tab, so deployers usually do not see the raw names. They appear here for grep/LogSQL queries run directly against VL (e.g. from inside the VL container):

Configuration

VictoriaLogs has no environment opt-out — it is always-on because the fluentd driver ships container logs to it unconditionally and the admin logs UI queries it for backend endpoints. The in-app OTel SDK init is controlled separately by ENABLE_OBSERVABILITY (see Observability → Configuration).

Tunables worth knowing (not env vars, but operator-facing knobs in components/gov-chat-backend/services/logs-service.js):

• VL health probe is lazy on first call: 3 retries × 5 s backoff (HEALTH_PROBE_ATTEMPTS, HEALTH_PROBE_BACKOFF_MS in shared/lib/melt/victorialogs-client.js). Constructor does not block on VL reachability.
• VL-unreachable warning logging is rate-limited to 1 per minute per host (VL_UNREACHABLE_LOG_COOLDOWN_MS = 60_000, persisted to /tmp/vl-unreachable-ts so backend restarts do not reset the cadence).
• VL mode has no date-range cap (LogSQL is unbounded — restrict the date range explicitly if a query times out).

Failure modes and debugging

What happens when VL is unreachable

LogsService._vlOrThrow (static method) classifies connection-class failures into a typed VlUnavailableError. Connection errors caught:

• VictoriaLogsHealthError (raised by the client when the health-probe retries are exhausted)
• ECONNREFUSED, ENOTFOUND, ETIMEDOUT, ECONNABORTED (Node network errors)
• 5xx upstream responses from VL

The thrown VlUnavailableError carries:

err.statusCode === 503
err.body      === { error: 'vl_unreachable', message: '<human-readable>' }

The global error middleware in components/gov-chat-backend/index.js reads err.statusCode + err.body and renders them verbatim, so the API contract is 503 + {error: 'vl_unreachable', message}.

Validation / programmer errors (bad query params, etc.) propagate unchanged so the route layer keeps its existing 400 / 500 semantics.

Carve-out for /api/admin/system-health. The AdminDashboardService.getSystemHealth() path deliberately does not call _vlOrThrow for the VL hits query. The error-rate tile is one derived metric of a larger health payload, and the dashboard stays well-formed (errorRate: 0, a warn log line) on a VL outage rather than 503-ing the whole health response. All other VL read paths in LogsService and AdminDashboardService.debugYesterdayLogs funnel through _vlOrThrow.

Why VL is always-on. The fluentd logging driver is configured on every container regardless of the observability profile, so container logs reach VL unconditionally. Removing VL would break log shipping entirely — the only configurable opt-out is the in-app SDK (ENABLE_OBSERVABILITY).

Verify it worked

Before trusting the Logs tab in production, run through this checklist once on a fresh deployment:

1. VL is reachable from the backend container

   docker exec <stack>_backend wget -qO- "http://victorialogs:9428/health"
   # → "OK"
   

2. VL is receiving logs

   # Wait ~30s for the OTel Collector to flush the first batch, then:
   curl -sk "http://victorialogs:9428/select/logsql/query?query=*&limit=5"
   # → a non-empty JSON array; each row has _stream.service, _time, _msg
   

3. The Logs tab returns rows, not a vl_unreachable error

   Open https://<NGINX_PUBLIC_DOMAIN>/admin → Logs tab, run a search with the “Yesterday” preset. You should see rows; the TYPE column should show human-readable text (e.g. error — Login failed), never the literal JSON envelope {"level":"info","message":...}.

4. The admin role is enforced

   # Without an admin token: 401/403
   curl -sk -w "%{http_code}\n" \
     "https://<NGINX_PUBLIC_DOMAIN>/api/admin/logs?limit=1"
   

5. The rollover deprecation behaves as expected

   # Cron-style caller gets 410
   curl -sk -o /dev/null -w "%{http_code}\n" \
     -X POST -H "User-Agent: curl/8.0.0" \
     "https://<NGINX_PUBLIC_DOMAIN>/api/admin/logs/rollover"
   # → 410
   
   # Browser / admin client (no cron-style User-Agent) gets 200 + deprecated:true
   curl -sk -X POST \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H "Content-Type: application/json" \
     -H "User-Agent: Mozilla/5.0 (admin-ui)" \
     "https://<NGINX_PUBLIC_DOMAIN>/api/admin/logs/rollover"
   # → {"deprecated":true}
   

If any of these fails, cross-reference with Failure modes and debugging.

Deprecation of rollover

POST /api/admin/logs/rollover is deprecated and intentionally returns 410 Gone for cron callers. The user-agent check (regex against cron|curl|wget|httpie|python-requests|python-urllib|go-http-client) exists so that:

• Browser / admin clients see 200 {"deprecated":true} so the UI can display a friendly “no longer needed” message without breaking.
• Old cron jobs see 410 Gone so they fail loudly and are removed (rather than silently succeeding forever against a no-op endpoint).

If you still have a crontab entry calling /api/admin/logs/rollover, remove it. Logs are written to VL directly — no rollover step is required.

Related

• Observability → Configuration — the broader observability stack (Collector, retention, alerts)
• Observability → Dashboards — Grafana Service Logs + VictoriaLogs dashboards
• Troubleshooting — VL unreachable, missing telemetry
• Configuration → Keycloak admin guide — Admin role assignment
• Configuration → Environment variables — the full env var reference