Operations on GENIE.AI

Backup & Restore

Mon, 01 Jan 0001 00:00:00 +0000

If you operate a GENIE.AI deployment and own the recovery story, this page is for you. A complete GENIE.AI recovery requires more than the knowledge base. Four stores hold state that is not reproducible from the codebase alone, plus a handful of supporting volumes. Losing any of them costs more than a rebuild incurs. This page covers all of them and the order to restore them in.

In an emergency (TL;DR). Restore order is fixed and non-negotiable: PostgreSQL → Redis → Document-repository uploads → ArangoDB → Supporting volumes. Each layer depends on the one below; skipping or reordering leaves the stack in an inconsistent state. Jump to the Restore procedure if you already know how to back up; everything below is for the operator who needs to build a backup routine from scratch.

Distributed tracing (operator how-to)

Mon, 01 Jan 0001 00:00:00 +0000

Tracing is what makes a GENIE.AI user query legible. Because every service propagates a single W3C traceparent header and every RAG stage emits a span, one user question produces one contiguous trace that can be read end-to-end in Grafana — revealing exactly where time went and which chunks were retrieved.

Trace propagation

The trace starts at the edge. Kong creates the root span and injects a traceparent header into the request. Each downstream hop reads that header, joins the trace as a child span, and re-injects the header for the next hop:

Updates

Mon, 01 Jan 0001 00:00:00 +0000

GENIE.AI has three kinds of update, with different blast radii. The critical thing to know is which updates force a knowledge-base re-ingestion. Plan backups before every change and keep the previous-known-good tag for rollback.

Prerequisites

A working deployment (Compose or Swarm) with .env populated.
GENIE_AI_GLOBAL_TAG rollback target noted before starting (per-service tags like GENIE_AI_BACKEND_IMAGE_TAG override just the backend).
For model pulls: HUGGING_FACE_HUB_TOKEN set in .env. Without it, vLLM silently stalls while retrying the private-model download with no error in the container logs.
docker service ls (Swarm) or docker compose ps (Compose) returns the stack — required to copy service names verbatim in the commands below.
Backup taken (see Backup & restore).

The re-ingestion rule

Change	Re-ingest required?	Restart target (Swarm service)
LLM (`VLLM_LLM_MODEL_ID`)	No — model is read at query time	`genieai_vllm` (model-serving vLLM)
Reranker (`RERANKER_MODEL_ID`)	No — query-time re-score	`genieai_tei_reranker` (the TEI serving the reranker)
Translation (`VLLM_TRANSLATION_MODEL_ID`)	No — query-time	`genieai_vllm-translation-guardrail` (the vLLM serving the translation model — not the `translation` OPEA wrapper)
Embedding model (`EMBEDDING_MODEL_ID`)	Yes — vectors live in the old model’s space	`genieai_tei` + full re-ingest
Labelling taxonomy (`LABEL_SELECTOR_SYSTEM_PROMPT`)	Recommended — labels are stored on chunks at ingest time	`genieai_dataprep-arango-service`
Chunking / dataprep logic	Yes — chunk boundaries and content change	`genieai_dataprep-arango-service` + re-ingest affected docs

Changing the embedding model is the expensive update. Plan a full re-ingest (see Knowledge base → Ingestion) when you change EMBEDDING_MODEL_ID. Everything else is a service restart.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This page lists the problems operators hit most often, the first place to look, and where the deeper notes live. For install-time problems, see the troubleshooting sections of the deployment guides.

Goal

Localise any operational issue to a single component, stage, or config value in minutes — without restarting services or grepping unbounded logs.

Prerequisites

A running GENIE.AI stack (Swarm or Compose) reachable via SSH on the manager node. See Installation & Configuration Guide for first-time setup.
Docker access: docker service ls, docker service logs, docker service ps must work for the operator user (member of the docker group, or root on the Swarm manager).
Observability stack on (ENABLE_OBSERVABILITY=1 in .env): Grafana reachable at https://<NGINX_PUBLIC_DOMAIN>/grafana/. Without it, the trace-first playbook below reduces to log-grep + admin Query Inspector. See Observability → Configuration.
Keycloak admin password (KEYCLOAK_ADMIN_PASSWORD in .env) for the Authentication / token errors section.
GPU node SSH access (read-only) for the GPU OOM section — VRAM headroom is read directly from the GPU host via nvidia-smi (no Grafana panel exposes GPU memory yet).

General principle: trace, then logs, then code

For any “why did this answer come out wrong / slow / empty” question, the fastest path is the trace, not the logs:

Admin Logs

Mon, 01 Jan 0001 00:00:00 +0000

The Logs tab in the admin dashboard (reachable at https://<NGINX_PUBLIC_DOMAIN>/admin → “Logs”) is the operator-facing surface for searching container logs across all GENIE.AI services. It is backed by VictoriaLogs via the backend’s MELT seam (MELT_PROVIDER=victorialogs, an abstraction layer in components/shared/lib/melt/ named for Metrics / Events / Logs / Traces — a future-proof seam that lets the storage backend swapped without changing call sites), with graceful degradation when VL is unreachable.

Ingestion Log

Mon, 01 Jan 0001 00:00:00 +0000

The ingestion log is the per-chunk progress feed that dataprep writes during document ingestion. Every chunk event — parse, label, embed, context generation, fallbacks — produces one row, addressed by file_id. The log is the operator’s primary tool for answering “why did this document land in failed state?” or “what step was reached before the pipeline broke?”

This page consolidates what the ingestion log is, how to read it (UI and ArangoDB), the canonical messages you will learn to grep for, and how to correlate a log entry with the matching RAG-pipeline trace.

Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

For operators and load balancers. GENIE.AI exposes two layered health endpoints on the backend service (port 3000):

/api/health — a public, dependency-free liveness probe (no auth).

/api/admin/system-health — a richer admin diagnostic that probes ArangoDB, Keycloak, and the backend itself (requires the admin realm role).

Use the first for liveness, the second for readiness and post-deploy smoke tests.

This page covers what each endpoint reports, how to wire them into your infrastructure, and how to interpret the results.

Security Hardening

Mon, 01 Jan 0001 00:00:00 +0000

For deployers and security reviewers. This page documents the security controls the GENIE.AI stack ships with by default: which requests are blocked before they reach the router, which HTTP headers Nginx and the backend set (or strip), which static surfaces are intentionally exposed, and how the document download flow actually gates access. Use it to audit your deployment, confirm Kong/Nginx is not re-enabling what the backend blocks, and adjust policy when reverse-proxying.

Scaling GENIE.AI

Mon, 01 Jan 0001 00:00:00 +0000

GENIE.AI runs ~30 services in docker-compose.yaml (excluding init/migration helpers and named volumes). Each scales differently depending on its workload profile. This guide groups services by scaling category and explains what to watch.

Prerequisites

A deployed stack (see Install guide or Docker Swarm setup).
Docker CLI access on a Swarm manager node (docker service ls returns the stack).
For GPU scaling: at least one Swarm node labelled gpu=true (see GPU deployment and Topologies).
For observability scaling: ENABLE_OBSERVABILITY=1 set in .env (or enable_observability: "1" in the Ansible vars).

Stack-name caveat: the Ansible playbook defaults stack_name: genieai (see deploy/ansible/group_vars/all.yml), so the examples below assume services are prefixed genieai_<service>. If you overrode stack_name at deploy time (--extra-vars stack_name=foo), use foo_<service> instead.