Operations on GENIE.AI

Backup & Restore

Mon, 01 Jan 0001 00:00:00 +0000

The two things you must be able to restore after a failure are the knowledge base (ArangoDB — chunks, embeddings, graph, labels) and the gateway configuration (Kong routes and plugins). GENIE.AI ships scripts for both.

ArangoDB backup

The backup script (components/arangodb/dump.sh) dumps every database using arangodump inside the arangodb/arangodb:3.12 container, into a timestamped directory:

/root/arango_backups/<YYYYMMDDHHMMSS>/

It requires the arango-vector-db container running on the chatqna_default network. Run it from a Swarm node that can reach the database container.

Updates

Mon, 01 Jan 0001 00:00:00 +0000

GENIE.AI has three kinds of update, with different blast radii. The critical thing to know is which updates force a knowledge-base re-ingestion.

The re-ingestion rule

Change	Re-ingest required?	Why
LLM (`VLLM_LLM_MODEL_ID`)	No	The LLM reads chunks at query time; it does not change stored data. Restart vLLM.
Reranker (`RERANKER_MODEL_ID`)	No	Reranking is a query-time re-score. Restart the reranker / TEI.
Translation (`VLLM_TRANSLATION_MODEL_ID`)	No	Translation is query-time. Restart the translation service.
Embedding model (`EMBEDDING_MODEL_ID`)	Yes	Existing chunk vectors are in the old model’s space. Re-ingest everything.
Labelling taxonomy	Recommended	Labels are stored on chunks at ingest time; a changed taxonomy needs re-labelling.
Chunking / dataprep logic	Yes	Chunk boundaries and content change.

Changing the embedding model is the expensive update. Plan a full re-ingest (see Knowledge base) when you change EMBEDDING_MODEL_ID. Everything else is a service restart.

Scaling

Mon, 01 Jan 0001 00:00:00 +0000

GENIE.AI runs on Docker Swarm, so horizontal scaling is a docker service scale command. Vertical scaling — especially GPU memory — is a configuration change per service. The two are independent and solve different problems.

Horizontal scaling (replicas)

Scale a stateless service to more replicas:

docker service scale genieai_<service>=<replicas>

Services that scale well (stateless, query-time):

Service	Scales?	Notes
Backend (BFF)	Yes	Stateless behind Kong.
ChatQnA	Yes	Stateless orchestrator.
Retriever	Yes	Stateless; shares the same ArangoDB.
Reranker / TEI	Yes	Stateless.
vLLM	With care	Each replica needs its own GPU memory budget; see below.
Dataprep	Yes (ingest)	More replicas = more parallel ingestion throughput.
ArangoDB	No (single-node)	The bundled ArangoDB is single-node; do not scale it. Add resources, not replicas.

ArangoDB is the bottleneck you cannot replica your way out of. The single-node database handles the whole retrieval load. If retrieval latency climbs under load, first check the database’s resources and query performance (see the VictoriaMetrics dashboard), not the retriever count.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This page lists the problems operators hit most often, the first place to look, and where the deeper notes live. For install-time problems, see the troubleshooting sections of the deployment guides.

General principle: trace, then logs, then code

For any “why did this answer come out wrong / slow / empty” question, the fastest path is the trace, not the logs:

Find the query’s trace in the Trace explorer.
Walk the RAG stage spans to see which stage is slow or returns nothing.
Pivot to the service logs for that stage’s error detail.

See Observability for how.