Operations

Day-to-day operations for a running GENIE.AI deployment — backup, restore, updates, scaling, troubleshooting, and operator-facing diagnostic surfaces.

Deployment gets GENIE.AI running; this section keeps it running. It is for operators and administrators responsible for a live deployment: backing up data, applying updates, scaling, and diagnosing problems.

For real-time health, dashboards, and alerting, see Observability — this section assumes you already have the observability stack on and are watching it.

Pages in this section

  • Backup & restore — ArangoDB dump/restore and Kong configuration restore, including the network-name and password gotchas in the shipped scripts.
  • Updates — updating models, service images, and the stack; what forces re-ingestion; the rollback procedure.
  • Scaling — horizontal replicas and vertical GPU memory; what scales well and what does not.
  • Troubleshooting — common problems, trace-first / logs-second methodology, and pointers to deeper notes.
  • Admin Logs — the operator-facing Logs tab, the /api/admin/logs/* endpoints, the MELT seam, and the VL fallback configuration.
  • Ingestion Log — per-chunk ingestion progress feed, canonical messages, ArangoDB AQL access, and trace correlation.
  • Health Checks — /api/health and /api/admin/system-health endpoints, Docker Swarm healthcheck wiring, Kubernetes probe recipes, and failure-mode troubleshooting.
  • Security Hardening — Helmet defaults, CORS, CSP, and rate-limiting configuration; per-service transport security knobs.

Backup & Restore

Backing up and restoring every persistent store in a GENIE.AI deployment — ArangoDB knowledge base, PostgreSQL (Kong + Keycloak), Redis cache, document-repository uploads, and supporting volumes.

Distributed tracing (operator how-to)

W3C traceparent propagation across the RAG pipeline, the span taxonomy, and automatic PII filtering.

Updates

Updating models, service images, and the GENIE.AI stack — and which changes require re-ingesting the knowledge base.

Troubleshooting

Common GENIE.AI operational problems, where to look first, and pointers to the deeper troubleshooting notes in the deployment guides.

Admin Logs

Operator-facing Logs tab in the GENIE.AI admin UI — VictoriaLogs-backed log search, summaries, and the MELT seam.

Ingestion Log

The per-chunk ingestion progress log surfaced by dataprep — how to read it, where it lives, and what its messages mean.

Health Checks

Operational health endpoints — the public /api/health probe and the admin /api/admin/system-health diagnostic — with wiring examples for Swarm, Kubernetes, and Prometheus.

Security Hardening

Defense-in-depth controls built into the GENIE.AI backend and frontend — the sensitive-path blocker, Helmet security headers, robots.txt policy, and the user-profile static file surface.

Scaling GENIE.AI

How to scale every service in the GENIE.AI stack — stateless services scale freely, stateful services need replicas with shared storage, AI services are GPU-bound.