Health Checks

For operators and load balancers. GENIE.AI exposes two layered health endpoints on the backend service (port 3000):

1. /api/health — a public, dependency-free liveness probe (no auth).
2. /api/admin/system-health — a richer admin diagnostic that probes ArangoDB, Keycloak, and the backend itself (requires the admin realm role).

Use the first for liveness, the second for readiness and post-deploy smoke tests.

This page covers what each endpoint reports, how to wire them into your infrastructure, and how to interpret the results.

Prerequisites

• A reachable GENIE.AI deployment via one of:
  • Docker Compose — see Deploy → Docker Compose Setup.
  • Docker Swarm — see Deploy → Docker Swarm Setup.
  • Ansible automated Swarm — see Deploy → Install Guide.
• The stack’s public domain (NGINX_PUBLIC_DOMAIN in .env, default localhost) and HTTPS port (NGINX_HTTPS_PORT, default 443).
• For the admin endpoint: a Keycloak user with the admin realm role — see Configure → Keycloak Admin Guide.
• For liveness wiring: write access to your load balancer, Kubernetes probe, or monitoring system that will consume the endpoint.

Endpoint summary

flowchart TD
    Start([Probe triggered]) --> Q{What do you want<br/>to detect?}
    Q -->|"Is the process alive?"| Live["/api/health<br/>(public, no deps)"]
    Q -->|"Is the whole stack ready?"| Ready["/api/admin/system-health<br/>(admin JWT required)"]
    Live -->|200| LiveOK["Backend responding"]
    Live -->|500| LiveDead["Backend crashed<br/>or import broken"]
    Ready -->|200| ReadyOK["Stack healthy<br/>+ healthServices[] snapshot"]
    Ready -->|401| ReadyNoAuth["Missing/invalid token"]
    Ready -->|403| ReadyNoRole["JWT lacks 'admin' role"]
    Ready -->|500| ReadyErr["Downstream failure<br/>(see error.message)"]

/api/health — public liveness probe

The simplest endpoint, defined in components/gov-chat-backend/index.js:711. It is registered in the public-paths list (middleware/keycloak-auth-middleware.js:10) so Keycloak authentication is skipped.

Request

# Compose (host): backend port 3000 is internal-only — go through nginx
curl -sk https://<NGINX_PUBLIC_DOMAIN>[:<NGINX_HTTPS_PORT>]/api/health

# Inside the Swarm overlay network (debug only): backend port 3000 is exposed
# internally; use the in-network DNS name "backend"
docker exec <stack>_backend wget -qO- http://localhost:3000/api/health

{
  "status": "ok",
  "serverTime": "2026-09-18 14:23:45",
  "uptime": "86423 seconds"
}

Response fields

Failure modes

• 500 {"message": "Health check failed"} — the handler itself threw (Express app crashed mid-request or a downstream import broke). The body is plain {message} — no status field — so a load balancer can distinguish “backend alive but sick” from “backend gone”.

When to use it

• Docker Swarm healthcheck — already configured in docker-compose.yaml for the backend service (see Wiring examples). Swarm reaps the container when the probe fails five times in a row.
• Kubernetes liveness probe — set livenessProbe.httpGet.path=/api/health. The endpoint is dependency-free, so it cannot false-positive when ArangoDB is briefly down — pair it with a dependency-checking readiness probe at the orchestrator layer if you need to drain traffic during a backend dependency outage.
• Reverse-proxy upstream checks — Kong/Nginx health_check blocks. Kong routes /api/* to the backend (see api-gateway-solution/new-config/kong_config.json:82).

When NOT to use it

• For readiness, do not use /api/health — it does not probe ArangoDB or Keycloak. Wire readiness to /api/admin/system-health instead (or a custom dependency-aware probe).
• For liveness of the whole stack (frontend, OPEA, document repository), /api/health is backend-only. Probe each service’s own port or rely on the Application Metrics dashboard.

/api/admin/system-health — admin diagnostic

A richer endpoint that probes the actual dependencies the backend talks to. Defined in components/gov-chat-backend/routes/admin-routes.js:65. Registered for /api/admin in the ROUTE_CONFIGS table at index.js:495 (entry { file: 'admin-routes', paths: ['/api/admin'] }), mounted at runtime by registerRoutes() called at index.js:766, then guarded by keycloakAuthMiddleware.requireAdmin at admin-routes.js:45 — so both authentication and the admin realm role are required.

Request

# Pull a realm token (Direct Access Grants are disabled by default —
# see /docs/operate/server-testing/ for the temporary-enable pattern).
# Defaults from env: KEYCLOAK_REALM=genie, KEYCLOAK_CLIENT_ID=genie-app.
ADMIN_TOKEN=$(curl -sk -X POST \
  "${KEYCLOAK_URL:-https://${NGINX_PUBLIC_DOMAIN}/auth}/realms/${KEYCLOAK_REALM:-genie}/protocol/openid-connect/token" \
  --data-urlencode "grant_type=password" \
  --data-urlencode "client_id=${KEYCLOAK_CLIENT_ID:-genie-app}" \
  --data-urlencode "username=<admin-user>" \
  --data-urlencode "password=<admin-password>" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

curl -sk -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://<NGINX_PUBLIC_DOMAIN>[:<NGINX_HTTPS_PORT>]/api/admin/system-health

Response shape

AdminDashboardService.getSystemHealth() (services/admin-dashboard-service.js:52) returns:

{
  "metrics": {
    "systemUptime": 99.99,
    "avgResponseTime": 142,
    "errorRate": 0.21,
    "monthlyActiveUsers": 87
  },
  "trends": {
    "uptime": 0.1,
    "responseTime": -3.2,
    "errorRate": 0.0,
    "activeUsers": 4.0
  },
  "resourceUsage": {
    "cpu": 23,
    "memory": 41,
    "storage": 12
  },
  "healthServices": [
    { "id": "apiServices", "name": "API Services",     "status": "good" },
    { "id": "database",     "name": "Database",         "status": "good" },
    { "id": "cache",        "name": "Cache",            "status": "good" },
    { "id": "storage",      "name": "Storage",          "status": "good" },
    { "id": "messageQueue", "name": "Message Queue",    "status": "good" },
    { "id": "externalApi",  "name": "External API",     "status": "good" }
  ]
}

Failure modes

• 401 {"error":"TOKEN_INVALID","message":"Missing or malformed Authorization header","details":{}} — no Authorization: Bearer header. Returned by the auth middleware at middleware/keycloak-auth-middleware.js:78 (the __tests__/routes/admin.test.js:140 test mocks this middleware to a shorter body, so its assertion body does not match the production shape).
• 403 {"error":"FORBIDDEN","message":"Admin access required","details":{}} — JWT is valid but lacks the admin realm role (test: __tests__/routes/admin.test.js:149).
• 500 {"message":"An unexpected error occurred","error":"<NODE_ENV-only>"} — getSystemHealth() threw (ArangoDB unreachable, malformed config). Handled by the error middleware at index.js:789.

When to use it

• Operator dashboard — the Admin UI’s Overview tab polls this on load and surfaces the result as colored status pills.
• Readiness probe for Kubernetes / Swarm when the load balancer should remove the backend from rotation only when the whole stack is sick.
• Post-deploy smoke test — the canonical first check after every release.

Wiring examples

Docker Swarm healthcheck (already configured)

docker-compose.yaml (root, backend service block at line 484) ships with:

healthcheck:
  test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/api/health"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 90s

The probe hits /api/health over the in-container loopback — no network hops, no auth. Swarm marks the task unhealthy after five consecutive failures (matching retries: 5) and replaces it. The start_period: 90s gives the Node.js process time to load AdminDashboardService and other heavy modules on first boot.

Backend image note: the runtime image is node:22-slim and ships wget (the entrypoint needs it to wait for ArangoDB). If you ever switch to a node:22-alpine base, install wget before the HEALTHCHECK line or the probe will silently fail.

Kubernetes probes

livenessProbe:
  httpGet:
    path: /api/health
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /api/admin/system-health
    port: 3000
  httpHeaders:
    - name: Authorization
      value: Bearer <service-account-token>
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

Important: the readiness probe needs a service-account JWT with the admin realm role. Mint one via Keycloak’s client_credentials grant with a dedicated genie-readiness client (recommended), or reuse the genie-app client if its service account carries the role. The default genie-app client does not carry the admin role by default — check Configure → Keycloak Admin Guide.

Prometheus / blackbox_exporter

/api/health is plain JSON, not Prometheus-format. Wire it through blackbox_exporter if you want to scrape it:

# prometheus.yml
scrape_configs:
  - job_name: genie_health
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['<NGINX_PUBLIC_DOMAIN>:<NGINX_HTTPS_PORT>']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Verify it worked

After wiring, exercise both endpoints end-to-end from the gateway host:

# 1. Public liveness — must return 200 + status:"ok" without a token
curl -sk -w "\nHTTP %{http_code}\n" \
  https://<NGINX_PUBLIC_DOMAIN>[:<NGINX_HTTPS_PORT>]/api/health

# 2. Admin diagnostic — must return 200 + the dependency snapshot
curl -sk -H "Authorization: Bearer $ADMIN_TOKEN" \
  -w "\nHTTP %{http_code}\n" \
  https://<NGINX_PUBLIC_DOMAIN>[:<NGINX_HTTPS_PORT>]/api/admin/system-health

# 3. Negative case — admin endpoint without a token must return 401, not 500
curl -sk -w "\nHTTP %{http_code}\n" \
  https://<NGINX_PUBLIC_DOMAIN>[:<NGINX_HTTPS_PORT>]/api/admin/system-health

# 4. From inside the Swarm network (Compose/Swarm overlay), the in-container
#    path uses port 3000 and DNS name "backend"
docker exec <stack>_backend wget -qO- http://localhost:3000/api/health

Anything else means a wiring mistake (typically: probe hitting a port the proxy does not expose, or the admin role missing from the service account).

Troubleshooting

Related

• Troubleshooting — broader diagnostic methodology (trace-first, logs-second).
• Admin Logs — when health is green but a service is silently degraded, the Logs tab shows you why.
• Observe → Dashboards — the Service Health Overview dashboard is the human-facing complement to these endpoints.
• Configure → Keycloak Admin Guide — how to assign the admin realm role for the diagnostic endpoint.
• Configure → CORS & CSP — when your monitoring system originates from outside the configured CORS_ALLOWED_ORIGINS and the browser blocks the probe.
• Deploy → Docker Swarm Setup — full deployment context (node labels, secrets, profiles).
• Architecture Overview — where the backend, Kong, ArangoDB, and Keycloak fit in the system topology.