OPEA Microservices

OPEA Microservices

The OPEA overlay layer (genie-ai-overlay/) is where the RAG pipeline actually runs. This page is the architectural reference for the layer: which microservice owns which concern, what they exchange with each other and with the backend, and where the per-service contract surfaces live.

Audience: Integrators extending the RAG pipeline, deployers debugging inter-service issues, and reviewers tracing a failure through the stack. For tuning knobs and operator-facing config, see the RAG Pipeline section.

1. Service Map

graph TB
    BE[Backend / BFF] -->|Bearer token + traceparent| CHATQNA[chatqna<br/>Orchestrator]
    BE -->|admin client_credentials| DATAPREP[dataprep<br/>Ingestion]
    CHATQNA -->|Bearer token + traceparent| RETRIEVER[retriever<br/>Vector + graph search]
    CHATQNA -->|traceparent| RERANKER[reranker<br/>TEI cross-encoder]
    CHATQNA -->|traceparent| VLLM[vLLM<br/>LLM inference]
    CHATQNA -->|traceparent| TRANS[Translation<br/>vLLM-based]
    CHATQNA -->|traceparent| EMB[embedding<br/>TEI wrapper]
    RETRIEVER --> ADB[(ArangoDB)]
    EMB --> TEI_EMB[TEI Embedding]
    RERANKER --> TEI_RERANK[TEI Reranking]
    DATAPREP --> ADB
    DATAPREP --> DOC[Docling<br/>Content extraction]
    DATAPREP -->|LLM labeling| VLLM
    BE --> ADB
    DR[Document Repository] --> DATAPREP

2. Request Flow (ChatQnA Pipeline)

sequenceDiagram
    participant BE as Backend
    participant CQ as ChatQnA
    participant Em as Embedding
    participant Ret as Retriever
    participant Re as Reranker
    participant LLM as vLLM (textgen)
    participant TR as Translation
    participant ADB as ArangoDB

    BE->>CQ: POST /v1/chatqna (Bearer + traceparent + query + history)
    CQ->>CQ: Validate JWT (JWKS) + extract user
    CQ->>BE: GET /api/me/context (user profile for prompt enrichment)
    BE-->>CQ: { name, role, emailVerified }
    CQ->>Em: encode (query) — single batched call including history turns
    Em-->>CQ: { query_emb, history_embs }
    CQ->>Ret: search (embedding + filter labels + history blend)
    Ret->>ADB: ArangoSearch (hybrid: dense + BM25, optional graph traversal)
    ADB-->>Ret: ranked candidates
    Ret-->>CQ: top-N candidates with metadata
    CQ->>Re: rerank (candidates, query, history)
    Re-->>CQ: strategy-selected chunks with scores
    CQ->>CQ: assemble source_documents + compute confidence_score
    CQ->>LLM: chat (system + context + history + query, stream=True)
    LLM-->>CQ: SSE chunks
    CQ->>CQ: strip internal markers, collapse markdown blanks
    opt STREAMING_TRANSLATION_ENABLED=1
      CQ->>TR: stream-translate per markdown unit
      TR-->>CQ: translated unit
    end
    CQ-->>BE: SSE: data chunks + metadata event + [DONE]

What the backend sends to ChatQnA

What ChatQnA returns to the backend

Two modes, controlled by OPEA_STREAMING:

Non-streaming (OPEA_STREAMING=false):

{
  "text": "<final answer, internal markers stripped>",
  "source_documents": [
    { "document_id": "<doc_key>", "score": 0.91, "metadata": { ... } },
    ...
  ],
  "metadata": {
    "retrieval_confidence_score": 0.78,
    "confidence_score": 0.82,
    "is_grounded": true,
    "self_confidence": 67   // only when LLM_SELF_CONFIDENCE_ENABLED=1
  }
}

Streaming (OPEA_STREAMING=true):

• A sequence of SSE events. Each non-terminal event is data: {"text":"<delta>"}.
• A single metadata event (typically just before [DONE]) carries the source_documents and confidence metadata.
• Terminal: data: [DONE].

Internal-marker stripping. ChatQnA scrubs |<-MSG->|, USER:, ASSISTANT: markers and collapses excess blank runs before the text reaches the user or the translation pipeline. This is essential to prevent prompt-template leakage.

3. Trust Boundaries (Microservice Level)

Each OPEA microservice that touches the network has its own authentication posture:

Failure mode: if ChatQnA’s JWKS call to Keycloak fails (Keycloak down or unreachable), chatqna returns 401/500 — the backend surfaces this as a 5xx to the user. The JWKS cache absorbs short Keycloak hiccups; cache miss on every request is a sign Keycloak itself is misconfigured.

ChatQnA token validation rules

genie-ai-overlay/chatqna/keycloak_token_validator.py (post-commit 2e5cb17b6 — final state) implements a standard OIDC Resource Server posture:

• No anonymous access. A request without a Bearer token (or with a malformed Authorization header) is rejected with 401 UNAUTHENTICATED before the JWKS call. There is no service-account fallback path for chatqna — every chat call must carry the originating user’s JWT.
• Per-request token forwarding. ChatQnA receives the user’s Bearer token on every request (forwarded by the backend, which itself received it from Kong). Dataprep, in contrast, uses a dedicated service account (KC_DATAPREP_CLIENT_ID/SECRET) and never sees a user token.
• JWKS-based signature validation. Tokens are verified against the Keycloak JWKS (no shared secret, no HS256). The validator fetches the JWKS from KEYCLOAK_INTERNAL_URL/realms/<KEYCLOAK_REALM>/protocol/openid-connect/certs and caches it.
• What is validated. signature (via JWKS), iss (must match the configured realm), exp (rejects expired tokens).
• What is NOT validated. azp (Authorized Party) is deliberately NOT checked — this aligns ChatQnA with gov-chat-backend and document-repository, which all follow standard OIDC Resource Server semantics where audience verification alone (aud, when present) is enough. The per-flavor KEYCLOAK_CLIENT_ID override was removed because it broke mobile auth (mobile tokens carry a different azp than web tokens).
• verify_aud: False. Keycloak 26+ uses aud=account on user tokens; ChatQnA trusts iss + exp + signature instead.

Failure mode: an unauthenticated request returns 401; a malformed token returns 401 TOKEN_INVALID; a token whose iss does not match the configured realm returns 401 INVALID_ISSUER. All three are surfaced to the user as 401 from the BFF (the backend does not auto-retry).

4. Telemetry Architecture

All OPEA services emit OTel spans through the shared genie-ai-overlay/tracing.py. The trace context flows via W3C traceparent headers — backend injects on the first hop, every downstream service extracts and propagates.

Dataprep spans (instrumented in genieai_dataprep_arangodb.py)

ChatQnA spans

For full per-phase breakdown and an end-to-end debug recipe (fetch a trace by ID, find the bottleneck, decode the traceparent chain), see the Debugging with Tracing & Logs developer KB (.claude/rules/DEBUGGING-TRACING.md in the repo).

5. Per-Request Overrides (ChatQnA)

The request body fields below act as per-request overrides over the deployment defaults (RERANKER_TOP_N, RERANKING_STRATEGY, RETRIEVER_ARANGO_K, RETRIEVER_ARANGO_FETCH_K, RETRIEVER_ARANGO_FILTER_STRATEGY, CONFIDENCE_RANK_DECAY, RERANKER_SCORE_CALIBRATION, RERANKER_SCORE_TEMPERATURE). They are top-level fields on the ChatQnA request (e.g. top_n, reranking_strategy), not nested inside a wrapper block — see genie-ai-overlay/chatqna/genieai_chatqna.py (chat_request.top_n, chat_request.reranking_strategy, GenieaiRetrieverParms, GenieaiRerankerParms). This is how the same chatqna service can serve different retrieval strategies for different user requests without restart.

Safety: per-request overrides cannot weaken global safety knobs (e.g. cannot disable Contextual Retrieval once it is enabled at the service level). The override is a forwarding mechanism, not an authentication-bypass one.

6. The Label-Filter Data Contract

A subtle but load-bearing detail: OPEA’s framework drops custom fields that aren’t part of its known schema. The retriever therefore receives category labels encoded inside the search_start parameter (a string with a ::labels: separator), not as a separate top-level field.

# chatqna → retriever
from core.label_contract import encode_filter_labels

search_start = encode_filter_labels("chunk", ["Agriculture.Crops.Corn", "Agriculture.Crops.Cassava"])
# => "chunk::labels:Agriculture.Crops.Corn,Agriculture.Crops.Cassava"

The retriever parses this back out (decode_filter_labels(search_start) in genie-ai-overlay/core/label_contract.py) and uses the label list to build the AQL FILTER clause before the vector search. If a chunk arrives without an expected label (dataprep returned []), the chunk is still stored but will not surface under a strict label filter at query time.

For the label-driven retrieval behavior (label selector prompt, taxonomy alignment, common pitfalls), see Knowledge Base — Labelling & Taxonomy.

7. Inter-Service Failure Modes

8. Verifying After a Stack Update

After upgrading any OPEA service image, run this verification sequence before opening it to users:

# 1. All OPEA services are healthy in Swarm
docker service ls | grep -E "chatqna|retriever|reranker|dataprep|embedding|textgen"

# 2. Internal health endpoints respond
docker exec $(docker ps --format "{{.Names}}" | grep chatqna | head -1) curl -s http://localhost:8888/health
docker exec $(docker ps --format "{{.Names}}" | grep retriever | head -1) curl -s http://localhost:7000/health
docker exec $(docker ps --format "{{.Names}}" | grep reranker | head -1) curl -s http://localhost:8000/health

# 3. End-to-end smoke: ask a known question, verify a trace lands in VictoriaTraces
# (use the API contracts doc for curl examples with a Keycloak ROPC token)
# Then query VictoriaTraces for the trace:
TRACE_ID=<from response headers>
docker exec $(docker ps --format "{{.Names}}" | grep backend | head -1) \
  curl -s "http://victoriatraces:10428/select/jaeger/api/traces/$TRACE_ID" | jq '.data[0].spans | length'
# Expected: an auto-instrumented backend HTTP server span (named after the route, e.g. `POST /api/chat`) plus chatqna.orchestrate, chatqna.reranker_selection, retriever.hybrid_search, reranker.rerank, and reranker.tei_invoke — at least 6 spans total, more if dataprep runs

A trace that returns fewer spans than expected means one stage silently no-ops — usually a service that’s reachable but misconfigured (e.g. TEI pointing at the wrong embedding model).

9. Where to Go Next