Generation

Generation is the final RAG stage: the kept chunks are assembled into a prompt and sent to the LLM, which produces a grounded answer streamed back to the user. The behaviour is shaped by three knobs — the system prompt, the abstention rule, and an optional self-confidence token — plus translation when the user’s UI locale is not English.

This page is for developers tuning answer quality and operators debugging the response stream.

LLM defaults

The default model in docker-compose is meta-llama/Meta-Llama-3.1-8B-Instruct (set via VLLM_LLM_MODEL_ID, docker-compose.yaml:1179). The recommended and validated model is ibm-granite/granite-4.1-8b because it supports the OpenAI-compatible guided JSON (response_format={"type":"json_object"}) that dataprep needs for labelling — without guided JSON, labels fall back to per-chunk markdown parsing (slower, lower quality). The in-code fallback (genieai_chatqna.py:148) is ibm-granite/granite-3.3-2b-instruct, reached only when neither VLLM_LLM_MODEL_ID nor the legacy LLM_MODEL env var is set.

All are served by vLLM (OpenAI-compatible API).

System prompt

The built-in default (canonical source: genieai_chatqna.py:490-502) instructs the model to:

• Do not invent or assume information; if the answer is not in the provided content, inform the user that the information is unavailable.
• Use the user’s name, gender, age, preferences, and chat history to tailor and personalise responses.
• Keep answers informative but concise; provide detailed explanations only when necessary or explicitly requested.
• Prefer information contained in the context.

Override the whole prompt with CHATQNA_SYSTEM_PROMPT. Partial adjustments live in CHATQNA_ABSTENTION_INSTRUCTIONS.

Two-tier priority. Environment variable (highest) beats the built-in default (lowest). This lets each deployment tailor tone and rules without touching code.

Abstention

When the retrieved context does not contain the answer, the model is told to abstain — to say so plainly rather than hallucinate. Abstention is enforced by default (CHATQNA_ENFORCE_ABSTENTION=true). This is the single most important guardrail for a public-sector knowledge base, where a confident but wrong answer is far worse than an honest “I don’t have that information.”

Verifying abstention fired. When abstention fires, the streamed answer ends with the configured refusal phrase, and the SSE metadata event carries is_grounded: false. The abstained value is exposed as a metric attribute on chat_requests_total in VictoriaMetrics (not as a chatqna span attribute); the chatqna.orchestrate root span in VictoriaTraces reflects the same request for trace-level correlation.

Self-confidence

Optionally, the LLM appends a self-confidence token to its answer:

[[CONF:90]]

The pipeline parses this token separately and strips it from the visible text, so the user never sees the raw marker but the system gets a second certainty signal (independent of the reranker-derived score). Enable with LLM_SELF_CONFIDENCE_ENABLED=1 (default 0, off).

The confidence_score displayed to the user is _display_confidence (genieai_chatqna.py:407-417) — prefers self_confidence if available, falls back to the reranker-aggregated score otherwise.

Translation

For multilingual deployments, the streamed English answer can be translated into the user’s UI language by a dedicated translation model served by vllm-translation-guardrail. The default model is google/gemma-3-4b-it (docker-compose.yaml:557, configured via VLLM_TRANSLATION_MODEL_ID). Two optional alternatives commented in the env template are google/translategemma-4b-it and google/translategemma-12b-it.

Supported language codes are defined in components/gov-chat-frontend/src/i18n/locales/ (the same whitelist the UI uses): ar, bn, de, en, es, fr, id, man, pt, ru, st, sw, th, zh (Mandinka uses the man code only; there is no separate mnk file).

Two modes:

• Default (STREAMING_TRANSLATION_ENABLED=0) — generate the full English answer, then translate. The user waits for generation to finish before seeing any non-English content.
• Streaming (STREAMING_TRANSLATION_ENABLED=1) — translate incrementally as the English tokens arrive, so the user reads the target language live instead of waiting for a final flip. Uses an AST-based translateMarkdown-per-unit approach so markdown structure (headings, lists, code blocks, links) is preserved across translation boundaries, and re-appends the verbatim separator so the original English fallback is still available if the translation stream is interrupted.

For model trade-offs see Choosing models → Role 4: Translation & Guardrails.

Latency note. Translation adds at least one extra LLM call per request (often more in streaming mode). The translation call itself is not wrapped in a dedicated chatqna.* span — the request hook in genie-ai-overlay/tracing.py (lines 185-216) names the OPEA HTTP client spans as METHOD /path (e.g. POST /v1/chat/completions), so look for those under the chatqna.orchestrate root span in VictoriaTraces.

Multilingual chat history

When the UI locale differs from English (or when original_language is falsy in the request body, triggering auto-detection), the backend translates prior-turn user and assistant messages to English before they reach ChatQnA. This keeps the bge-large-en-v1.5 embedding space consistent regardless of UI language and feeds the multi-turn blending embedding only when MULTI_TURN_BLEND_ENABLED=true.

The translation cost is one translation call per prior turn at request time. Disable by serving English-only.

User profile enrichment

ChatQnA calls GET /api/me/context (the backend’s sanitized user profile endpoint) after JWKS validation. The returned fields (role, organisation, preferences — PII stripped by the backend) are injected into the LLM prompt as persona context. If the call fails, ChatQnA proceeds without enrichment — no hard error.

See Backend → API contracts → /api/me/context for the request/response shape.

Streaming & metadata envelope

The answer is streamed token-by-token from vLLM through ChatQnA to the client. The exact SSE wire format, the metadata envelope, and the stripping pass that removes internal conversation markers (|<-MSG->|, USER:, ASSISTANT:) are documented in Streaming & metadata events.

Knobs

Model requirement. The generation model must support OpenAI- compatible guided JSON (response_format={"type":"json_object"}) where structured output is expected downstream (dataprep labelling). Not every model does — see Choosing models.