RAG Pipeline on GENIE.AI

Ingestion and chunking strategies

Mon, 01 Jan 0001 00:00:00 +0000

Ingestion is step 0 of the RAG pipeline. It runs before the embedding model sees anything, before the retriever ever queries ArangoDB, and before any context prefix is generated. Its job is to turn an uploaded file into a list of well-sized, well-labelled chunks whose vectors (computed downstream by the embedding model) will actually match what users ask.

This page consolidates the RAG-specific side of ingestion: how the chunker sizes chunks per format, how the Contextual Retrieval prefix is generated (per-chunk vs doc-level), and how the decoupled-labelling mode keeps the labeler honest. For the operator side — supported formats, file-size limits, ClamAV scanning, the file lifecycle states — see Knowledge base → Ingestion.

Pipeline Architecture

Mon, 01 Jan 0001 00:00:00 +0000

The RAG pipeline is orchestrated by the ChatQnA mega-service (genie-ai-overlay/chatqna/genieai_chatqna.py), a Python/FastAPI service built on the OPEA mega-service framework. It wires together five specialist microservices into a directed acyclic graph (DAG) and streams the final answer back to the client.

This page describes the lifecycle of a single chat request — from the browser submitting a query to the streamed tokens reaching the screen — and the configuration surface that governs it.

Retrieval

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval is the stage that decides which chunks of the knowledge base the LLM is allowed to see. GENIE.AI does not rely on a single signal: it fuses dense-vector search with lexical (BM25) search, and can additionally walk the knowledge graph to surface related chunks. The result is high recall (semantic matches the exact-keyword search would miss) without losing precision on proper nouns, codes, and rare terms that dense models handle poorly.

Reranking

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval hands the reranker a candidate set of chunks; the reranker’s job is to re-order them by true relevance to the query and decide how many (if any) are good enough to send to the LLM. A cross-encoder reranker is more accurate than the bi-encoder used for retrieval because it reads the query and each chunk together, but it is also slower — which is why it runs only on the small candidate set, not the whole corpus.

Generation

Mon, 01 Jan 0001 00:00:00 +0000

Generation is the final RAG stage: the kept chunks are assembled into a prompt and sent to the LLM, which produces a grounded answer streamed back to the user. The behaviour is shaped by three knobs — the system prompt, the abstention rule, and an optional self-confidence token — plus translation when the user’s UI locale is not English.

This page is for developers tuning answer quality and operators debugging the response stream.

Contextual Retrieval

Mon, 01 Jan 0001 00:00:00 +0000

Standard retrieval embeds each chunk in isolation. A chunk removed from its section can be hard to match: a paragraph about “renewing it” will not retrieve well for a query about “permit renewal” because the chunk alone does not say what “it” is. Contextual Retrieval (an Anthropic-style technique) fixes this by prepending a short, LLM-generated doc-context prefix to each chunk before it is embedded and labelled, so the chunk’s vector carries its surrounding meaning.

Choosing and Configuring Models

Mon, 01 Jan 0001 00:00:00 +0000

Overview

GENIE.AI uses a multi-model architecture where different AI models serve distinct purposes across the entire framework — from document ingestion and knowledge graph construction through to retrieval, reranking, and response generation. Not every model is suitable for every role. Choosing the wrong model for a given purpose will cause silent failures — ingestion batches being skipped, labels returning garbage, or graph data not being created.

This guide explains what each model does, what capabilities it must have, what license it is released under, and provides specific recommendations for both open-source and commercial models across different hardware tiers.

Data Labelling Strategy

Mon, 01 Jan 0001 00:00:00 +0000

Labelling is what keeps a multi-domain knowledge base from bleeding across topics. Every chunk is assigned one or more labels drawn from the service-category taxonomy, and at query time the retriever uses those labels to return only on-topic chunks. Without labels, a question about one service could pull in unrelated chunks from another.

This page is for developers configuring dataprep and operators debugging why retrieval is (or is not) constrained to the right topic.

Streaming & Metadata Events

Mon, 01 Jan 0001 00:00:00 +0000

When the backend calls POST /v1/chatqna with stream: true, ChatQnA streams the answer back as Server-Sent Events (SSE). Each event is a single line data: <json>\n\n; the terminal event is data: [DONE]\n\n. Between the first and the last, ChatQnA interleaves per-token content events and a single terminal metadata event carrying source_documents, retrieval_confidence_score, confidence_score, is_grounded, and (when LLM_SELF_CONFIDENCE_ENABLED=1) self_confidence.

This page is for backend developers parsing the stream, frontend developers rendering the citation list, and operators debugging what the user actually receives.

Multi-Turn Retrieval

Mon, 01 Jan 0001 00:00:00 +0000

By default ChatQnA retrieval is stateless: only the last user message is embedded and searched against the knowledge base. This is fine for standalone questions but breaks for multi-turn follow-ups:

User (turn 1): How do I renew my residence permit? User (turn 2): Can you elaborate on this?

“Can you elaborate on this?” has no semantic content on its own. Embedding it in isolation retrieves nothing useful, even though the conversation is clearly about permit renewal.

Streaming Translation

Mon, 01 Jan 0001 00:00:00 +0000

For backend developers and deployers. This page explains the STREAMING_TRANSLATION_ENABLED toggle that fixes issue #829: when a user asks in Spanish, the chat must show Spanish text as it streams, not an English stream that flips at the end. The flag is off by default — turning it on requires the GPU translation service and a recent enough deployment to support unit boundaries.

The problem (issue #829)

Before this feature, the flow for a Spanish-speaking user was:

Per-Request Overrides

Mon, 01 Jan 0001 00:00:00 +0000

Most RAG knobs in GENIE.AI are deployment-wide (set in .env). A subset can be overridden on a single chat request via the OpenAI-compatible retriever_parameters and reranker_parameters fields on the request body — useful for an admin tool that wants to A/B a chunk count, for an experiment harness that varies top_n per query, or for an integration test that needs to lock in a known reranker strategy.

This page is for backend developers building integrations against ChatQnA and operators experimenting with retrieval and reranking knobs.

Model Capability Cache

Mon, 01 Jan 0001 00:00:00 +0000

For backend developers and operators. The genie-ai-overlay/core/model_cache.py module probes a remote vLLM endpoint’s /v1/models route, caches the result for MODEL_DETECT_TTL seconds (default 60), and serves the cached value on subsequent calls. On expiry the endpoint is re-probed; on probe failure the last known value is returned. This lets services pick up model changes on the GPU node without a restart.

This page documents the cache contract, the env var, the failure modes, and how to clear the cache for testing.