RAG Pipeline on GENIE.AI

Pipeline Architecture

Mon, 01 Jan 0001 00:00:00 +0000

The RAG pipeline is orchestrated by the ChatQnA mega-service (genie-ai-overlay/chatqna/), a Python/FastAPI service built on the OPEA mega-service framework. It wires together five specialist microservices into a directed acyclic graph (DAG) and streams the final answer back to the client.

Service roles

Stage	Service	Stack	Role
Orchestration	ChatQnA	Python / FastAPI	Owns the DAG; calls each stage; enforces abstention and confidence.
Embedding	TEI Embedding	Hugging Face TEI	Vectorises the query (and chunks at ingest time).
Retrieval	Retriever	Python / ArangoDB	Hybrid vector + graph search over indexed chunks.
Reranking	TEI Reranker	Hugging Face TEI	Re-scores retrieved chunks by query relevance.
Generation	vLLM	vLLM (OpenAI-compatible)	Produces the grounded answer.
Translation	Translation (optional)	TranslaTron or vLLM	Translates the stream into the user’s UI language.

All services are containerised and communicate over the internal Docker network. Only the backend (BFF) and the public Nginx gateway are reachable from outside the cluster.

Retrieval

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval is the stage that decides which chunks of the knowledge base the LLM is allowed to see. GENIE.AI does not rely on a single signal: it fuses dense-vector search with lexical (BM25) search, and can additionally walk the knowledge graph to surface related chunks. The result is high recall (semantic matches the exact-keyword search would miss) without losing precision on proper nouns, codes, and rare terms that dense models handle poorly.

Reranking

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval hands the reranker a candidate set of chunks; the reranker’s job is to re-order them by true relevance to the query and decide how many (if any) are good enough to send to the LLM. A cross-encoder reranker is more accurate than the bi-encoder used for retrieval because it reads the query and each chunk together, but it is also slower — which is why it runs only on the small candidate set, not the whole corpus.

Generation

Mon, 01 Jan 0001 00:00:00 +0000

Generation is the final RAG stage: the kept chunks are assembled into a prompt and sent to the LLM, which produces a grounded answer streamed back to the user. The default LLM is meta-llama/Meta-Llama-3.1-8B-Instruct (the docker-compose default). ibm-granite/granite-4.1-8b is the recommended and validated model; ibm-granite/granite-3.3-2b-instruct is the in-code fallback. All are served by vLLM (OpenAI-compatible API).

The generation behaviour is shaped by three things: the system prompt, the abstention rule, and an optional self-confidence token. Each is an environment variable with a built-in default.

Data Labelling Strategy

Mon, 01 Jan 0001 00:00:00 +0000

This repository implements a data labeling and enrichment pipeline designed to maximize the accuracy and explainability of Retrieval-Augmented Generation (RAG) systems.
Unlike conventional data labeling systems that prepare training sets for supervised learning, this process focuses on labelling document chunks for ingestion into a RAG pipeline, creating both semantic and structural signals that boost retrieval performance.

Why This Approach?

Most RAG pipelines rely solely on vector embeddings for semantic similarity. While powerful, vector-only retrieval often suffers from:

Contextual Retrieval

Mon, 01 Jan 0001 00:00:00 +0000

Standard retrieval embeds each chunk in isolation. A chunk removed from its section can be hard to match: a paragraph about “renewing it” will not retrieve well for a query about “permit renewal” because the chunk alone does not say what “it” is. Contextual Retrieval (an Anthropic-style technique) fixes this by prepending a short, LLM-generated doc-context prefix to each chunk before it is embedded and labelled, so the chunk’s vector carries its surrounding meaning.

Choosing and Configuring Models

Mon, 01 Jan 0001 00:00:00 +0000

Overview

GENIE.AI uses a multi-model architecture where different AI models serve distinct purposes across the entire framework — from document ingestion and knowledge graph construction through to retrieval, reranking, and response generation. Not every model is suitable for every role. Choosing the wrong model for a given purpose will cause silent failures — ingestion batches being skipped, labels returning garbage, or graph data not being created.

This guide explains what each model does, what capabilities it must have, what license it is released under, and provides specific recommendations for both open-source and commercial models across different hardware tiers.