Choosing and Configuring Models

Overview

GENIE.AI uses a multi-model architecture where different AI models serve distinct purposes across the entire framework — from document ingestion and knowledge graph construction through to retrieval, reranking, and response generation. Not every model is suitable for every role. Choosing the wrong model for a given purpose will cause silent failures — ingestion batches being skipped, labels returning garbage, or graph data not being created.

This guide explains what each model does, what capabilities it must have, what license it is released under, and provides specific recommendations for both open-source and commercial models across different hardware tiers.

Model Licensing — What You Need to Know

Every model has a license that governs how you can use it. This is your responsibility to verify before deployment. Licenses can change between model versions. The information below was accurate at the time of writing but you should always check the model card on HuggingFace for the latest terms.

License Categories

Important: If you are deploying GENIE.AI for a government, enterprise, or high-traffic public service, check whether your expected monthly active users exceed 700 million. If so, models under the Gemma and Llama community licenses require you to request a special license from Google or Meta respectively.

The Framework Pipeline: Where Models Are Used

Understanding where each model fits in the GENIE.AI pipeline helps you make informed choices:

DOCUMENT INGESTION PIPELINE:
                                                                                    
  Upload → Docling/OCR → Chunking → Guardrails → Labeling → Graph Extraction → Embedding → ArangoDB
              (GPU)       (config)    (optional)    (LLM)    (LLM + FC?)      (TEI)
                                           ↑          ↑           ↑               ↑
                                      Translation  Translation  Translation    Translation
                                      LLM (shared)  LLM (shared)  LLM (shared)  (separate vLLM)

QUERY PIPELINE:

  User Query → Translation → Embedding → Retrieval → Graph Traversal → Reranking → LLM Inference → Response
                 (vLLM)       (TEI)     (ArangoDB)   (optional)       (TEI)        (vLLM)

The Six Model Roles

GENIE.AI has six distinct AI model roles, served by two vLLM instances and two TEI instances:

Critical architecture point: Roles 1, 2, and 3 all share the same vLLM instance (the main inference server at http://vllm:8000). The model you choose for VLLM_LLM_MODEL_ID must be capable of serving all three of these roles simultaneously. Role 4 (Translation) has its own dedicated vLLM instance. In docker-compose, VLLM_MODEL_ID is derived from VLLM_LLM_MODEL_ID, so setting the latter configures all three shared roles — you only set one.

Role 1: Main Inference (Chat/QnA)

What It Does

This is the “brain” of the chatbot. It receives the user’s query along with retrieved document chunks from the RAG pipeline and generates a natural language answer. It is the most visible model in the system and has the greatest impact on answer quality.

Required Capabilities

How It Is Called

The ChatQnA orchestrator sends retrieved chunks as system/user context and asks the model to answer the user’s question. The CHATQNA_SYSTEM_PROMPT environment variable controls the instruction framing.

Model Recommendations

Open Source

Commercial (API-based)

Note on commercial models: The default GENIE.AI architecture uses local vLLM instances. To use commercial API-based models, you would need to configure VLLM_ENDPOINT to point to an OpenAI-compatible proxy that routes to the commercial API, or modify the ChatQnA orchestrator code.

Configuration

# In your .env file
VLLM_LLM_MODEL_ID=Qwen/Qwen2.5-7B-Instruct
VLLM_GPU_UTILIZATION=0.6
VLLM_MAX_MODEL_LEN=65536
VLLM_DTYPE=auto

Role 2: Chunk Labeling

What It Does

During document ingestion, the dataprep service sends each chunk of text to the LLM along with the full taxonomy (list of labels). The LLM must return a JSON response selecting 1–4 labels from the taxonomy that best match the chunk content.

Required Capabilities

How It Is Called

The dataprep service sends:

• System prompt: LABEL_SELECTOR_SYSTEM_PROMPT (configurable in .env)
• User prompt: "Input: {chunk_text}\nLabels: {taxonomy_list}"
• Sampling: temperature=0.0, response_format={"type": "json_object"} (guided JSON), bounded max_tokens. Chunks may be batched (DATAPREP_LLM_LABEL_BATCH_SIZE, default 4) into one call returning {"0": ["..."], "1": [...]}.

The LLM must return only a JSON object with a "labels" key (or an index→labels map when batched) containing an array of plain strings. response_format enforces valid JSON at the token level; lenient client-side parsing tolerates residual variations (null → [], bare string → [string]).

Contextual Retrieval (optional). On by default (CONTEXTUAL_RETRIEVAL_ENABLED=true); the same vLLM model (override with DATAPREP_CONTEXTUAL_MODEL; empty = reuse this labeling model) is also called once per chunk to generate a ~50-100 word document-context prefix that is prepended before embedding + labeling. It has the same guided-JSON requirement (response_format={"type":"json_object"}, returning {"context": "..."}). On any failure the chunk is stored raw — ingestion never blocks. See the Data Labelling Strategy doc (§7).

What Goes Wrong with Small Models

Models below ~7B parameters frequently fail at labeling:

• Dict instead of string: Returns {"labels": [{"name": "Procurement"}]} instead of {"labels": ["Procurement"]}. The dict is unhashable and crashes downstream set operations.
• Missing JSON structure: Returns conversational text like “Here are the labels: Procurement, Finance” instead of JSON.
• Invented labels: Returns labels that don’t exist in the taxonomy (less critical — the code handles this with warnings).
• Inconsistent format: Works on some chunks, fails on others (non-deterministic).

Model Recommendations

Open Source

Important: Labeling Shares the Main vLLM

Labeling uses VLLM_ENDPOINT + VLLM_MODEL_ID — the same vLLM instance as main inference. There is no separate configuration for a labeling-specific model. If your main inference model is too small for labeling (e.g., Granite 2B), consider using the embedding or bm25 labeling strategies instead (see Step 8 in the Installation Guide).

Alternative Labeling Strategies

If your LLM cannot reliably produce JSON, switch to a non-LLM strategy:

# Embedding-based labeling (recommended for production)
LABELING_STRATEGY=embedding
EMBEDDING_LABEL_THRESHOLD=0.75

# BM25 keyword matching (strict, CPU-only)
LABELING_STRATEGY=bm25
BM25_LABEL_THRESHOLD=2.0

Role 3: Graph Extraction

What It Does

The dataprep service uses LangChain’s LLMGraphTransformer to extract entities (nodes) and relationships (edges) from document chunks. These are stored in ArangoDB as a knowledge graph used for graph traversal during retrieval.

Required Capabilities

How It Is Called

The LLMGraphTransformer sends each document chunk to the LLM and expects a response containing:

• head: entity name (string)
• head_type: entity type (string) — e.g., “Person”, “Organization”
• tail: related entity name (string)
• tail_type: related entity type (string)
• relation: relationship type (string) — e.g., “WORKS_FOR”, “LOCATED_IN”

When function calling is supported, the LLM uses structured tool schemas which guarantee correct types. Without function calling, the LLM must return this as raw JSON — and small models routinely return dicts instead of strings for the type fields, causing unhashable type: 'dict' crashes.

Model Recommendations

Open Source

Why Function Calling Matters

When ignore_tool_usage=False (function calling supported), LangChain uses the model’s native structured output API. This guarantees that head_type, tail_type, and relation are strings.

When ignore_tool_usage=True (no function calling), LangChain parses raw JSON from the model’s text output. Small models return malformed types — dicts, arrays, or nested objects — which crash when added to Python sets.

The OPEA code sets ignore_tool_usage=True when using vLLM (line 173 of arangodb.py), because vLLM’s OpenAI-compatible API does not advertise function calling support in a way LangChain detects. This means graph extraction relies entirely on the model’s ability to return correct raw JSON.

Important: Graph Extraction Shares the Main vLLM

Graph extraction uses the same VLLM_ENDPOINT + VLLM_MODEL_ID as main inference and labeling. There is no separate configuration. The model you choose for VLLM_LLM_MODEL_ID must handle all three roles.

If Your Model Cannot Do Graph Extraction

If your main inference model lacks function calling or produces unreliable JSON, the graph extraction batches will be silently skipped (logged as WARN). Documents will still be ingested with chunks and embeddings, but without knowledge graph relationships. This means:

• Hybrid retrieval will still work (vector + label filtering)
• Graph traversal in the retriever will return no additional context
• Related Documents feature will not function

Role 4: Translation & Guardrails

What It Does

This dedicated vLLM instance handles two tasks:

1. Translation: Translates user queries, chat history, and system prompts between languages for the multi-language RAG pipeline. Uses structured chat prompts with language codes.
2. Guardrails: The OPEA guardrails service sends content through this model for safety classification. (Currently disabled by default: GUARDRAIL_ENABLED=false)

Required Capabilities

How It Is Called

The ChatQnA orchestrator sends translation requests in a structured format:

Translate the following text from {source_lang} to {target_lang}:
{text}

For TranslateGemma fine-tuned models, a specific chat template with language codes is used.

Model Recommendations

Open Source — Dedicated Translation Models (Recommended)

Open Source — General Instruction Models

Commercial

Configuration

# In your .env file
VLLM_TRANSLATION_MODEL_ID=Infomaniak-AI/vllm-translategemma-4b-it
VLLM_TRANSLATION_GPU_UTILIZATION=0.3
VLLM_TRANSLATION_MAX_MODEL_LEN=2048
VLLM_TRANSLATION_DTYPE=auto

Language Support

The translation system supports these language codes (defined in the backend’s language maps): ar, bn, de, en, es, fr, id, man/mnk, pt, ru, st, sw, th, zh

For low-resource languages (e.g., Mandinka), the system uses Dyula as a linguistic proxy from the NLLB-200 model.

Role 5: Embeddings

What It Does

Converts document chunks and user queries into mathematical vectors (embeddings). These vectors are stored in ArangoDB and used for semantic similarity search during retrieval. The embedding model is critical for RAG accuracy — if the embeddings don’t capture meaning well, the retriever will return irrelevant results.

Required Capabilities

Model Recommendations

T4 Warning: Do not use BAAI/bge-reranker-v2-m3 on a Tesla T4. It uses an XLM-RoBERTa architecture that may have compatibility issues with T4-optimized TEI images and its memory footprint is too large for a shared 16GB card.

Configuration

# In your .env file
EMBEDDING_MODEL_ID=BAAI/bge-base-en-v1.5
TEI_EMBED_MODEL=BAAI/bge-base-en-v1.5

Role 6: Reranking

What It Does

After the retriever fetches candidate chunks, the reranker performs a more computationally expensive “cross-encoder” analysis to re-score each chunk against the actual user query. This is the final quality filter before chunks are sent to the LLM.

Required Capabilities

Model Recommendations

Configuration

# In your .env file (default; multilingual, requires >=24 GB VRAM)
RERANKER_MODEL_ID=BAAI/bge-reranker-v2-m3
# T4 / constrained GPUs (English only) — override with the lighter model:
# RERANKER_MODEL_ID=cross-encoder/ms-marco-MiniLM-L-6-v2

Hardware Profiles and Model Combinations

Understanding GPU Memory Allocation

GENIE.AI runs multiple AI services on the same GPU. Each service claims a portion of VRAM, and the total must fit within your GPU’s capacity. Here is what consumes GPU memory:

Key parameter: VLLM_GPU_UTILIZATION

This controls what fraction of GPU memory vLLM reserves for the model weights and KV cache. It does not control the total memory used — vLLM will use the full allocation for model weights plus KV cache for concurrent requests.

Key parameter: VLLM_MAX_MODEL_LEN

This controls the maximum context window (in tokens). Higher values consume more VRAM for the KV cache, reducing the number of concurrent requests the model can handle.

Key parameter: VLLM_DTYPE

The weight precision directly affects model quality and VRAM usage:

T4 Warning: The Tesla T4 does not support bfloat16. Always use VLLM_DTYPE=half on T4.

The GPU Memory Budgeting Process

Before choosing models, calculate your GPU memory budget:

1. Start with total VRAM (e.g., 16 GB for T4, 48 GB for A100)
2. Reserve 1 GB for OS/CUDA overhead
3. Reserve ~500 MB for TEI embedding
4. Reserve ~200 MB for TEI reranker
5. Divide remaining memory between main vLLM and translation vLLM

Example for T4 (16 GB):

Total: 16 GB
- OS/CUDA:    1 GB
- TEI embed:  0.5 GB
- TEI rerank: 0.2 GB
- Remaining: 14.3 GB
  - Main vLLM (40%):    5.7 GB  → fits Granite 2B or Gemma 4B
  - Translation (60%):  8.6 GB  → fits Gemma 4B

Example for A100 (48 GB):

Total: 48 GB
- OS/CUDA:    1 GB
- TEI embed:  1 GB (bge-m3)
- TEI rerank: 0.2 GB
- Remaining: 45.8 GB
  - Main vLLM (60%):   27.5 GB → fits Qwen 2.5 14B
  - Translation (40%): 18.3 GB → fits Qwen 2.5 7B

Loading Models and Verifying GPU Allocation

After configuring your .env, bring up the services and monitor GPU usage:

# Step 1: Start services
docker compose up -d --build

# Step 2: Watch GPU allocation in real-time
watch nvidia-smi

What to look for in nvidia-smi:

Expected timeline:

• TEI models (embedding, reranker): Load in 30–60 seconds
• Translation vLLM: 2–5 minutes (model download + GPU loading)
• Main vLLM: 3–10 minutes (model download + GPU loading)

vLLM healthcheck timeout is normal. The healthcheck often fails during initial startup because the model is still loading. Wait for the model to fully load, then run docker compose up -d again to restart any services that timed out.

# Check which services are healthy
docker compose ps

# If vllm is unhealthy, wait and retry
docker compose up -d

Testing Model Loading

After all services are up, run these verification steps:

1. Verify main inference is working:

# Check vLLM is serving the model
curl http://localhost:8000/v1/models

2. Verify embedding service:

# 7000 is the internal retriever port (dev/Compose only). In Swarm, run inside the net:
# docker exec <retriever-container> curl http://tei:80/embed -X POST ... \
curl http://localhost:7000/embed -X POST \
  -H "Content-Type: application/json" \
  -d '{"inputs": "test query"}'

3. Verify translation service:

curl http://localhost:9031/v1/models

4. Test end-to-end RAG: Log into the application and ask a question about an ingested document. If the response is coherent and references the source material, the pipeline is working correctly.

Profile A: Tesla T4 (16GB VRAM) — Entry Level

This is the most constrained profile. Multiple models must share a single 16GB GPU. The T4 uses Turing architecture (compute 7.5) which does not support bfloat16.

Hardware specs: 16 GB GDDR6, compute 7.5, CUDA 12.4 driver typical

Main vLLM (serves Inference + Labeling + Graph):

Recommendation: Use Granite 3.3 2B for inference and switch labeling to embedding or bm25 strategy. Accept that graph extraction will not work reliably.

Translation vLLM:

Total GPU allocation (T4): Main (~4-8 GB) + Translation (~8 GB) + TEI Embedding (~500 MB) + TEI Reranker (~200 MB) = ~13-17 GB. This is extremely tight on 16 GB.

# T4 Configuration
VLLM_LLM_MODEL_ID=ibm-granite/granite-3.3-2b-instruct
VLLM_GPU_UTILIZATION=0.4
VLLM_MAX_MODEL_LEN=65536
VLLM_DTYPE=half
LABELING_STRATEGY=embedding    # Do NOT use llm with 2B model

VLLM_TRANSLATION_MODEL_ID=google/gemma-3-4b-it
VLLM_TRANSLATION_GPU_UTILIZATION=0.4
VLLM_TRANSLATION_DTYPE=half
VLLM_TRANSLATION_KV_CACHE_DTYPE=fp8

EMBEDDING_MODEL_ID=BAAI/bge-base-en-v1.5
RERANKER_MODEL_ID=cross-encoder/ms-marco-MiniLM-L-6-v2

Profile B: RTX 4060 Laptop (8GB VRAM) — Development Only

Not recommended for production. Even more constrained than T4. The RTX 4060 has Ada Lovelace architecture (compute 8.9) which supports bfloat16 and fp8.

Hardware specs: 8 GB GDDR6, compute 8.9, CUDA 12.8 driver typical

Recommendation: Use Gemma 3 4B for inference with LABELING_STRATEGY=embedding. Accept that graph extraction will not work. Translation may need to use the CPU-based NLLB fallback or share the main vLLM (reducing available VRAM further).

VLLM_LLM_MODEL_ID=google/gemma-3-4b-it
VLLM_GPU_UTILIZATION=0.30
VLLM_MAX_MODEL_LEN=65536
VLLM_DTYPE=half
LABELING_STRATEGY=embedding

VLLM_TRANSLATION_MODEL_ID=google/gemma-3-4b-it
VLLM_TRANSLATION_GPU_UTILIZATION=0.35
VLLM_TRANSLATION_MAX_MODEL_LEN=2048
VLLM_TRANSLATION_DTYPE=half

EMBEDDING_MODEL_ID=BAAI/bge-base-en-v1.5
RERANKER_MODEL_ID=cross-encoder/ms-marco-MiniLM-L-6-v2

Profile C: RTX 6000 Ada / A40 / L40S / A100 (48GB VRAM) — Enterprise

This profile has room for high-quality models across all roles. These GPUs support bfloat16.

Hardware specs: 48 GB GDDR6 (RTX 6000 Ada) / 48 GB HBM2e (A100), compute 8.9 (Ada) / 8.0 (Ampere)

Main vLLM (serves Inference + Labeling + Graph):

Recommendation: Use Qwen 2.5 7B or Llama 3.1 8B as the best balance of quality, capability, and VRAM. Both support function calling and reliably produce structured JSON for labeling and graph extraction. For maximum quality, use Llama 3.1 70B-Instruct-AWQ if the GPU supports 48GB+.

Translation vLLM:

# Enterprise Configuration (Recommended)
VLLM_LLM_MODEL_ID=Qwen/Qwen2.5-7B-Instruct
VLLM_GPU_UTILIZATION=0.6
VLLM_MAX_MODEL_LEN=65536
VLLM_DTYPE=auto
LABELING_STRATEGY=llm    # LLM strategy works well with 7B+ models

VLLM_TRANSLATION_MODEL_ID=Infomaniak-AI/vllm-translategemma-4b-it
VLLM_TRANSLATION_GPU_UTILIZATION=0.3
VLLM_TRANSLATION_MAX_MODEL_LEN=2048
VLLM_TRANSLATION_DTYPE=auto

EMBEDDING_MODEL_ID=BAAI/bge-m3          # Multilingual upgrade
RERANKER_MODEL_ID=BAAI/bge-reranker-v2-m3   # Multilingual reranker (enterprise GPU)

Profile D: Multi-GPU / Separate Nodes

If you have multiple GPUs or separate servers, you can dedicate resources more effectively:

Option 1: Two GPUs, single node (e.g., 2x A100 48GB)

# GPU 1: Main inference + labeling + graph extraction
VLLM_LLM_MODEL_ID=meta-llama/Meta-Llama-3.1-70B-Instruct-AWQ
VLLM_GPU_UTILIZATION=0.9
NVIDIA_VISIBLE_DEVICES=0

# GPU 2: Translation + embeddings + reranking
VLLM_TRANSLATION_MODEL_ID=Infomaniak-AI/vllm-translategemma-4b-it
VLLM_TRANSLATION_GPU_UTILIZATION=0.5
NVIDIA_VISIBLE_DEVICES=1

Option 2: Separate servers (recommended for production)

Other Notable Hardware

Decision Matrix: Which Model Should I Choose?

For Main Inference + Labeling + Graph Extraction (shared vLLM)

Do you have 48GB+ VRAM?
├── YES → Use Qwen 2.5 7B or Llama 3.1 8B
│         (or Llama 3.1 70B-AWQ for maximum quality)
└── NO
    ├── Do you have 24GB+ VRAM?
    │   ├── YES → Use Qwen 2.5 7B or Llama 3.1 8B (INT8/AWQ)
    │   └── NO
    │       ├── Do you have 16GB VRAM?
    │       │   ├── YES → Use Mistral 7B with LABELING_STRATEGY=embedding
    │       │   │         (graph extraction will be unreliable)
    │       │   └── NO
    │       │       └── Use Gemma 3 4B with LABELING_STRATEGY=embedding
    │       │           (graph extraction will NOT work)

For Translation (dedicated vLLM)

Do you need high-quality translation for many languages?
├── YES → Use Infomaniak-AI/vllm-translategemma-4b-it
└── NO
    └── Use Gemma 3 4B (default, adequate for most use cases)

Key Environment Variables Reference

Main Inference (Roles 1, 2, 3)

Translation (Role 4)

Labeling Configuration

Embeddings & Reranking (Roles 5, 6)

Document Extraction (Non-LLM AI)

OCR Models (Pre-downloaded, not configured)

These models are downloaded at build time (see the Dockerfile) and stored inside the container. They do not require GPU allocation — EasyOCR can run on CPU if needed, but GPU (DOCLING_DEVICE=cuda) is much faster for PDF processing.

Non-AI Components Worth Understanding

While this guide focuses on LLM and model selection, the following non-AI components are critical to the framework’s performance and should be understood:

Common Pitfalls

1. Using a 1B-2B model for everything

Small models (Gemma 1B, Granite 2B) cannot reliably produce structured JSON. Using them for labeling or graph extraction will cause silent failures — batches skipped, garbage labels, empty graphs.

2. Assuming all models work for graph extraction

Graph extraction requires either function calling support or extremely reliable JSON formatting. Most models below 7B fail at this. If your model lacks function calling, graph batches will be skipped with unhashable type: 'dict' warnings.

3. Not accounting for shared VRAM

The main vLLM, translation vLLM, embedding TEI, and reranking TEI all run on the same GPU (in single-node deployments). A 7B model at FP16 uses ~14 GB. On a 16 GB T4, that leaves almost nothing for translation and embeddings.

4. Ignoring the labeling strategy option

If your LLM cannot do labeling reliably, switch to LABELING_STRATEGY=embedding. This uses the embedding model (which is much more reliable) for semantic label matching instead of the LLM.

5. Using the wrong dtype for your GPU

• T4 (Turing): Does not support bfloat16. Use half (float16).
• Ampere+ (A100, RTX 3090+, RTX 6000 Ada): Supports bfloat16. Use auto or bfloat16.
• RTX 4060 (Ada Lovelace): Supports bfloat16. Use auto.

6. PyTorch CUDA version mismatch

When building the dataprep Docker image, pip install torch without an explicit index URL pulls the latest PyTorch (currently CUDA 13.0). If your host driver is CUDA 12.8, this makes CUDA unavailable inside the container. To fix this, add a TORCH_CUDA_VERSION build ARG to the Dockerfile and pass the correct value from docker-compose. See the Dockerfile comments for details.