Knowledge Base on GENIE.AI

Ingestion

Mon, 01 Jan 0001 00:00:00 +0000

Ingestion turns a raw uploaded file into searchable, labelled, embedded chunks in ArangoDB. It runs automatically once a file is uploaded; no manual trigger is needed. The pipeline is designed so that a failure at any stage never loses the file — the upload is stored first, then processed.

The pipeline

1. Upload — file received by the document-repository service
2. Antivirus scan — ClamAV scans the file; infected files are rejected
3. Store — file saved to disk; a file_id is assigned
4. Parse — docling extracts text + structure (headings, tables)
5. Chunk — text split into retrieval-sized chunks
6. Label — each chunk labelled against the service taxonomy
7. Context prefix — (if enabled) an LLM doc-context prefix is prepended
8. Embed — each chunk vectorised with the embedding model
9. Store — chunks, embeddings, labels, and graph nodes written to ArangoDB

Steps 4–9 are owned by the dataprep service (genie-ai-overlay/dataprep/). The whole sequence is observable end-to-end — see Observability for how to watch a specific file’s ingestion in the traces and the ingestion log.

Document Lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

Every document in the knowledge base has a lifecycle: it is uploaded, processed, made retrievable, and eventually updated or removed. Each document is tracked by a stable file_id, so its status, chunks, and graph footprint are all addressable.

States

A document moves through these states (recorded as dataprep.status):

State	Meaning
uploaded	File received and virus-scanned; stored on disk. Not yet searchable.
processing	Dataprep is parsing, chunking, labelling, and embedding.
completed	All chunks indexed; the document is retrievable.
failed	Processing failed; the document is not searchable. Check the ingestion log for the reason.

Only documents in the completed state contribute to answers. A document stuck in processing or in failed is invisible to retrieval.

Labelling & Taxonomy

Mon, 01 Jan 0001 00:00:00 +0000

Labelling is what keeps a multi-domain knowledge base from bleeding across topics. Every chunk is assigned one or more labels drawn from the service-category taxonomy, and at query time the retriever uses those labels to return only on-topic chunks. Without labels, a question about one service could pull in unrelated chunks from another.

The taxonomy

The taxonomy is the service-category hierarchy — the set of categories and services the deployment answers about. It is managed in the backend (serviceCategories, services) and is the same structure the application uses to organise its service catalogue.

Content Guidance

Mon, 01 Jan 0001 00:00:00 +0000

The single biggest factor in GENIE.AI answer quality is the quality of the documents in the knowledge base. No amount of retrieval tuning fixes documents that are poorly structured, ambiguous, or full of content the parser cannot read. This page is practical guidance for the people choosing and preparing source documents.

What retrieves well

Clean, text-based documents — real text the parser can extract, not scanned images.
Clear structure — headings, sections, lists. Structure helps both chunking (cleaner chunk boundaries) and the knowledge graph.
Self-contained sections — a chunk that makes sense on its own retrieves far better than one full of “see above” or dangling pronouns.
One topic per section — focused sections produce precise labels.
Explicit language — the words a user would actually type. A document that calls it “the Programme” everywhere will not retrieve for “permit renewal”.

What retrieves poorly

Scanned / image-only PDFs — there is little or no extractable text; the chunk is near-empty. OCR these into real text first.
Table-heavy documents — most chunks become padded table rows with low information density and poor labels. This is a known, expected pattern, not a labelling bug.
Documents that are all boilerplate — headers/footers repeated on every page pollute chunks.
Enormous, undifferentiated documents — without structure, chunks are arbitrary windows of text.

Chunking

The dataprep splits parsed text into retrieval-sized chunks. You do not control chunk boundaries directly, but you influence their quality through document structure: well-headed documents chunk cleanly; wall-of-text documents chunk arbitrarily. If a specific document retrieves badly, the first fix is almost always the document’s structure, not a pipeline knob.