OKF Configuration Variables

These variables control the OKF subsystem: creating knowledge repositories from website crawls, and the indexing pipeline that makes repository content retrievable. They are set in the deployment environment file (env template → .env, or the Ansible vault/group vars on Swarm deployments) and consumed by the okf-server and dataprep services.

All defaults are safe for small deployments — every feature works out of the box with no variables set. Tune them when your sources get large or your machine has spare capacity.

The OKF master switch (OKF_ENABLED)

With the flag parked (0):

• The three OKF containers — okf-server, pii-service, ragas-service — run at zero replicas (nothing OKF executes anywhere).
• The /api/okf gateway route is not created — the API surface does not exist from outside.
• The Admin Dashboard hides the OKF Studio and Models tabs. The Query Inspector tab is not gated and stays visible.
• The chat backend answers with the legacy retrieval posture — no OKF connection is attempted, so a parked deployment never burns timeouts against a service that is not running.

With the flag on (1) everything described on this page and in the OKF Studio docs works as documented.

Creating repositories from crawls

A crawled website (a single large Markdown file produced by the crawler) is converted into an OKF repository server-side: the okf-server streams the file, splits it into one concept per page, and ingests the batch — the browser is never asked to download or process the file. Conversion runs asynchronously: the create request returns immediately and progress is shown per repository on the file’s Dashboard tab.

Notes:

• Files up to the cap stream through the server a page at a time — the whole source is never held in memory, so the 10 GB default is a policy limit, not a memory limit. Raise it only if your storage and crawler genuinely produce larger files.
• The same crawl file can be converted multiple times; each repository gets a unique name (an automatic -2, -3 … suffix) so conversions stay traceable.

Parallel indexing

After a repository’s concepts are created, each one is indexed (chunked, labelled, embedded) by the dataprep service. Historically this ran strictly one concept at a time — one global pipeline — so large crawls took hours to index. Repositories are independent units of work and now index in parallel, capped by two variables that should be raised together:

Sizing guidance:

• The heavy work inside dataprep is calling the LLM (labelling) and the embedding service — both are HTTP calls to services that handle concurrency natively, so throughput scales close to linearly with the slot count until document chunking (CPU) becomes the bottleneck.
• Set both halves to the same value. Higher OKF_INGEST_CONCURRENCY than DATAPREP_INGEST_CONCURRENCY just produces 429 backoffs; the reverse under-uses the worker. Start at half your cores (12/12 on a 24-vCPU workstation) and raise it while the model services have headroom — this pair is limited by the GPU/model services, not by the host, so more slots stop helping once the models saturate.
• Indexing progress per repository is visible on the repository card and in the editor; the pending count is a shared queue across repositories — with lanes draining first-in-first-out across all repos, one repository’s pending count can sit still while another’s clears.

Why the two pairs are limited by different things

The four settings form two pairs, and the pairs are bounded by different resources. This is the part that matters when sizing:

• Import is bounded by the host’s own CPU and RAM. The dominant cost is the PII sidecar’s language-model scan, which runs on the machine and scales with cores.
• Indexing is bounded by the model services, not by the host. Each concept calls the language model for labelling and the embedding service — both shared, and often backed by a single GPU. Adding slots there does not use more of the machine; it queues on the model, and on a shared GPU it takes capacity away from chat answering.

So a higher index default would not be “faster by default” — it would be a default that slows down the assistant. Raise that pair deliberately, against the model services’ headroom, not against the core count.

Sizing all four settings for your host

The shipped defaults are conservative and safe, not a recommendation for a machine of your size. On a 24-core host the import pair of 4/4 uses roughly a fifth of the available processing power — set it deliberately.

All four are configuration, not hard-coded behaviour. Set them in your .env (the Docker Compose deployment) or in the Ansible group_vars; nothing needs to be rebuilt. Changing them requires recreating the affected containers, because the values are read once at process start:

docker compose up -d --force-recreate okf-server pii-service dataprep-arango-service

The import pair’s ceiling is 16. PII_WEB_CONCURRENCY is a memory setting before it is a CPU setting: each worker process loads its own copy of the detection model, measured at roughly 640 MB per worker on the current image. Multiply by the value you choose and leave headroom for everything else on the host — at 12 that is about 7.7 GB. A host that cannot afford that should lower the import pair, not raise it into swap.

The index pair has no hard ceiling but a real one: it is whatever the model services will absorb. If a labelling or embedding call starts failing, or chat answers get slower during a large drain, this pair is set too high.

Confirm what a host actually got

Do not infer the setting from timing. Every import writes the capacity it really started with to the okf-server log, under OKF import lane capacity:

OKF import lane capacity  repo_id=… concepts=… import_concurrency=12
  pii_web_concurrency=12 paired=true cores=24

paired is a configuration check, not a contention gauge. It is true when this import’s lanes do not exceed the sidecar’s worker count; false means the extra lanes are queueing at the PII sidecar, which costs latency without adding throughput. If pii_web_concurrency reads unset, the two settings were not applied to the same deployment.

It cannot tell you whether other imports are running. Nothing stops a second import while one is in flight, and each import takes its own lanes — so two concurrent imports demand twice the scan capacity and both still report paired: true. If you run imports back to back, set the pair for the number you might run at once, not the number you are running now.

Import capacity (creating a repository)

Importing a repository — from a bundle, from documents, or from a crawl — is a different pipeline from indexing one, and has its own pair of settings. They follow the same rule as the drain pair above: set both halves to the same value, because lanes above the worker’s capacity queue instead of adding throughput.

Size both from the cores and RAM of the host you are deploying to, not from the default. min(16, cores / 2) is the rule of thumb — 12/12 on a 24-core host, 8/8 on 16 cores, 4/4 on 8. Check the memory budget first (see Sizing all four settings), then set both halves to the same value: import lanes above the sidecar’s worker count queue instead of adding throughput, so an unpaired setting costs latency and buys nothing.

Every import writes the capacity it actually started with to the service log under OKF import lane capacity, including whether the two settings match — so an under-provisioned host can be seen in the logs rather than inferred from how long the import took.

Import caps

OKF_INGEST_MAX_CONCEPTS is applied after oversized documents are split, so the number of documents you selected is not the number the cap sees. A single 999-document crawl was measured expanding to 1112 concepts across 89 split families — an import that looks comfortably within any reasonable limit. If a large import is rejected, raise the variable rather than splitting the work up; the limits that genuinely bound the work are the maximum concept length and the lane settings above.

The per-concept document ceiling (OKF_MAX_CONCEPT_DOC_CHARS)

The ceiling is enforced on every path that can write a concept body, so no oversized document can reach the corpus whichever way it arrives:

The ceiling governs the body, not the frontmatter: a concept with a large frontmatter and a small body passes, and the index concept is not exempt.

What to do when a document exceeds it. Split it (the import path already does this losslessly), or raise the ceiling. Raise it only deliberately — it is a static literal, while the Contextual Retrieval document budget it is meant to track is derived at runtime from the window your model endpoint serves. On the default 65 536-token window the derived per-chunk budget measures ≈ 219 700 characters, which is below this 250 000 default: a concept at the ceiling therefore has its document context truncated on the default deployment (the truncation is recorded in the ingestion log). On a 4 096- or 2 048-token window there is no room for a document context at all, and contextualization is skipped for every document.

The ceiling is the bound that actually protects the machine when OKF_INGEST_MAX_CONCEPTS is raised — keep it at the default unless you have measured that your model window tolerates larger concepts.

Suggested values:

Stuck-drain protection (the reaper)

A concept whose indexing never completes would block its repository from ever publishing (the publish gate waits for all pending concepts). The worker’s reaper dead-letters such concepts to a visible failed state so the repository can be recovered by re-ingesting. Two properties make it safe for deep crawl backlogs:

Waiting concepts — the normal case in a large backlog — are never reaped, no matter how old they are or how deep the queue. Only claim state (a stale claim or too many attempts) triggers a dead-letter.

Drain worker internals (rarely touched)

The worker’s cadence and job windows. Defaults are safe everywhere; these are listed for completeness because they are real, wired variables.

Lifecycle, bundles and studio drafts

The LLM curation engine

The curation calls (labels, frontmatter drafting, topic-link authoring) run inside okf-server against the deployment’s model endpoint (VLLM_ENDPOINT, VLLM_LLM_MODEL_ID, VLLM_API_KEY — unset endpoint = the engine fail-softs to heuristics).

A closed opt-in is reported, not hidden

With VLLM_ENDPOINT unset, the legs that follow the deployment opt-in are OFF: topic links, CSV topics and the manifest summary. A named model configuration (the Admin Dashboard’s Models tab) does not enable them — it serves the services its own checkboxes name — so the tab warns when a configuration is active while the opt-in is closed, and VLLM_ENDPOINT is what turns those legs on.

An import whose CSV topics leg degraded is recorded rather than silent: the conversion report’s per-file entry carries the topic source and a warning, and the source document’s ingestion log mirrors the reason (see Ingestion log). The file still imports — the concepts carry the filename slug as their topic.

Whole-corpus topic-link authoring

The import pass (and the editor’s Step 7 Cross-link topics) authors the cross-links a corpus that never cites itself is born without. It is generic — candidate pairs come from token overlap plus an assistant pass, no domain assumption — and re-running is safe (existing links are kept, only new ones are added).

Frontmatter suggestion internals

Access control mode (OKF_SCOPE_MODE)

Controls which repositories a user’s chat queries can reach. Full capability reference: Access control.

In both modes: a granted scope still restricts its holder to exactly the scoped repositories (scopes restrict — their absence no longer disables), and the write side (repo CRUD, ingest, lifecycle, governance) always requires an admin scope or the super-role. Flipping the value needs an okf-server container recreate, not a rebuild.

The grant facility behind per-repo restriction has its own credentials:

RAGAS benchmarking (the Lab’s Benchmarks tab)

Backs the RAGAS evaluation feature (see OKF Studio). No feature is indexed on a specific model — the judge/embedder variables point at the deployment’s chosen OpenAI-compatible endpoints and default to the deployment’s LLM configuration; per-task model minimums (strict JSON output, window sizes, no reasoning-token leak into judge completions) are part of the feature’s spec.

RAGAS sidecar tuning (rarely touched)

Defaults below were set from live measurements on a shared GPU; change them only when you have measured the same symptom.

The Testing Lab’s model calls

The PII sidecar

Retrieval mode governance

The carrier semantics and fail-closed behavior on a config outage are in Architecture → OPEA microservices.

dataprep OKF labeling retries

How dataprep’s OKF labeling rides out model-server storms (shared GPU, vLLM restarts):

Related variables

These are set once per deployment and rarely touched; they are listed in the environment template with the rest of the secrets:

• OKF_INTERNAL_SECRET — shared secret authenticating dataprep’s completion callbacks to okf-server (fail-closed: must be set on both sides).
• KC_OKF_SERVER_CLIENT_SECRET — okf-server’s service-account secret for service-to-service calls.
• OKF_INGEST_WORKER_ENABLED — the drain worker’s on/off switch.