Ingestion Log

The ingestion log is the per-chunk progress feed that dataprep writes during document ingestion. Every chunk event — parse, label, embed, context generation, fallbacks — produces one row, addressed by file_id. The log is the operator’s primary tool for answering “why did this document land in failed state?” or “what step was reached before the pipeline broke?”

This page consolidates what the ingestion log is, how to read it (UI and ArangoDB), the canonical messages you will learn to grep for, and how to correlate a log entry with the matching RAG-pipeline trace.

Prerequisites

• A deployed GENIE.AI stack with the document-repository service running (the component that owns the ingestion_log collection and exposes the /api/files/:fileId/ingestion-log endpoints). See Deployment → Docker Swarm if the stack is not yet up.
• An admin Keycloak user (the Admin role reads the log from the admin UI). Service-to-service callers authenticate as the dataprep-service-client OIDC client (mapped to the realm role dataprep-service).
• For ArangoDB-level reads: shell access to a Swarm node and the deployed .env file (<deploy_dir>/.env for Ansible Swarm — default /opt/genieai/.env, the path follows the deploy_dir group var; project-root .env for Docker Compose) so you can read ARANGO_PASSWORD, ARANGO_USERNAME, and ARANGO_DB.
• For trace correlation: the observability stack must be enabled — ENABLE_OBSERVABILITY=1 in .env, otherwise VictoriaTraces carries no spans and step 4 of the correlation recipe cannot work.

Goal

Read the per-chunk ingestion log to answer “what happened to file X?”: identify the failing stage (Labeling, Embedding, Contextualization, …), find the canonical failure message, and pivot to the matching RAG-pipeline trace in VictoriaTraces to confirm the root cause.

Data flow

flowchart LR
    AdminUI["Admin UI<br/>(Document Management tab)"] -->|"GET /api/files/:id/ingestion-log"| DocRepo["document-repository<br/>validates: level INFO/WARN/ERROR"]
    AdminUI -->|"GET /api/files/:id"| DocRepo
    Dataprep["dataprep-arango-service<br/>service.name=genieai-dataprep"] -->|"POST /api/files/:id/ingestion-log<br/>(rate-limited, semaphore-guarded)"| DocRepo
    Dataprep -->|"PATCH /api/files/:id/status"| DocRepo
    Dataprep -. "OTel spans<br/>dataprep.ingest (file_id)<br/>dataprep.llm.label_chunk" .- VictoriaTraces
    DocRepo --> Arango[("ArangoDB<br/>ingestion_log collection<br/>indexed on file_id + timestamp")]
    DocRepo --> Arango2[("ArangoDB<br/>files.dataprep.status")]
    AdminUI -. "step-by-step view" .-> Dataprep

What it is, and what it is not

The two views are correlated: a failed document status almost always has one or more failed-themed ingestion-log rows leading up to it.

Endpoints

The admin Logs tab does not show the ingestion log directly — the per-file ingestion log is read from the Document Management tab → File Details Dialog in the admin UI.

Schema of an entry

Stored in ArangoDB ingestion_log collection:

Canonical messages you will grep for

These are the strings produced by genie-ai-overlay/dataprep/genieai_dataprep_arangodb.py via _write_ingestion_log(...). Learning them saves hours of debugging.

Reading the log via ArangoDB

When the admin UI is unavailable (or you want a one-off script), query ArangoDB directly. Use the base64 remote-script pattern from .claude/rules/DEBUGGING-TRACING.md §3 / §5 to keep quoting sane.

# Save as /tmp/analyze_ingestion_log.py, base64-encode, ssh + run on the swarm node
# Adjust ENV_PATH to match your deployment:
#   /opt/genieai/.env    — Ansible Swarm
#   <project-root>/.env  — Docker Compose (run on the Swarm node where arango lives)
import json, urllib.request, base64
ENV_PATH = "/opt/genieai/.env"
env = {k:v for k,v in (l.strip().split("=",1) for l in open(ENV_PATH)
       if "=" in l and not l.startswith("#"))}
env = {k:v.strip().strip('"').strip("'") for k,v in env.items()}
url = "http://localhost:%s/_db/%s/_api/cursor" % (
    env.get("ARANGO_PORT","8529"), env.get("ARANGO_DB","genie-ai"))
auth = base64.b64encode(("%s:%s" % (env.get("ARANGO_USERNAME","root"),
       env.get("ARANGO_PASSWORD",""))).encode()).decode()
def q(aql):
    req = urllib.request.Request(url, data=json.dumps({"query":aql}).encode(),
        headers={"Authorization":"Basic "+auth, "Content-Type":"application/json"})
    return json.load(urllib.request.urlopen(req)).get("result",[])

FILE_ID = "<file_id>"

# All log rows for the file, newest first
for m in q('FOR d IN ingestion_log FILTER d.file_id=="%s" '
           'SORT d.timestamp DESC RETURN {ts: d.timestamp, msg: d.message}' % FILE_ID):
    print(m)

# Just the failures
for m in q('FOR d IN ingestion_log FILTER d.file_id=="%s" AND d.message LIKE "%%failed%%" '
           'RETURN d.message' % FILE_ID):
    print(m)

AQL gotcha: the field is file_id (snake_case), not fileId. The %% is an escaped % in AQL string literals. The default ARANGO_DB is genie-ai (see docker-compose.yaml); the script falls back to that only if the env var is unset. The default ARANGO_USERNAME is root (the actual env var name is ARANGO_USERNAME, not ARANGO_USER).

Correlating the ingestion log with the RAG-pipeline trace

A failed ingestion usually corresponds to a failed trace in VictoriaTraces. The trace ID is propagated via the W3C traceparent header from the admin UI / backend → dataprep. To find the matching trace:

1. Note the file_id from the admin UI Document Management tab.
2. Open the Trace Explorer dashboard in Grafana (Observability → Dashboards).
3. Filter by service.name=genieai-dataprep and attribute.dataprep.file_id=<file_id>.
4. Open the trace and look for dataprep.llm.label_chunk / dataprep.llm.label_batch spans with red status — those are the per-chunk failures that produced failed / raw-chunk fallback ingestion-log rows.

For the trace-fetch recipe (including the startTime / duration microsecond gotcha), see .claude/rules/DEBUGGING-TRACING.md §1.

When to re-ingest vs when to fix

Verify it worked

After you read or query the ingestion log, confirm the full pipeline is consistent:

# 1. The collection exists and is indexed
docker exec $(docker ps --format '{{.Names}}' | grep arango-vector-db | head -1) \
  arangosh --server.password "$ARANGO_PASSWORD" --javascript.execute-string '
    print(db._collections().filter(c => c._name === "ingestion_log").length);
    print(db.ingestion_log.indexes.map(i => i.id).join(","));
'
# Expected: prints "1" and a list including idx_ingestion_log_file_id, idx_ingestion_log_timestamp.

# 2. A known file returns rows (replace FILE_ID with a real id from the admin UI)
docker exec $(docker ps --format '{{.Names}}' | grep arango-vector-db | head -1) \
  arangosh --server.password "$ARANGO_PASSWORD" --javascript.execute-string '
    print(db.ingestion_log.byExample({file_id: "FILE_ID"}).count());
'
# Expected: > 0 (the file is at least partially ingested).

# 3. The HTTP endpoint round-trips for an admin token
# Mint a realm access token from the admin Keycloak user, then call the endpoint.
# (Keycloak ROPC/direct grant is disabled by default; the admin-cli flow below uses
#  the master admin password already present in /opt/genieai/.env.)
KEYCLOAK_URL="https://$(grep ^NGINX_PUBLIC_DOMAIN= /opt/genieai/.env | cut -d= -f2)"
KC_ADMIN_PWD=$(grep ^KEYCLOAK_ADMIN_PASSWORD= /opt/genieai/.env | cut -d= -f2)
ADMIN_TOKEN=$(curl -sk -X POST "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" \
  -d "client_id=admin-cli" -d "username=admin" -d "password=$KC_ADMIN_PWD" \
  -d "grant_type=password" | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
curl -sk -w "\nHTTP %{http_code}\n" \
  "$KEYCLOAK_URL/api/files/FILE_ID/ingestion-log" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | tail -5
# Expected: HTTP 200 and a JSON array of ingestion_log rows.

# 4. The OTel Collector is reachable from dataprep
# (Reachability check — for an actual service.name probe see step 5.)
docker exec $(docker ps --format '{{.Names}}' | grep dataprep | head -1) \
  sh -c "echo \${OTEL_EXPORTER_OTLP_ENDPOINT:-http://otel-collector:4318}"
# Expected: prints the OTLP endpoint (default http://otel-collector:4318).

# 5. VictoriaTraces actually carries spans with service.name=genieai-dataprep
# (Requires ENABLE_OBSERVABILITY=1 and a recent ingest so a trace exists.)
VT_SERVICE=$(docker ps --format '{{.Names}}' | grep victoriatraces | head -1)
curl -sk "http://${VT_SERVICE}:10428/select/jaeger/api/services" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('genieai-dataprep present:', 'genieai-dataprep' in d.get('data',[]))"
# Expected: prints "genieai-dataprep present: True" once a dataprep ingest has emitted spans.

If any step fails, the troubleshooting table below pinpoints the cause.

Troubleshooting

Troubleshooting by symptom

The table above maps symptoms to likely causes in one row. The playbook below maps the most common symptoms to a numbered recipe — useful when the diagnosis needs a probe, not a one-liner.

Symptom: file stuck in Ingesting for > 1 hour

1. Open the ingestion log; check whether entries are still being written.
2. If yes — the worker is alive but slow. Check dataprep container CPU/memory (docker service ps genieai_dataprep-arango-service for Swarm — in Swarm mode, container names are randomized; the service-style command works, while docker stats requires the exact randomized container name).
3. If no — the worker probably crashed. Restart the dataprep container; the file stays in Ingesting until the worker resumes or you kill it.

Symptom: file ends in Ingestion Error with labelling parse failures

The labelling LLM returned malformed JSON. Almost always a model issue, not a document issue. Steps:

1. Probe the live vLLM with the real LABEL_SELECTOR_SYSTEM_PROMPT and a real chunk from the file (see .claude/rules/DEBUGGING-TRACING.md §7 — use the base64 ssh pattern).
2. If the probe succeeds — check whether the file’s content includes very long sections that may have hit DATAPREP_CONTEXTUAL_MAX_TOKENS (default 512).
3. If the probe fails — the model is overloaded or misconfigured; check vLLM health and the model’s guided-JSON support.

Symptom: many files show batch labelling fallbacks

The primary labelling model is degraded or returning malformed JSON for many chunks. Steps:

1. Check the vllm-llm / vllm container logs. The OTel service.name for queries in VictoriaLogs is genieai-dataprep.
2. Check the LLM endpoint is reachable from dataprep-arango-service:
   • VLLM_LLM_MODEL_ID (default labelling model)
   • DATAPREP_CONTEXTUAL_MODEL if CONTEXTUAL_RETRIEVAL_ENABLED=true The endpoint URL is <vllm-host>:8000 (or <gpu-node>:8000 for remote GPU). Probe liveness: curl http://<vllm>:8000/v1/models.
3. If vLLM is OOM-killed, scale down concurrency (VLLM_MAX_NUM_SEQS) or upgrade GPU memory.
4. If model supports guided JSON but vLLM is overloaded, lower concurrency and re-ingest.

Two parse-failure modes exist: batch-failures cite the chunk indices and trigger automatic per-chunk fallback; per-chunk failures are silent per-chunk downgrades logged individually. The Dataprep label-fallback flow is documented in .claude/rules/DEBUGGING-TRACING.md §3 (ArangoDB ingestion_log queries) and §7 (live vLLM probe).

Symptom: file in Ingested with Warnings

Ingestion succeeded but at least one LLM phase could not complete. The document IS retrievable, but part of it was indexed from a fallback rather than from the model. Open the ingestion log: the System line written at the end of the ingest names each cause and the knob that fixes it.

The status is Ingested with Warnings rather than Ingested because some of the document really did not land. The document is not retracted — the batches that did finish are real work and the chunks are indexed and searchable — but the operator is not told “Ingested” for a document that was only partially processed.

Four causes, with different fixes:

A handful of chunks falling back while the rest succeeded is normal operation and does not set this status — that is what the per-batch fallback is for. The status is reserved for a phase that accomplished nothing, so it keeps meaning something.

After fixing the cause, re-ingest for a clean result. Do it from the Documents table in the admin dashboard: tick the document’s row checkbox, then the batch Ingest action. Re-ingesting a degraded document clears its existing chunks first, so it replaces the previous copy rather than adding a second one to the knowledge base.

Do not use Retract in the file-details dialog for this. Retract is the right tool when you want the document gone, but it also deletes its ingestion log — the record of which cause degraded it, which you need before you can fix it.

Related

• Knowledge base → Document lifecycle — the per-document state machine
• Knowledge base → Ingestion — the ingestion pipeline overview
• RAG → Data labelling strategy — labelling failure modes in detail
• RAG → Contextual Retrieval — what the context-generation step does
• Troubleshooting — ingestion-stuck recipes
• Admin Logs — the VictoriaLogs search for service-level errors