Model Capability Cache

For backend developers and operators. The genie-ai-overlay/core/model_cache.py module probes a remote vLLM endpoint’s /v1/models route, caches the result for MODEL_DETECT_TTL seconds (default 60), and serves the cached value on subsequent calls. On expiry the endpoint is re-probed; on probe failure the last known value is returned. This lets services pick up model changes on the GPU node without a restart.

This page documents the cache contract, the env var, the failure modes, and how to clear the cache for testing.

Prerequisites

• A vLLM endpoint reachable from the OPEA service (default http://<GPU_NODE_HOST>:8000/v1).
• VLLM_API_KEY set (the cache sends Authorization: Bearer $VLLM_API_KEY if non-empty).
• Python 3.9+ for the module’s dict[str, dict] syntax (PEP 585); the rest of the OPEA overlay requires Python 3.11+.
• Read access to genie-ai-overlay/core/model_cache.py.

Why the cache exists

The remote-GPU deployment pattern runs vLLM on a separate node (GPU_NODE_HOST); the OPEA services run on CPU nodes and need to know which model the GPU is currently serving. Without the cache, every chat would probe /v1/models (a ~50–200 ms call), and after every model swap the services would need a restart to pick up the new model ID.

With the cache:

• The first chat after a deploy probes once and caches the result.
• All subsequent chats within the TTL use the cached value (no probe).
• After the TTL expires, the next call re-probes — so a model swap propagates within MODEL_DETECT_TTL seconds, without a restart.
• If a probe fails (GPU down, network blip), the stale cached value is served rather than None — “stale > broken” is the design principle.

Configuration

Set MODEL_DETECT_TTL for each OPEA service that imports core/model_cache.py (chatqna, retriever, etc.). The env template does not currently declare a MODEL_DETECT_TTL entry; it is a Python-side env var consumed by core/model_cache.py (_resolve_ttl, genie-ai-overlay/core/model_cache.py:36-43). To set it, add the line to your deployment’s .env:

# Model capability cache TTL (seconds). Lower for fast model-swap
# propagation; raise for fewer probes on a stable GPU.
MODEL_DETECT_TTL=60

There is no restart needed for the change to take effect — the next cache miss picks up the new TTL.

How it works

The module exports two functions:

from core.model_cache import get_model_id, clear_cache

get_model_id(endpoint_url, ttl_seconds=None) is the main entry point. The logic, line by line (model_cache.py:69-105):

def get_model_id(endpoint_url, ttl_seconds=None):
    if not endpoint_url:
        return None
    now = time.monotonic()
    ttl = _resolve_ttl(ttl_seconds)            # explicit > env > 60s
    entry = _cache.get(endpoint_url)

    if entry and (now - entry["ts"]) < ttl:
        return entry["model_id"]               # cache HIT

    # Cache miss or expired — re-probe.
    model_id = _probe(endpoint_url)
    if model_id is not None:
        _cache[endpoint_url] = {"model_id": model_id, "ts": now}
        return model_id

    # Probe failed. Return stale value if available (stale > broken).
    if entry:
        return entry["model_id"]

    return None                                # never seen before, still failing

The cache is a process-local dict — not shared across the OPEA services or across container restarts. Each service has its own cache; the TTL keeps them roughly in sync.

Probe behaviour

_probe(endpoint_url) (model_cache.py:46) makes one HTTPS call:

resp = httpx.get(f"{endpoint_url}/v1/models", headers=headers, timeout=10, verify=verify)
resp.raise_for_status()
models = resp.json()
if models.get("data"):
    return models["data"][0]["id"]

The first entry in data is always the one cached. vLLM serves the loaded model first; the rest are reserved slots in the model store. The cache is therefore always “the model vLLM is currently using”.

When services call it

The module is consumed by any service that needs to address the LLM by name (rather than letting vLLM pick the default). The most common caller is the prompt-template resolution layer — the prompt the service sends to vLLM is sometimes model-specific, and knowing the current model avoids hard-coding it.

To check which services call get_model_id:

cd genie-ai-overlay
grep -rn "from core.model_cache import\|get_model_id" --include="*.py"

Expected: a handful of call sites in chatqna/, retriever/, and the embedding pipeline. Each call site owns its own cache state — the module is process-local.

Verifying it works

1. Confirm the cache works in isolation

cd genie-ai-overlay
python -c "
from core.model_cache import get_model_id, clear_cache
import time

url = 'https://<GPU_NODE_HOST>:8000'

clear_cache()
t0 = time.time()
m1 = get_model_id(url)
t1 = time.time()
print(f'first call:  {m1!r}  ({1000*(t1-t0):.0f} ms)')

t0 = time.time()
m2 = get_model_id(url)
t1 = time.time()
print(f'cached call: {m2!r}  ({1000*(t1-t0):.0f} ms)')
assert m1 == m2
print('cache HIT path confirmed')
"

Expected: the second call is ~1 ms (no HTTP), the first is ~50–200 ms (real probe). If both are equally slow, the cache is not engaged (see failure modes).

2. Confirm TTL expiry triggers re-probe

MODEL_DETECT_TTL=2 python -c "
from core.model_cache import get_model_id
import time
url = 'https://<GPU_NODE_HOST>:8000'
print(get_model_id(url))   # probe #1
print(get_model_id(url))   # cache HIT
time.sleep(3)
print(get_model_id(url))   # probe #2 (cache expired)
"

3. Confirm stale-on-failure

Stop vLLM, then:

python -c "
from core.model_cache import get_model_id
url = 'https://<GPU_NODE_HOST>:8000'
print(get_model_id(url))      # probe fails — logs warning
print(get_model_id(url))      # stale value served (None if never seen)
"

Expected: the second call returns the cached value (or None if the process has never probed successfully), not an exception.

Failure modes and troubleshooting

Operational playbook

Related

• OPEA Protocol Extensions — the request/response models that travel with each service call.
• Service Registry — the ServiceType enum that names each microservice.
• Choosing Models — which models the vLLM endpoint should serve for embedding, reranking, and chat.
• Deploy → Remote GPU node — the deployment topology where the cache matters most.