KV REUSE / CORRECTNESS

Run the L4 verification

A deliberate, 24-generation comparison of LMCache and native prefix caching on one NVIDIA L4. These commands run on your Linux GPU machine, not in this website.

1. Prepare the machine

Use a Linux machine or Colab L4 runtime with Python 3.11+, Git, a compatible NVIDIA driver and enough disk space for the runtime and model. Stop other inference engines. In Colab, select an L4 and use a terminal or a persistent shell; these are Bash commands, not Python cells. This GPU run may incur provider charges.

nvidia-smi

Confirm that the GPU is NVIDIA L4. The acceptance fixture is hardware-specific; do not substitute another GPU and interpret it as the same validation.

2. Get the tested verifier

git clone https://github.com/georgxzh/efim.git
cd efim
git checkout 54a0af2ae8db9f8b303be0a9a0683ba8128e7121
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements/colab-vllm-lmcache.lock
python -m pip check
python -m diagnostics.verify_kv_reuse --help

In an already prepared, matching environment, activate that environment and skip reinstalling packages. The lock pins vLLM 0.28.0, LMCache 0.5.4, torch 2.13.0 and transformers 5.16.1. Stop if installation or pip check fails. The CLI runs from the repository root. Do not assume a different branch or a package installed from PyPI contains this tool.

3. Run once

Use a fresh report path. This command starts three engines sequentially and allocates exactly 24 generations across four probes. Do not add another smoke or native run to this acceptance budget.

python -m diagnostics.verify_kv_reuse \
  --model Qwen/Qwen3-0.6B \
  --model-revision c1899de289a04d12100db370d81485cdf75e47ca \
  --kv-connector LMCacheConnectorV1 \
  --lmcache-config diagnostics/fixtures/lmcache-local-cpu.yaml \
  --dtype float16 --gpu-memory-utilization 0.45 \
  --max-model-len 2048 --block-size 16 --max-tokens 16 --seed 0 \
  --suite acceptance --compare-native \
  --fixture diagnostics/fixtures/efim-018-l4.json \
  --report tmp/efim-019-l4/report.json

Immediately inspect the process exit:

echo $?

4. Confirm the expected contrast

Overall exit 1 is expected because LMCache fails correctness. Success of this acceptance test means the verifier distinguishes the two configurations; it does not turn the bad configuration into a pass.

• acceptance.actual_generations == 24
• acceptance.matched == true and acceptance.same_environment == true
• targets.connector.exit_code == 1; four failing probes.
• targets.native.exit_code == 0; four passing probes with observed reuse.

On the original fixture, LMCache cached counts are 0 then 1280 for each reuse; native counts are 0 then 1488. Exit 2, incomplete answers or a different contrast needs investigation, not repeated runs to obtain a preferred result.

5. Preserve the evidence

tar -czf efim-019-l4-evidence.tar.gz -C tmp efim-019-l4

Keep report.json and the complete sibling report.evidence/ directory. Review evidence before sharing it. No upload occurs on this website.

Troubleshooting

• GitHub 404 / authentication: the repository is private; use an authorized account.
• No module named diagnostics: activate the environment and change into the cloned repository root at the pinned revision.
• Report already exists: choose a new --report path. Keep the earlier evidence.
• Wrong prompt length, package drift or configuration error: preserve the error report; do not edit fixture expectations to force a pass.
• GPU memory or startup error: stop other workloads and inspect the engine log. Do not automatically rerun.

Return to documentation