Troubleshooting

Run the doctor first: a bare docker run --rm --gpus all <image> diagnoses most host problems.

Reading the engine log

When the engine fails, docker logs shows only the runtime's summary; the root cause is in the engine log at /var/log/carbonforge/vllm.log inside the container. While the container is running:

docker exec carbonforge cat /var/log/carbonforge/vllm.log

After it has exited, read it straight from the logs volume (the carbonforge-logs volume from the quick start's launch command):

docker run --rm -v carbonforge-logs:/logs alpine cat /logs/vllm.log

Every entry below that mentions vllm.log means this file.

could not select device driver "" with capabilities: [[gpu]]

Symptom: any docker run --gpus all fails immediately with this error.

Docker cannot see the NVIDIA runtime. Either the NVIDIA Container Toolkit is not installed, or it was installed but never wired into Docker: run sudo nvidia-ctk runtime configure --runtime=docker, restart Docker, and retry. Verify with docker run --rm --gpus all ubuntu nvidia-smi.

The container exits instead of serving

Symptom: container starts, prints a report, exits 0.

You launched without CF_CONTEXT_KEY (or arguments), which runs the doctor by design. Set a context key.

"no license key provided" despite the mount

Symptom: the license file is mounted but the container refuses premium recipes.

The file is unreadable inside the container. Check mode and ownership on the host (chmod 600, readable by the container user), or pass CARBONFORGE_LICENSE_KEY as an environment variable instead.

Refused as unlicensed with an explicit model override

Symptom: could not determine which model to gate or an unlicensed refusal when overriding the entrypoint.

The model must be the first argument after -- (or passed as --model). The entitlement gate reads that position.

Crash at engine warm-up: Could not find nvcc

Symptom: dies at compile_or_warm_up_model with cuda_home='/usr/local/cuda' doesn't exist.

The engine JIT-compiles a sampling kernel and needs a CUDA toolkit. The container ships one, so this only affects native (non-container) installs: either install the CUDA toolkit on the host, or set VLLM_USE_FLASHINFER_SAMPLER=0.

Engine dies at start-up with a GPU out-of-memory

Symptom: the runtime logs vLLM exited (code 1) before becoming ready; vllm.log shows memory allocation failed with OOM on device 0 during "Capturing CUDA graphs" (or earlier, while loading weights).

The GPU does not have enough free memory for the recipe. If the card is smaller than the recipe's tuned hardware (the last segment of the context key), this failure is expected, and can be intermittent when the margin is a few MiB: a run that once succeeded can fail the next time. Shrink the engine's footprint with -e CF_GPU_MEMORY_UTILIZATION=0.6, or add -e CF_VLLM_EXTRA_ARGS="--enforce-eager" to skip CUDA graph capture at some throughput cost; both overrides depart from the benchmarked recipe. Also check nothing else is holding GPU memory (nvidia-smi). Note that docker start reuses the container's original configuration; to change an environment variable you must docker rm -f the container and run it again.

Model not found despite prefetched weights

Symptom: offline host, weights prefetched, engine reports the model missing.

The engine looks in the hub/ subdirectory of the cache. Set HUGGINGFACE_HUB_CACHE to the directory you prefetched into, and keep it consistent with the volume mount.

/v1/models refuses connections after launch

Symptom: container is up and healthy but the port does not answer.

Normal during first boot: image plus weights take 10 to 20 minutes on a fresh machine. Follow docker logs -f carbonforge and wait for the engine ready line.

Gateway integration returns 404 for the model

Symptom: direct curl works; the gateway in front returns model not found.

The served model name must match the gateway's model list character for character. Set --served-model-name explicitly rather than relying on the default.

Tail latency collapses under load

Symptom: p50 fine, p99 grows without bound as concurrency rises.

The engine's --max-num-seqs is a hard ceiling; excess requests queue. Bound client concurrency, and disable gateway retries when measuring.

Tracing enabled but no traces appear

Symptom: --request-trace standard set, sinks empty.

Two causes: no sink configured (a level alone logs a warning and disables tracing), or the OpenTelemetry SDK is missing from the serving venv on a native install (spans are silently dropped; the doctor reports it as a WARN).