Quick start

This guide takes you from nothing to a serving OpenAI-compatible endpoint on your own GPU. Budget about 20 minutes of wall-clock for image and weight downloads on a fresh machine.

Prerequisites

• A Linux host (bare metal or cloud VM). Windows works via WSL2 with GPU passthrough; macOS cannot run the container (no NVIDIA GPU support). All commands are bash.

• An NVIDIA GPU. The example recipe in this guide fits on a single GPU with 8 GB of memory using the memory override shown in the launch step; production recipes target datacenter GPUs, up to 8x H100

• Docker with the NVIDIA Container Toolkit installed and configured for Docker (nvidia-ctk runtime configure --runtime=docker, then restart Docker). Check it works before anything else:

  docker run --rm --gpus all ubuntu nvidia-smi

• About 150 GiB of free disk: the image is roughly a 19 GB download and 40 GB unpacked, plus about 34 GB of weights for the 27B FP8 models

1. Get your credentials

Two separate credentials are involved. Do not conflate them:

The registry token is shown once and never stored. After the hour the pull is refused and you mint another; nothing already running is affected. For a credential that outlives the hour, create a named API token and have your pipeline mint a fresh registry token each run.

2. Sign in to the registry

export CF_REGISTRY_TOKEN='<paste the generated token>'

echo "$CF_REGISTRY_TOKEN" | docker login us-central1-docker.pkg.dev \
  --username oauth2accesstoken --password-stdin

Piping the token keeps it out of the docker login command itself, so it never appears in the process list on a shared host.

3. Pull

docker pull us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4

cf-bundle is one image for every model: the tag pins the measured build (runtime and scheduler releases), and the context key at run time selects the pre-optimised recipe for your model and hardware. Pin the tag (or better, the digest) in anything automated.

4. Check your host

Run the container with no arguments to diagnose the machine. It prints a report and exits; it does not serve anything.

docker run --rm --gpus all \
  us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4

A healthy host ends with:

GPU / NVML:
  OK    1 GPU(s), driver 595.84

doctor: OK

A WARN no license key provided line here is normal if you have not mounted the license yet; free-tier recipes run without one. Fix anything reported as FAIL before continuing.

5. Launch with a context key

A context key selects a benchmarked serving recipe: model, quantization, workload shape and hardware. Your container's page in the library shows the key to use.

This example uses the smallest free recipe so you can try the whole flow on a laptop. The recipe is tuned for a 24 GB L4; the CF_GPU_MEMORY_UTILIZATION override makes it fit reliably on an 8 GB consumer GPU (at that point you are trying the flow, not the benchmarked configuration; on an L4 or larger, drop the override):

docker run -d --name carbonforge --gpus all -p 8000:8000 \
  -e CF_CONTEXT_KEY="qwen2.5-0.5b:fp8:chat:l4x1" \
  -e CF_GPU_MEMORY_UTILIZATION=0.6 \
  -v hf-cache:/root/.cache/huggingface \
  -v carbonforge-logs:/var/log/carbonforge \
  us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4

For a production model, use the context key shown on your container's library page instead, and add --shm-size 16g on multi-GPU recipes. For a premium container, also pass your license: -e CARBONFORGE_LICENSE_KEY="<LICENSE_KEY>". The GPU must match the key's hardware segment and have enough free memory; nvidia-smi --query-gpu=name,memory.free,driver_version --format=csv confirms both. A hardware mismatch warns loudly but still serves; the performance numbers only hold on the recipe's tuned hardware.

6. Verify

First boot downloads model weights before serving: a few minutes for a small model, 10 to 20 minutes for a 27B model. The weights persist in the hf-cache volume, so later boots skip the download, but every boot still spends 2 to 3 minutes initializing the engine (kernel warm-up and CUDA graph capture) before the port answers. Follow progress with docker logs -f carbonforge. Then:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"What is the capital city of France?"}]}'

If this answers, the container is serving on port 8000. If something fails, see Troubleshooting.

7. Enable tracing (optional)

Per-request traces are off by default. See Tracing for sinks and privacy levels.