# For agents (/docs/agents)
Every page on this site is available as plain markdown, and the whole
site is indexed for agents. If you use a coding agent, point it here
instead of pasting pages by hand.
## Routes [#routes]
| Route | What it returns |
| -------------------------------- | -------------------------------------------------------- |
| [/llms.txt](/llms.txt) | An index of every page: title, URL, one-line description |
| [/llms-full.txt](/llms-full.txt) | The entire documentation as one markdown document |
| any page URL + `.md` | That page as raw markdown, e.g. `/docs/endpoint.md` |
The plain page URLs also content-negotiate: a request with
`Accept: text/markdown` gets the markdown representation from the same
URL that serves the HTML.
## Prompt to give your agent [#prompt-to-give-your-agent]
Paste this into your agent's instructions or context:
```
CarbonForge documentation is at https://docs.carbonforge.ai. Fetch
/llms.txt for an index of all pages. Append .md to any page URL to get
its raw markdown, or fetch /llms-full.txt for the entire documentation
in one file.
```
# Overview (/docs)
CarbonForge serves tuned, energy-aware LLM inference through two
pathways:
## Choosing a pathway [#choosing-a-pathway]
| | API endpoint | Self-hosted container |
| -------------- | ------------------------------ | -------------------------------- |
| Setup | Faster | Slower |
| Whose hardware | CarbonForge's | Yours |
| Data security | SOC 2 compliant infrastructure | Never leaves your infrastructure |
For a quick evaluation, the endpoint is the shortest path. For
data-residency or compliance constraints, the container keeps everything
in-house.
# Configuration (/docs/container/configuration)
## Launch modes [#launch-modes]
The container entrypoint has three modes:
1. **No arguments**: runs the doctor. Diagnoses the host and exits.
2. **`CF_CONTEXT_KEY` set**: serves the recipe named by the
[context key](/docs/container/context-keys). The normal path.
3. **Arguments after the image name**: full override; bypasses the recipe
and passes your arguments to the runtime directly.
## Volumes [#volumes]
| Mount | Purpose |
| ------------------------------------------------ | ---------------------------------------------------- |
| `:/etc/carbonforge/license.key:ro` | License (alternative: `CARBONFORGE_LICENSE_KEY` env) |
| `hf-cache:/root/.cache/huggingface` | Model weights cache; persists across restarts |
| `carbonforge-logs:/var/log/carbonforge` | Runtime and engine logs, trace files |
## Environment variables [#environment-variables]
| Variable | Required | Effect |
| --------------------------- | ------------------- | --------------------------------------------- |
| `CF_CONTEXT_KEY` | For serving | Recipe to serve; unset runs the doctor |
| `CARBONFORGE_LICENSE_KEY` | For premium recipes | License key value (instead of the file mount) |
| `HF_TOKEN` | Gated models only | Hugging Face auth for weight download |
| `CF_DRY_RUN` | No | Observe-only mode; no control actions |
| `CF_MAX_MODEL_LEN` | No | Override the recipe's context length |
| `CF_GPU_MEMORY_UTILIZATION` | No | Override the recipe's GPU memory fraction |
| `CF_VLLM_EXTRA_ARGS` | No | Extra vLLM flags, space-separated |
> **Overrides depart from the benchmark.** Every `CF_*` override makes the
> container serve a configuration different from the one the recipe was
> benchmarked on. Performance and energy numbers no longer apply.
## Ports [#ports]
| Port | What |
| ---- | ---------------------------------------------------------- |
| 8000 | OpenAI-compatible API |
| 9105 | Prometheus metrics, only with `--metrics` (off by default) |
## Concurrency [#concurrency]
The engine enforces a hard ceiling on concurrent sequences
(`--max-num-seqs`, recipe-dependent). A gateway in front of the container
will happily accept more; the excess queues inside the engine and shows up
as tail latency. Size your client concurrency to the recipe.
# Context keys (/docs/container/context-keys)
A context key names the situation you are serving in. The container resolves
it to a recipe: the model, its quantization, every serving knob, and the
scheduler and controller configuration that was benchmarked for that
situation.
## Format [#format]
```
: : :
```
For example:
```
qwen3-32b:fp8:chat:a100-80x1
```
* **model** and **quant** identify what is served.
* **workload** describes the traffic shape the recipe was tuned for
(`chat`, `coding`, `agentic`).
* **hardware** is your declaration of the GPUs you run on. It is validated
against the detected hardware: a mismatch warns loudly but still runs,
unless you pass `--strict-hardware`.
## Available launch keys [#available-launch-keys]
| Context key | Plan |
| -------------------------------------- | ----------------------- |
| `qwen3-32b:fp8:chat:a100-80x1` | Free |
| `qwen3-coder-32b:fp8:coding:a100-80x1` | Free |
| `qwen2.5-0.5b:fp8:chat:l4x1` | Free (validation-sized) |
| `qwen3-100b:fp8:chat:h100x2` | Premium |
| `qwen3-coder-next:fp8:coding:h100x4` | Premium |
| `qwen3.6-27b:fp8:agentic:h100x8` | Premium |
A valid license key grants the premium tier. Without one, the container
serves free-tier recipes only.
## How it is used [#how-it-is-used]
```bash
docker run ... -e CF_CONTEXT_KEY="qwen3-32b:fp8:chat:a100-80x1" ...
```
or as a flag when overriding the entrypoint:
```bash
carbonforge-runtime --context-key qwen3-32b:fp8:chat:a100-80x1
```
# Data boundary (/docs/container/data-boundary)
The container is built for workloads that cannot leave your
infrastructure. This page states the boundary precisely, so your
security review does not have to reverse-engineer it.
## What transits CarbonForge [#what-transits-carbonforge]
| When | What | Where it goes |
| ------------------- | ---------------------------------------------------- | ------------------------------------------------------ |
| Sign-in, token mint | Your account identity; which container you requested | CarbonForge console and registry token service |
| `docker pull` | The container image, downloaded to your host | From CarbonForge's registry (Google Artifact Registry) |
| Licence issue | The signed licence key, issued once | From the console to you |
That is the complete list, and all of it happens before the container
serves its first request.
## What never leaves your infrastructure [#what-never-leaves-your-infrastructure]
| Data | Where it stays |
| ----------------------- | --------------------------------------------------------------------- |
| Prompts and outputs | Your host: in GPU memory and the response to your client |
| Model weights | Downloaded from Hugging Face (not CarbonForge) into your cache volume |
| Engine and runtime logs | Your log volume (`/var/log/carbonforge`) |
| Request traces | Off by default; when enabled, only the sinks you configure |
| Metrics | Off by default; when enabled, a port on your host |
## No runtime phone-home [#no-runtime-phone-home]
The running container makes no calls to CarbonForge:
* **Licence checking is offline.** The licence is an Ed25519-signed
document verified inside the container against an embedded public key.
No licence server is contacted, so the container runs in fully
air-gapped networks. (The flip side: a licence cannot be revoked
before its expiry, which is why licences are issued with bounded
terms.)
* **No telemetry.** Nothing is reported back: not usage, not metrics,
not errors.
* **Tracing is yours.** `--request-trace` writes to a file on your
volume or an OpenTelemetry collector you name. At the `full` level
traces contain raw prompt and output text, so treat that setting with
the care described in [Tracing](/docs/container/tracing).
## Network requirements [#network-requirements]
Outbound access is needed only at set-up time: to CarbonForge's registry
for the image pull, and to Hugging Face for the model weights. After
[prefetching weights](/docs/container/quickstart), the container serves
with no outbound network access at all (`HF_HUB_OFFLINE=1`).
# Quick start (/docs/container/quickstart)
This guide takes you from nothing to a serving OpenAI-compatible endpoint on
your own GPU. Budget about 20 minutes of wall-clock for image and weight
downloads on a fresh machine.
## Prerequisites [#prerequisites]
* A Linux host (bare metal or cloud VM). Windows works via WSL2 with GPU
passthrough; macOS cannot run the container (no NVIDIA GPU support).
All commands are bash.
* An NVIDIA GPU. The example recipe in this guide fits on a single GPU
with 8 GB of memory using the memory override shown in the launch step;
production recipes target datacenter GPUs, up to 8x H100
* Docker with the [NVIDIA Container
Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
installed **and configured** for Docker
(`nvidia-ctk runtime configure --runtime=docker`, then restart Docker).
Check it works before anything else:
```bash
docker run --rm --gpus all ubuntu nvidia-smi
```
* About 150 GiB of free disk: the image is roughly a 19 GB download and
40 GB unpacked, plus about 34 GB of weights for the 27B FP8 models
## 1. Get your credentials [#1-get-your-credentials]
Two separate credentials are involved. Do not conflate them:
| Credential | What it does | Where you get it |
| -------------- | ------------------------------------------------------------- | ----------------------------------------- |
| Registry token | Docker registry password, valid one hour, this container only | Container page, "Generate registry token" |
| License key | Unlocks premium containers at run time | Console, account page |
The registry token is shown once and never stored. After the hour the pull
is refused and you mint another; nothing already running is affected. For
a credential that outlives the hour, create a named API token and have
your pipeline mint a fresh registry token each run.
## 2. Sign in to the registry [#2-sign-in-to-the-registry]
```bash
export CF_REGISTRY_TOKEN=''
echo "$CF_REGISTRY_TOKEN" | docker login us-central1-docker.pkg.dev \
--username oauth2accesstoken --password-stdin
```
Piping the token keeps it out of the `docker login` command itself, so it
never appears in the process list on a shared host.
## 3. Pull [#3-pull]
```bash
docker pull us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4
```
`cf-bundle` is one image for every model: the tag pins the measured build
(runtime and scheduler releases), and the context key at run time selects
the pre-optimised recipe for your model and hardware. Pin the tag (or
better, the digest) in anything automated.
## 4. Check your host [#4-check-your-host]
Run the container with no arguments to diagnose the machine. It prints a
report and exits; it does not serve anything.
```bash
docker run --rm --gpus all \
us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4
```
A healthy host ends with:
```
GPU / NVML:
OK 1 GPU(s), driver 595.84
doctor: OK
```
A `WARN no license key provided` line here is normal if you have not
mounted the license yet; free-tier recipes run without one. Fix anything
reported as `FAIL` before continuing.
## 5. Launch with a context key [#5-launch-with-a-context-key]
A [context key](/docs/container/context-keys) selects a benchmarked serving
recipe: model, quantization, workload shape and hardware. Your container's
page in the library shows the key to use.
This example uses the smallest free recipe so you can try the whole flow
on a laptop. The recipe is tuned for a 24 GB L4; the
`CF_GPU_MEMORY_UTILIZATION` override makes it fit reliably on an 8 GB
consumer GPU (at that point you are trying the flow, not the benchmarked
configuration; on an L4 or larger, drop the override):
```bash
docker run -d --name carbonforge --gpus all -p 8000:8000 \
-e CF_CONTEXT_KEY="qwen2.5-0.5b:fp8:chat:l4x1" \
-e CF_GPU_MEMORY_UTILIZATION=0.6 \
-v hf-cache:/root/.cache/huggingface \
-v carbonforge-logs:/var/log/carbonforge \
us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4
```
For a production model, use the context key shown on your container's
library page instead, and add `--shm-size 16g` on multi-GPU recipes. For
a premium container, also pass your license:
`-e CARBONFORGE_LICENSE_KEY=""`. The GPU must match the
key's hardware segment and have enough free memory;
`nvidia-smi --query-gpu=name,memory.free,driver_version --format=csv`
confirms both. A hardware mismatch warns loudly but still serves; the
performance numbers only hold on the recipe's tuned hardware.
## 6. Verify [#6-verify]
First boot downloads model weights before serving: a few minutes for a
small model, 10 to 20 minutes for a 27B model. The weights persist in the
`hf-cache` volume, so later boots skip the download, but every boot still
spends 2 to 3 minutes initializing the engine (kernel warm-up and CUDA
graph capture) before the port answers. Follow progress with
`docker logs -f carbonforge`. Then:
```bash
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"What is the capital city of France?"}]}'
```
If this answers, the container is serving on port 8000. If something
fails, see [Troubleshooting](/docs/container/troubleshooting).
## 7. Enable tracing (optional) [#7-enable-tracing-optional]
Per-request traces are off by default. See
[Tracing](/docs/container/tracing) for sinks and privacy levels.
# Tracing (/docs/container/tracing)
The scheduler emits one trace record per completed request: queue wait,
total latency, token counts, and scheduler decisions. This page covers
enabling it on the container; what the records contain at each level is
documented in [Tracing](/docs/tracing).
## Enabling [#enabling]
Tracing needs **a level and at least one sink**. A level without a sink
logs one warning and stays off; the server still serves.
Tracing flags are runtime arguments, so they are passed as the container
command. To keep serving your recipe, include the context key as a flag
(the `CF_CONTEXT_KEY` environment variable is ignored once a command is
given):
```bash
docker run ... \
us-central1-docker.pkg.dev/potent-arcade-491015-g7/cf-containers-dev/cf-bundle:v0.1.28-v0.1.4 \
--context-key "qwen2.5-0.5b:fp8:chat:l4x1" \
--request-trace standard \
--trace-file /var/log/carbonforge/traces.jsonl
```
Each completed request appends one JSON line to the trace file:
```json
{"request_id": "chatcmpl-...", "t_arrival": 1788962815.33, "t_completion": 1788962816.58,
"prompt_tokens": 30, "output_tokens": 8, "e2e_latency_s": 1.25, "ttft_s": null,
"queue_wait_s": 0.0067, "scheduler_family": "ltr-promptlen", "priority_score": 96.0}
```
Or export to an OpenTelemetry collector:
```bash
--request-trace standard \
--trace-endpoint http://my-collector:4317 \
--trace-format otlp
```
Both sinks can be set at once (the trace is written to the file and
exported).
## Levels and privacy [#levels-and-privacy]
| Level | Contents |
| ---------- | -------------------------------------------------------- |
| `off` | Nothing (default) |
| `standard` | Timings, token counts, scheduler metadata. Content-safe. |
| `full` | Everything above **plus raw prompt and output text** |
> **Warning: `full` ships your prompts.** At `full`, raw prompt and
> completion text is written to every configured sink. Use it only against
> sinks you control, never in production with customer data.
## Span shape [#span-shape]
OTLP export emits one span per request named `llm_request`, with
`gen_ai.usage.prompt_tokens`, `gen_ai.usage.completion_tokens` and
scheduler attributes. Time-to-first-token is not instrumented; streaming
and non-streaming requests produce the same span shape.
## Securing the export [#securing-the-export]
The OTLP exporter currently supports no authentication headers and no TLS.
Run the collector on the same host or private network as the container.
# Troubleshooting (/docs/container/troubleshooting)
Run the doctor first: a bare `docker run --rm --gpus all ` diagnoses
most host problems.
## Reading the engine log [#reading-the-engine-log]
When the engine fails, `docker logs` shows only the runtime's summary; the
root cause is in the engine log at `/var/log/carbonforge/vllm.log` inside
the container. While the container is running:
```bash
docker exec carbonforge cat /var/log/carbonforge/vllm.log
```
After it has exited, read it straight from the logs volume (the
`carbonforge-logs` volume from the quick start's launch command):
```bash
docker run --rm -v carbonforge-logs:/logs alpine cat /logs/vllm.log
```
Every entry below that mentions `vllm.log` means this file.
## `could not select device driver "" with capabilities: [[gpu]]` [#could-not-select-device-driver--with-capabilities-gpu]
**Symptom**: any `docker run --gpus all` fails immediately with this error.
Docker cannot see the NVIDIA runtime. Either the NVIDIA Container Toolkit
is not installed, or it was installed but never wired into Docker: run
`sudo nvidia-ctk runtime configure --runtime=docker`, restart Docker, and
retry. Verify with `docker run --rm --gpus all ubuntu nvidia-smi`.
## The container exits instead of serving [#the-container-exits-instead-of-serving]
**Symptom**: container starts, prints a report, exits 0.
You launched without `CF_CONTEXT_KEY` (or arguments), which runs the
doctor by design. Set a [context key](/docs/container/context-keys).
## "no license key provided" despite the mount [#no-license-key-provided-despite-the-mount]
**Symptom**: the license file is mounted but the container refuses premium
recipes.
The file is unreadable inside the container. Check mode and ownership on
the host (`chmod 600`, readable by the container user), or pass
`CARBONFORGE_LICENSE_KEY` as an environment variable instead.
## Refused as unlicensed with an explicit model override [#refused-as-unlicensed-with-an-explicit-model-override]
**Symptom**: `could not determine which model to gate` or an unlicensed
refusal when overriding the entrypoint.
The model must be the **first argument after `--`** (or passed as
`--model`). The entitlement gate reads that position.
## Crash at engine warm-up: `Could not find nvcc` [#crash-at-engine-warm-up-could-not-find-nvcc]
**Symptom**: dies at `compile_or_warm_up_model` with
`cuda_home='/usr/local/cuda' doesn't exist`.
The engine JIT-compiles a sampling kernel and needs a CUDA toolkit. The
container ships one, so this only affects native (non-container) installs:
either install the CUDA toolkit on the host, or set
`VLLM_USE_FLASHINFER_SAMPLER=0`.
## Engine dies at start-up with a GPU out-of-memory [#engine-dies-at-start-up-with-a-gpu-out-of-memory]
**Symptom**: the runtime logs `vLLM exited (code 1) before becoming ready`;
`vllm.log` shows `memory allocation failed with OOM on device 0` during
"Capturing CUDA graphs" (or earlier, while loading weights).
The GPU does not have enough free memory for the recipe. If the card is
smaller than the recipe's tuned hardware (the last segment of the context
key), this failure is expected, and can be intermittent when the margin
is a few MiB: a run that once succeeded can fail the next time. Shrink
the engine's footprint with `-e CF_GPU_MEMORY_UTILIZATION=0.6`, or add
`-e CF_VLLM_EXTRA_ARGS="--enforce-eager"` to skip CUDA graph capture at
some throughput cost; both overrides depart from the benchmarked recipe.
Also check nothing else is holding GPU memory (`nvidia-smi`). Note that
`docker start` reuses the container's original configuration; to change
an environment variable you must `docker rm -f` the container and run it
again.
## Model not found despite prefetched weights [#model-not-found-despite-prefetched-weights]
**Symptom**: offline host, weights prefetched, engine reports the model
missing.
The engine looks in the `hub/` subdirectory of the cache. Set
`HUGGINGFACE_HUB_CACHE` to the directory you prefetched into, and keep it
consistent with the volume mount.
## `/v1/models` refuses connections after launch [#v1models-refuses-connections-after-launch]
**Symptom**: container is up and healthy but the port does not answer.
Normal during first boot: image plus weights take 10 to 20 minutes on a
fresh machine. Follow `docker logs -f carbonforge` and wait for the engine
ready line.
## Gateway integration returns 404 for the model [#gateway-integration-returns-404-for-the-model]
**Symptom**: direct curl works; the gateway in front returns model not
found.
The served model name must match the gateway's model list **character for
character**. Set `--served-model-name` explicitly rather than relying on
the default.
## Tail latency collapses under load [#tail-latency-collapses-under-load]
**Symptom**: p50 fine, p99 grows without bound as concurrency rises.
The engine's `--max-num-seqs` is a hard ceiling; excess requests queue.
Bound client concurrency, and disable gateway retries when measuring.
## Tracing enabled but no traces appear [#tracing-enabled-but-no-traces-appear]
**Symptom**: `--request-trace standard` set, sinks empty.
Two causes: no sink configured (a level alone logs a warning and disables
tracing), or the OpenTelemetry SDK is missing from the serving venv on a
native install (spans are silently dropped; the doctor reports it as a
`WARN`).
# Errors (/docs/endpoint/errors)
Errors follow the OpenAI JSON shape: a body of
`{"error": {"message", "type", "param", "code"}}`.
| Status | When | What to do |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `401` | Missing or invalid API key (`"Authentication Error, No api key passed in."`) | Check the `Authorization: Bearer` header and the key value |
| `400` | Invalid request: malformed JSON (`"Invalid JSON payload"`), a missing required parameter (`"Missing required parameter: 'messages'"`), or an unaccepted `reasoning_effort` (`"Unexpected reasoning effort"`) | Fix the request; for effort, only `low`, `medium`, `xhigh` and `{"enable_thinking": false}` are accepted; `high` is not |
| `403` | Model not available to your key (`"key not allowed to access model"`; the message lists the models you can access) | Use a model from `GET /v1/models`; the list is filtered per key |
No rate-limit responses are currently issued on the evaluation
deployment. The 400-concurrent-requests figure in the quick start is an
operational ceiling to respect, not an enforced limit.
## Failures that do not return an error [#failures-that-do-not-return-an-error]
Two failure modes arrive as **HTTP 200** and need client-side checks:
1. **Capped output with reasoning on**: `finish_reason: "length"` with
truncated (sometimes empty) `content`. Check `finish_reason` on every
response if you pass a token cap at all; better, do not cap (see the
[client configuration advice](/docs/endpoint#2-recommended-client-configuration)).
2. **Client timeout**: your SDK aborts a request the endpoint was still
serving. This surfaces as a client-side exception, not an endpoint
error; set the timeout to at least 600 seconds.
## When reporting a problem [#when-reporting-a-problem]
Include the `id` field from the failing response (for example
`chatcmpl-...`); it lets support find the exact request.
# Quick start (/docs/endpoint)
The CarbonForge endpoint is OpenAI-compatible: use the OpenAI SDK,
LiteLLM, LangChain, or plain HTTP. Anthropic and Google clients will not
work against it.
## What you need [#what-you-need]
* For the curl examples: any OS with `curl` (preinstalled on Linux and
macOS; on Windows, use PowerShell 7+, WSL, or Git Bash). Shell examples
use bash syntax; on native Windows adapt the quoting and use
`$env:CF_API_KEY` instead of `$CF_API_KEY`.
* For the Python examples: Python 3.9 or newer and the client of your
choice (`pip install openai`, `litellm`, or `langchain-openai`). Any
OpenAI-compatible client works the same way; only the base URL and key
change.
## 1. Get your credentials [#1-get-your-credentials]
Your CarbonForge contact sends the API key separately.
```
BASE_URL https://endpoint.carbonforge.ai/v1
API_KEY
MODELS cf-chat cf-batch
```
## 2. Recommended client configuration [#2-recommended-client-configuration]
Two SDK defaults work against you on this endpoint; results are much
more reliable with both adjusted before the first real request:
1. **Prefer leaving the output uncapped.** The model reasons before
answering and the reasoning counts against `max_tokens`, so a capped
request can spend part or all of its budget before the answer starts,
returning HTTP 200 with truncated (sometimes empty) `content` and
`finish_reason: "length"` rather than an error. If bounded output is
a hard requirement, pairing the cap with `{"enable_thinking": false}`
(see [Reasoning](/docs/endpoint/reasoning)) restores normal
behaviour.
2. **A client timeout of at least 600 seconds is preferable.** Most
SDKs default to 60 s, which aborts work the endpoint would have
completed: in a representative run, 6.6% of requests took longer
than 60 s (slowest: 221 s; nothing exceeded 300 s).
## 3. First request [#3-first-request]
**These examples use `cf-chat`**, the model for interactive traffic;
bulk work belongs on `cf-batch`. The split is explained in
[Pick the right model](#4-pick-the-right-model) below.
```bash
curl https://endpoint.carbonforge.ai/v1/chat/completions \
-H "Authorization: Bearer $CF_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"cf-chat","messages":[{"role":"user","content":"Hello"}]}'
```
```python
from openai import OpenAI
client = OpenAI(
base_url="https://endpoint.carbonforge.ai/v1",
api_key=CF_API_KEY,
timeout=600.0, # recommendation 2 above
)
resp = client.chat.completions.create(
model="cf-chat",
messages=[{"role": "user", "content": "Hello"}],
# no max_tokens: recommendation 1 above
)
print(resp.choices[0].message.content)
```
```python
from litellm import completion
resp = completion(
model="openai/cf-chat",
api_base="https://endpoint.carbonforge.ai/v1",
api_key=CF_API_KEY,
messages=[{"role": "user", "content": "Hello"}],
timeout=600,
)
print(resp.choices[0].message.content)
```
```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="https://endpoint.carbonforge.ai/v1",
api_key=CF_API_KEY,
model="cf-chat",
timeout=600,
)
print(llm.invoke("Hello").content)
```
The model reasons before answering. The reasoning comes back in a
separate field and can be ignored:
```python
resp.choices[0].message.reasoning_content # the thinking
resp.choices[0].message.content # the answer
```
## 4. Pick the right model [#4-pick-the-right-model]
| model | use it for |
| ---------- | -------------------------------------- |
| `cf-chat` | someone is waiting on the response |
| `cf-batch` | scheduled or bulk work, nobody waiting |
Throughput is the same or better on `cf-batch`; it simply carries no
latency guarantee, so a single request may take considerably longer when
the system is busy. Routing work to the matching tier lets each be tuned
independently and keeps a large batch job from slowing your interactive
traffic.
`GET /v1/models` returns the authoritative list for your key.
### Batch work is just concurrency [#batch-work-is-just-concurrency]
There is no separate batch API. `cf-batch` takes ordinary concurrent
requests, so a batch is many requests in parallel. Start around 16 to 32
concurrent and ramp while watching your own latency. Do not exceed 400
concurrent requests on one key.
## Images [#images]
Inline base64 in the message content, using the OpenAI content-array
shape with a data URL. There is no file upload endpoint.
```json
{"model": "cf-chat", "messages": [{"role": "user", "content": [
{"type": "text", "text": "What color is this image? Answer with one word."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
]}]}
```
## Support [#support]
Reply to your CarbonForge contact for issues or key rotation. Include the
`id` field from any failing response (for example `chatcmpl-...`); it
lets us find the exact request.
# Reasoning (/docs/endpoint/reasoning)
The model thinks before it answers. The thinking arrives in
`message.reasoning_content`, the answer in `message.content`, and the
amount of thinking is the main lever on both latency and output volume,
because reasoning counts toward the tokens generated.
## Setting the effort [#setting-the-effort]
```json
{
"model": "cf-chat",
"messages": [{"role": "user", "content": "..."}],
"chat_template_kwargs": {"reasoning_effort": "low"}
}
```
| value | effect |
| -------------------------------- | --------------------------------- |
| `{"reasoning_effort": "low"}` | the default if you send nothing |
| `{"reasoning_effort": "medium"}` | more thinking, more output tokens |
| `{"reasoning_effort": "xhigh"}` | deepest available |
| `{"enable_thinking": false}` | skips reasoning entirely |
Only those four are accepted; anything else returns
`400 Unexpected reasoning effort`. In particular **`high` is not valid**
even though `xhigh` is, and neither are `max`, `minimal` or `none`.
## Output caps and reasoning do not mix [#output-caps-and-reasoning-do-not-mix]
A `max_tokens` cap is only unsafe *with reasoning on*: the reasoning is
emitted first and spends the budget, and a request that hits the cap
returns HTTP 200 with `finish_reason: "length"` and truncated content,
with no error to catch. How much content survives depends on how much
the model decided to think: it can be a partial answer or nothing at
all. One measured example where the reasoning consumed the entire cap:
```
max_tokens=256 content: 0 chars reasoning: 1150 chars
max_tokens=256 + enable_thinking:false content: 1170 chars reasoning: 0 chars
```
If you need bounded output, pair the cap with
`{"enable_thinking": false}`. Otherwise leave the cap off: the endpoint
imposes no ceiling and the model stops when it is finished.
## What to expect [#what-to-expect]
Worth calibrating before you point real workloads at it:
```
a one-line question ~1,500 output tokens, most of it reasoning
per-request generation rate ~60 tokens/second
6 concurrent realistic requests 11 to 17 seconds each
```
Output length is unbounded by design, so latency and volume scale with
how much the model decides to think. `reasoning_effort` is the control;
`enable_thinking: false` is the floor.
## Prefix caching [#prefix-caching]
A shared prompt prefix is cached, and a cache hit skips prefill
entirely. In our measurements a stable \~2,000-token prefix cut
time-to-first-token by roughly 100 ms against an uncached one.
If you have a fixed system prompt or tool schema, keeping it
byte-identical across requests is worth doing and costs nothing. The
block size and exact hit rates are still being characterised, so do not
pad prompts to a specific length yet; firmer numbers will follow.
# Streaming (/docs/endpoint/streaming)
Supported and recommended for anything user-facing:
```json
{"model": "cf-chat", "messages": [...], "stream": true}
```
```python
stream = client.chat.completions.create(
model="cf-chat",
messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
```
With `stream_options.include_usage`, the final chunk carries prompt and
completion token counts.
## What the example prints [#what-the-example-prints]
The loop above prints each `content` delta as it arrives, so the answer
appears fragment by fragment:
```
Thousands of cores hum
fanning out to paint the dark,
green light in the case
```
The same response as it arrived on the wire, one chunk per delta (from a
real run; note chunks split mid-word):
```
"\n\nThousands" " of" " cores" " hum" "\n" "f" "anning" " out" " to"
" paint" " the" " dark" "," "\n" "green" " light" " in" " the" " case"
```
In that run, those 19 content chunks were preceded by 201
`reasoning_content` chunks that the loop deliberately skips.
## The first chunk is thinking, not the answer [#the-first-chunk-is-thinking-not-the-answer]
One caveat if you are driving a UI. The first chunk arrives fast (about
0.4 s in the run above), but it is **reasoning**, not the answer: the
first chunk of actual `content` lands only once the model has finished
thinking (2.2 s in that run; longer when the model thinks harder). So
either render the reasoning as it streams, or show a thinking indicator
until `content` starts, otherwise the user watches an empty box.
# Full level (/docs/tracing/full)
`full` is `standard` plus two fields: `prompt_text` and `output_text`,
the raw message content of the request. It exists for one purpose:
capturing a workload faithfully enough to replay it, so a recipe can be
tuned against your real traffic rather than a synthetic approximation.
> **Full ships customer content.** The raw prompt and output go to every
> configured sink: the trace file, and the `gen_ai.prompt` /
> `gen_ai.completion` attributes of exported spans. Enable it only
> against sinks you control, never in production with customer data.
## The record [#the-record]
Same fields as [standard](/docs/tracing/standard), plus:
| Field | Meaning |
| ------------- | ----------------------------------------------------------- |
| `prompt_text` | The full rendered prompt, system and chat template included |
| `output_text` | The complete generated text |
## Example [#example]
One record from a support-assistant workload (content abridged):
```json
{"request_id": "chatcmpl-7d3fa9b2", "t_arrival": 1789040231.522, "t_completion": 1789040235.108,
"prompt_tokens": 634, "output_tokens": 187, "e2e_latency_s": 3.586, "ttft_s": null,
"queue_wait_s": 0.006, "scheduler_family": "ltr-promptlen", "priority_score": -512.0,
"prompt_text": "<|im_start|>system\nYou are the support assistant for Northwind Analytics. Answer from the provided context only.<|im_end|>\n<|im_start|>user\nContext: [invoice INV-2093: issued 2026-08-01, net-30, paid 2026-08-19] ...\nQuestion: My dashboard says invoice INV-2093 is overdue but we paid it three weeks ago. Can you check?<|im_end|>\n<|im_start|>assistant\n",
"output_text": "I checked invoice INV-2093 against the records in the context: it was issued on 2026-08-01 with net-30 terms and marked paid on 2026-08-19, well within terms. The overdue flag on your dashboard is therefore stale rather than a missed payment. Two things usually cause this: ..."}
```
Note that `prompt_text` is the *rendered* prompt: chat-template markup
and the system message are included, because that is what the engine
actually served and what a replay must reproduce.
## How a conversation appears in a trace [#how-a-conversation-appears-in-a-trace]
Traces carry **no session identifier**: one span per request, flat. A
multi-turn conversation is still visible, because each turn's rendered
prompt embeds every prior turn. Two consecutive spans from one session
(OTLP export, abridged; `` markers are the chat
template's thinking-disabled artifact):
```json
{"name": "llm_request",
"attributes": [
{"key": "cf.request_id", "value": {"stringValue": "chatcmpl-850f15e9afd74036-82d9d8e1"}},
{"key": "gen_ai.usage.prompt_tokens", "value": {"intValue": "36"}},
{"key": "cf.e2e_latency_s", "value": {"doubleValue": 0.1829}},
{"key": "cf.queue_wait_s", "value": {"doubleValue": 0.0136}},
{"key": "gen_ai.prompt", "value": {"stringValue": "system\nYou are the billing assistant. Be brief.\nuser\nWhere do I find my invoice history?\nassistant\n\n\n\n\n"}},
{"key": "gen_ai.completion", "value": {"stringValue": "Billing > Invoices, top right of the console."}}
]}
{"name": "llm_request",
"attributes": [
{"key": "cf.request_id", "value": {"stringValue": "chatcmpl-93bcfebf2cf706f4-91470f16"}},
{"key": "gen_ai.usage.prompt_tokens", "value": {"intValue": "73"}},
{"key": "cf.e2e_latency_s", "value": {"doubleValue": 0.1221}},
{"key": "cf.queue_wait_s", "value": {"doubleValue": 0.0099}},
{"key": "gen_ai.prompt", "value": {"stringValue": "system\nYou are the billing assistant. Be brief.\nuser\nWhere do I find my invoice history?\nassistant\n\n\n\n\nBilling > Invoices, top right of the console.\nuser\nAnd past payment receipts?\nassistant\n\n\n\n\n"}},
{"key": "gen_ai.completion", "value": {"stringValue": "Same page: each invoice row expands to its receipt."}}
]}
```
The second span's prompt contains the first span's prompt and answer as
a prefix. That growing-prefix structure is what lets sessions be
reconstructed from a flat trace, and it is also why full-level capture
is required for faithful replay: the session grouping lives in the
text, not in any metadata field.
## Caveats [#caveats]
* **Scheduler coverage**: the `ltr`, `ltr-promptlen`, `affinity` and
`trace` schedulers include the text fields; `tie` and `hybrid`
currently omit them.
* **Production guidance**: run `standard` in production and switch to
`full` only for bounded capture windows on traffic you are entitled
to record.
* On the container pathway, enabling either level is your action and
your sinks; see [Container tracing](/docs/container/tracing).
# What tracing is (/docs/tracing)
CarbonForge tracing emits **one record per completed request**, produced
by the scheduler inside the serving engine. A trace answers, per
request: when it arrived, how long it queued, how long it took end to
end, how many tokens went in and out, and how the scheduler treated it.
What traces are for:
* **Capacity planning**: queue wait and end-to-end latency under your
real traffic, not a benchmark's.
* **Workload replay**: a captured trace can regenerate statistically
faithful traffic for tuning and regression runs.
* **Accounting**: per-request token counts from the engine itself.
## The two levels [#the-two-levels]
| Level | Contents | Safe for |
| ------------------------------------ | -------------------------------------------------------------- | ------------------------------------- |
| [`standard`](/docs/tracing/standard) | Timings, token counts, scheduler metadata. No message content. | Production |
| [`full`](/docs/tracing/full) | Everything in standard **plus the raw prompt and output text** | Development and workload capture only |
There is also `off`, the default: nothing is emitted.
## Where tracing runs, per pathway [#where-tracing-runs-per-pathway]
* **Self-hosted container**: tracing is off by default and entirely
yours. Records go only to sinks you configure: a JSONL file on your
log volume, an OpenTelemetry collector you name, or both. Nothing is
sent to CarbonForge. Enabling it is covered in
[Container tracing](/docs/container/tracing).
* **API endpoint**: tracing is operated by CarbonForge on the serving
side; there is nothing to configure. The record shape is the same,
which is what makes workloads comparable across the two pathways.
## Record shape in one line [#record-shape-in-one-line]
```json
{"request_id": "...", "t_arrival": ..., "t_completion": ..., "prompt_tokens": ...,
"output_tokens": ..., "e2e_latency_s": ..., "ttft_s": null, "queue_wait_s": ...,
"scheduler_family": "...", "priority_score": ...}
```
`ttft_s` is currently always `null`: first-token time is not visible to
the scheduler, so the record covers whole-request timing only, for
streaming and non-streaming requests alike. The
[standard](/docs/tracing/standard) page documents every field.
# Standard level (/docs/tracing/standard)
`standard` is the production level: it records how each request was
served, never what it said. No prompt text, no output text, no field
that could carry customer content.
## The record [#the-record]
One JSON object per completed request, appended to the trace file (or
exported as one OTLP span named `llm_request`):
| Field | Meaning |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `request_id` | The engine's request id; matches the `id` in the API response, so a slow response can be joined to its trace |
| `t_arrival` | Unix timestamp: request arrived at the engine |
| `t_completion` | Unix timestamp: last token generated |
| `prompt_tokens` | Input size, from the engine's own tokenization |
| `output_tokens` | Tokens generated, reasoning included |
| `e2e_latency_s` | `t_completion` minus `t_arrival`, in seconds |
| `ttft_s` | Always `null` today: first-token time is not visible to the scheduler |
| `queue_wait_s` | Arrival to first-scheduled, in seconds: the congestion signal |
| `scheduler_family` | Which scheduler served it (e.g. `ltr-promptlen`, `fcfs`) |
| `priority_score` | The scheduler's priority for this request; meaningful only for reordering schedulers |
## Example [#example]
A short burst against a chat recipe. Note `queue_wait_s` growing as the
burst saturates the engine's concurrent-sequence ceiling; that pattern,
not any single record, is what capacity planning reads:
```json
{"request_id": "chatcmpl-9f2c41d0", "t_arrival": 1789040112.104, "t_completion": 1789040114.312, "prompt_tokens": 412, "output_tokens": 128, "e2e_latency_s": 2.208, "ttft_s": null, "queue_wait_s": 0.004, "scheduler_family": "ltr-promptlen", "priority_score": -318.0}
{"request_id": "chatcmpl-c07aa1e9", "t_arrival": 1789040112.371, "t_completion": 1789040116.899, "prompt_tokens": 1288, "output_tokens": 256, "e2e_latency_s": 4.528, "ttft_s": null, "queue_wait_s": 0.011, "scheduler_family": "ltr-promptlen", "priority_score": -1104.5}
{"request_id": "chatcmpl-51b8e277", "t_arrival": 1789040112.590, "t_completion": 1789040119.443, "prompt_tokens": 96, "output_tokens": 512, "e2e_latency_s": 6.853, "ttft_s": null, "queue_wait_s": 1.937, "scheduler_family": "ltr-promptlen", "priority_score": -71.2}
```
## The same record as an OTLP span [#the-same-record-as-an-otlp-span]
Exported to a collector, each record is one span named `llm_request`
from service `carbonforge-scheduler`, with the same facts as
attributes (span shown abridged):
```json
{"name": "llm_request", "kind": 1,
"startTimeUnixNano": "1788450022633673216", "endTimeUnixNano": "1788450022799637248",
"attributes": [
{"key": "cf.request_id", "value": {"stringValue": "chatcmpl-b8af6f4d128fdd4c-b31b5b1f"}},
{"key": "gen_ai.usage.prompt_tokens", "value": {"intValue": "54"}},
{"key": "gen_ai.usage.completion_tokens", "value": {"intValue": "2"}},
{"key": "cf.e2e_latency_s", "value": {"doubleValue": 0.1659}},
{"key": "cf.queue_wait_s", "value": {"doubleValue": 0.0119}},
{"key": "cf.scheduler.family", "value": {"stringValue": "trace"}}
]}
```
Span start and end are arrival and completion, so any OTLP-compatible
backend (Tempo, Jaeger, ADOT) renders latency without configuration.
## Reading it [#reading-it]
* **Queueing vs serving**: `e2e_latency_s - queue_wait_s` is time
actually spent in the engine. If `queue_wait_s` dominates, add
capacity or reduce client concurrency; the engine itself is fine.
* **Token accounting**: `output_tokens` counts everything generated,
including reasoning on reasoning models, which is why it can be far
larger than the visible answer.
* **Joining to your logs**: `request_id` equals the `id` field your
client received, so client-side and server-side views of one request
can be lined up exactly.