coding-stack-qwen3.8-27b-huihui

vLLM backend — Huihui Qwen3.8-27B-abliterated (AWQ W4A16)

Self-contained vLLM version of the parent coding-stack-huihui-qwen3.8-27b project. Instead of the Q8 GGUF it serves the pre-quantized INT4 checkpoint shawnw3i/Huihui-Qwen3.8-27B-abliterated-AWQ-MTP (auto-round AWQ, group_size 64, 18.65 GiB on disk) as the model foundation, behind the same OpenAI-compatible proxy that injects a SearXNG web_search tool.

This is the fast backend: vLLM's paged KV cache, chunked prefill and Marlin INT4 kernels are what make the 250k-token context practical on one 48 GB card. It is also the least flexible one — one sequence at a time, no vision, no FP8 on this GPU.

Deliberately not included: the llama.cpp and Ollama compose files, the LibreChat/MongoDB/Meilisearch stack, GGUF/mmproj handling, and the parent's shared container names/API key.

                   ┌─────────────── web_search ──────────────► SearXNG :8889 (external)
                   │
Client ──► :11433 proxy (Flask) ──► :18080 vLLM (OpenAI API)
                   ▲
                   └── :18080 is also published directly, bypassing the proxy

Containers: coding-stack-vllm (vllm/vllm-openai:nightly) and coding-stack-proxy (python:3.11-slim). Port 11433 and the proxy container name clash with the parent and Ollama stacks — stop those first if you switch back and forth.

Files

File	Purpose
docker-compose.yaml	vLLM server + web_search proxy (the whole stack)
proxy-passthrough.py	OpenAI facade on :11433; injects and executes web_search
download-model.sh	Pulls the AWQ checkpoint into the HF cache vLLM reads
benchmark.sh	TTFT + decode tok/s from a streaming request
.gitignore	Keeps caches and weights out of git

There is no Modelfile and no import step: vLLM loads the HF checkpoint directly, and the chat template comes from the checkpoint's chat_template.jinja.

Requirements

• Docker + Docker Compose v2, NVIDIA driver + Container Toolkit
• One whole 48 GB GPU (RTX A6000 here): 18.65 GiB weights + ~15.5 GiB KV at 253952 context + CUDA graphs/activations, at --gpu-memory-utilization 0.93
• Compute capability ≥ 8.9 only for FP8 — the A6000 is 8.6, so weights and KV stay FP16 and INT4 comes from AWQ + Marlin kernels. Don't add --kv-cache-dtype fp8 (or an FP8 checkpoint) on this card: it won't run.
• ~62 GB RAM is comfortable; vLLM reported 55 GiB available while loading 18.65 GiB
• Image: vllm/vllm-openai:nightly (~29 GB, 0.27.2rc1 at the time of writing, 2 months old in this host's cache). It is a moving tag — re-pull deliberately.
• Optional: SearXNG at host.docker.internal:8889 for web_search

The checkpoint is already in this host's cache: /storage-models/vllm/hub/models--shawnw3i--Huihui-Qwen3.8-27B-abliterated-AWQ-MTP (7 shards + model_extra_tensors.safetensors, size-verified in the parent project). download-model.sh is a no-op when it is there.

Quick start

cd vllm
./download-model.sh          # ~19 GB; no-op if the HF cache already has it
docker compose up -d         # first start: load + compile + CUDA graph capture (minutes)
docker compose ps            # wait for (healthy) — start_period is 900 s
./benchmark.sh               # TTFT + tok/s

Cold start is slow by design: vLLM compiles the model's kernels (torch.compile, Triton GDN kernels) and captures CUDA graphs before it reports healthy. Expect several minutes on first boot; subsequent boots reuse the compile cache in the image's /root/.cache only if you persist it (this stack does not — add a volume if you care).

Verify

# Direct to vLLM (OpenAI-compatible)
curl http://localhost:18080/v1/models
curl http://localhost:18080/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# Through the proxy (+ web_search injected)
curl http://localhost:11433/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# Streaming (proxy fast-path: SSE passed straight through)
curl -N http://localhost:11433/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"stream":true,"max_tokens":64}'

curl http://localhost:18080/health

vLLM is started without --api-key, so any Authorization value works (or none). Add --api-key and BACKEND_API_KEY to the proxy if the published port must be closed.

Endpoints

Port	Service	Notes
11433	proxy (Flask)	OpenAI /v1/*; injects web_search; everything else is transparent
18080	vLLM	OpenAI /v1/*, /health, /metrics; no auth

Model name: foundation (--served-model-name), regardless of the HF repo name.

Configuration

vLLM flags (docker-compose.yaml)

Flag	Value	Why
--max-model-len	253952	fits 18.65 GiB weights + ~15.5 GiB f16 KV on 48 GB; native max is 262144
--gpu-memory-utilization	0.93	leaves ~3 GiB for the desktop/other processes
--max-num-seqs	1	one 250k sequence consumes the whole KV pool
--max-num-batched-tokens	16384	chunked prefill granularity for long prompts
--enable-chunked-prefill / --enable-prefix-caching	on	keeps a 250k-token agent prompt responsive and cached across turns
--reasoning-parser	qwen3	splits the reasoning trace out of content
--tool-call-parser	qwen3_coder	required for tool calls (the proxy's web_search)
--enable-auto-tool-choice	on	without it vLLM rejects requests that carry tools
--language-model-only	on	skips the vision tower (see Limitations)
--dtype	float16	the checkpoint's dtype; bf16 would double nothing but adds no benefit here
--trust-remote-code	on	auto-round checkpoint ships custom config

The VLLM_* environment variables in the compose file are the parent project's tuning for this exact card (Marlin atomic add, FlashInfer sampler, graph estimation, spawn workers); they are not required for correctness.

Changing context length

KV is f16 on the 16 full-attention layers only (the 48 GDN layers keep a small recurrent state), so 64 KiB/token:

Context	KV (f16)	Weights	Approx. total
32k	~2.0 GiB	18.65 GiB	~22 GiB
128k	~8.2 GiB	18.65 GiB	~29 GiB
253952 (default)	~15.5 GiB	18.65 GiB	~37 GiB
262144 (native max)	~16 GiB	18.65 GiB	~38 GiB

Lower --max-model-len (and drop VLLM_ALLOW_LONG_MAX_MODEL_LEN if you go below the model's 262144) if you need to share the GPU. Going up is not possible without a second card or a smaller/quantized KV cache — and FP8 KV needs CC ≥ 8.9.

Thinking / reasoning

--reasoning-parser qwen3 moves the trace out of content. The model keeps Qwen3.8's default reasoning level, so it thinks by default and reasoning tokens count against max_tokens — give it 2048–4096 for real questions. Verified on this build (vLLM 0.27.2rc1): the trace arrives in the message's reasoning field (streaming: delta.reasoning), the answer in content, and the split is reported in usage.completion_tokens_details.reasoning_tokens. (Older vLLM releases used reasoning_content; the proxy re-streams either.)

Thinking controls pass through chat_template_kwargs (verified):

# no reasoning at all
curl localhost:18080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hi"}],
       "chat_template_kwargs":{"enable_thinking":false}}'

# shorter trace (named level from the Qwen3.8 template)
  ... "chat_template_kwargs":{"reasoning_effort":"low"}

With default settings a 300- and even a 1000-token reply came back as pure reasoning (finish_reason: "length", empty content) — this checkpoint thinks a lot. Set a high max_tokens or lower the effort.

The proxy does not disable thinking and does not touch the reasoning field — it passes completions through unchanged.

Tools and web_search

--enable-auto-tool-choice --tool-call-parser qwen3_coder is what makes the proxy's injected web_search tool usable. Verified: a get_weather request came back as tool_calls: [{"function":{"name":"get_weather","arguments":"{\"city\": \"Sofia\"}"}}] with finish_reason: "tool_calls". Behaviour is the same as the other backends:

• Non-streaming through :11433: the proxy runs the search loop server-side (MAX_SEARCH_ITER, default 10) against SearXNG and returns only the final answer. The client's own tool calls are returned untouched, unless the model mixes them into a search turn (then they are stubbed — inherited edge case).
• Streaming: passed through natively; if the model calls web_search, the client gets the tool_call and must execute it.
• SEARXNG_URL / SEARXNG_RESULTS tune the backend (defaults http://host.docker.internal:8889, 5). extra_hosts is what resolves that hostname.
• When SearXNG returns nothing, this checkpoint keeps retrying with new queries and the loop ends with an explicit error instead of an answer — verified during testing (SearXNG had all engines rate-limited). The error now reports the counts: Exceeded max search iterations (2 searches, 2 empty) — SearXNG returned no usable results. Lower MAX_SEARCH_ITER (env on the proxy) if you prefer to fail fast.

Pointing a client at it

base_url = http://localhost:11433/v1   # or http://localhost:18080/v1 to skip web_search
model    = foundation
api_key  = anything

Use the proxy for coding agents so web_search works, and set max_tokens high enough for the reasoning trace. Direct vLLM (:18080) is also OpenAI-compatible, including native json_schema structured output — the vLLM build does not need the grammar stripping that llama.cpp did (dropped from this proxy).

Limitations

• No vision. --language-model-only drops the visual tower and vLLM logs running in text-only mode. Remove the flag and lower --max-model-len if you need images; the processor configs are in the checkpoint.
• One sequence at a time. --max-num-seqs 1: a second concurrent request queues.
• No FP8. A6000 is compute capability 8.6; FP8 weights and FP8 KV need 8.9+.
• No auth out of the box (--api-key omitted).
• Nightly image. vllm/vllm-openai:nightly moves; v0.18.0 (also cached on this host) predates qwen3_5 support. Pin the digest if you need reproducibility.
• INT4 is below the Q8 GGUF in fidelity. AWQ keeps mtp, lm_head, the visual merger and some linear_attn.in_proj_a/b tensors in fp16, but the bulk of the weights are 4-bit; the llama.cpp/Ollama stacks with UD-DW-Q8_K_L are the near-lossless option.

Performance

Measured with ./benchmark.sh on this RTX A6000 with the GPU otherwise idle (only ~0.3 GiB of desktop usage), default settings, 253952 context:

Metric	Value
decode (300 tokens)	38.5 tok/s
decode (1000 tokens)	38.3 tok/s
TTFT (92-token prompt, warm)	0.10 s
prefill (9,062-token prompt)	~1.8k tok/s (5.9 s wall, including 32 decoded tokens)
cold start → healthy	~4 min (06:52:28 → 06:56:41, kernels compiled + graphs captured)
VRAM at steady state	42.9 GiB: 17.7 GiB weights/non-torch, 2.74 GiB peak activations, 0.04 GiB CUDA graphs, 23.6 GiB KV pool

Comparison: llama.cpp with the Q8 GGUF measures 9.95 tok/s on the same card, so vLLM INT4 is ~3.9× the decode rate (the parent project hoped for 2–3×). The trade is fidelity: 4-bit AWQ vs near-lossless Q8.

The KV pool is 379,388 tokens (65 KiB/token f16) — --max-model-len 253952 only uses ~67% of it, so the context could be raised to the native 262144; the rest is headroom for one long prompt, not a second sequence (--max-num-seqs 1).

Two implementation details that affect speed on this card (from the vLLM log):

• Using MarlinLinearKernel for AutoAWQMarlinLinearMethod — the INT4 path is Marlin.
• Falling back to the Triton GDN decode path — the fused CUDA GDN kernel requires a BF16 model with K=V=128; this FP16 checkpoint uses the Triton kernel instead.

Troubleshooting

Problem	Fix
ValueError: No available memory for the cache blocks / OOM at startup	lower --max-model-len (e.g. 131072) or --gpu-memory-utilization; free the GPU first (nvidia-smi)
Container stuck in health: starting	it is compiling/capturing graphs — watch docker compose logs -f vllm; allow up to 15 min on first boot
model ... not found / 404	./download-model.sh; confirm HF_HOME=/models and the cache path under /storage-models/vllm/hub
AWQ kernel / quant errors	try --quantization compressed-tensors, or add --enforce-eager (slower but skips CUDA graphs)
Requests queue behind each other	expected with --max-num-seqs 1; raise it only if you lower --max-model-len
Tool calls rejected / not parsed	the parsers are mandatory: --enable-auto-tool-choice --tool-call-parser qwen3_coder
reasoning empty	check --reasoning-parser qwen3 is present; it is what extracts the trace
Empty content, finish_reason: "length"	thinking ate the budget — raise max_tokens or pass chat_template_kwargs: {"enable_thinking": false}
Exceeded max search iterations (N searches, M empty)	SearXNG returned nothing; check its engines, or lower MAX_SEARCH_ITER on the proxy
Proxy returns 502	docker compose logs vllm — usually a failed load; also confirm BACKEND_URL=http://vllm:8000
web_search → Search failed	SearXNG on host.docker.internal:8889 is down (external to this stack)
Startup warning about multimodal limits	harmless: it is the text-only mode you asked for

Reset

docker compose down                      # keeps the HF cache and the image
# nuclear: delete the checkpoint cache (19 GB) and re-download with ./download-model.sh
sudo rm -rf /storage-models/vllm/hub/models--shawnw3i--Huihui-Qwen3.8-27B-abliterated-AWQ-MTP