work / case
Multi-Host GPU Inference Pool
Designed and tuned a pool of mixed GPUs across several machines that serves a stack of self-hosted models — an 80B flagship, a faster mid-size model, and embeddings — fast enough for a whole team to use at once, without paying per-seat cloud bills.
Problem
A company-wide agentic AI platform needed a local model serving backend that could run a large open-weights model — Qwen3-Next-80B, a sparse mixture-of-experts model (~3B active parameters) — at interactive speed for an entire team, not a single user at a time. Commercial API inference would have meant a recurring per-seat bill that scaled linearly with headcount, plus sending internal context to a third party. The available hardware was a constraint in itself: a fleet of consumer and workstation GPUs spread across separate hosts, each with different memory capacity and interconnect characteristics, with no single card large enough to hold the model on its own.
Approach
Built a multi-host inference pool from heterogeneous GPUs, coordinated behind a single OpenAI-compatible gateway (LiteLLM) so callers hit one endpoint and the right backend answered. Each model landed on the hardware that suited it:
- Qwen3-Next-80B-A3B — the flagship 80B mixture-of-experts model, tensor-parallel across a multi-card consumer-GPU server (4× 24 GB) and 4-bit quantised to stay resident in VRAM.
- Qwen3.6-35B-A3B — a faster mid-size model on a 48 GB professional card, carrying higher-volume, lower-latency calls.
- DeepSeek-V4-Flash — a ~153 GB mixed-precision (4-bit-experts) build on an Apple Silicon Mac Studio (M3 Ultra, 512 GB unified memory), served through a compiled llama.cpp build on the Metal backend at ~28 tok/s — leaning on unified-memory bandwidth a discrete card can’t match at that size.
- An embeddings tier — open embedding models (nomic, mxbai, jina, snowflake-arctic) served from Ollama for the platform’s retrieval/RAG.
Engines were matched to workload — vLLM for the large GPU-served chat models, Ollama for the embedding models, and a compiled llama.cpp build (Metal backend) for the Mac. The core tuning was the throughput / latency / concurrency triangle: sizing the KV cache and batching to hold a 256k-token context window per session, scheduling concurrent requests so per-user speed stayed interactive rather than collapsing under load, and right-sizing each host to its role instead of over-provisioning uniformly.
Outcome
The flagship 80B model serves at roughly 120 tokens/sec per user — about 350 tokens/sec aggregate across concurrent sessions — with a 256k-token context window, sustaining 4–5 concurrent users at interactive latency, while the mid-size and embedding models absorb higher-volume traffic. This became the inference backend the company’s agentic AI platform depends on, removing per-seat external API costs entirely and keeping internal context on hardware under our control. The right-sizing exercise turned a collection of mismatched GPUs into a coherent, multi-model serving tier rather than idle, oversized boxes.