Inference Serving Systems

Inference serving systems are the software and hardware infrastructure responsible for deploying trained machine learning models and responding to prediction requests at production scale, managing latency, throughput, and resource utilization simultaneously. They matter because a model that cannot serve millions of requests per second with sub-100ms latency has no real-world value, regardless of its validation accuracy. The counterintuitive fact: the engineering complexity of inference serving often exceeds that of training — a model trains once, but serves billions of times under wildly unpredictable load.

The core insight

Training a model and serving a model are fundamentally different problems. Training optimizes for final accuracy over many hours on a fixed dataset; serving optimizes for latency, throughput, and cost per query under variable, bursty, real-world traffic — often with contradictory constraints. The "aha moment" is realizing that every design decision in a serving system — batching, caching, quantization, scheduling — is a deliberate trade-off on the latency–throughput–cost trilemma, and there is no universally correct answer, only the right answer for your specific service-level objective (SLO).

Common trapCommon Misconception

The most common mistake is conflating model optimization with serving system optimization and treating them as the same problem. Practitioners often apply quantization or pruning and assume the job is done — then ship the model behind a naive single-threaded REST endpoint. The correct mental model is a two-layer stack: the model layer (quantization, distillation, operator fusion) reduces the arithmetic cost of a single forward pass, while the system layer (batching, async scheduling, multi-instance serving, KV-cache management) determines how efficiently that pass is amortized over concurrent requests. A highly quantized model behind a poor serving system will be slower in production than a full-precision model behind a well-engineered one. Both layers must be co-optimized.

How It Works

The Latency–Throughput Trade-off

At the heart of every inference server is a scheduling loop that decides when to execute a batch of requests. Let $B$ be the batch size, $T_{\text{compute}}(B)$ be the time to run one forward pass on a batch of size $B$ , and $T_{\text{wait}}$ be the time a single request spends waiting to be included in a batch.

\quad \text{Throughput} = \frac{B}{T_{\text{compute}}(B) + T_{\text{wait}}}

Explanation: Throughput is the number of requests completed per unit time; increasing batch size $B$ amortizes fixed overhead but increases $T_{\text{wait}}$ for early arrivals.

$B$ — batch size (number of concurrent requests processed together)
$T_{\text{compute}}(B)$ — GPU kernel execution time, which grows sub-linearly with $B$ due to parallelism
$T_{\text{wait}}$ — queuing latency before a request is dispatched

For transformer models, $T_{\text{compute}}(B)$ is dominated by matrix multiplications. Given weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and an input batch $X \in \mathbb{R}^{B \times d_{\text{in}}}$ :

\quad Y = X W^{\top}, \quad \text{FLOPs} = 2 \cdot B \cdot d_{\text{in}} \cdot d_{\text{out}}

Explanation: Doubling batch size doubles the FLOPs linearly, but memory-bandwidth-bound operations amortize the weight-load cost across more requests, so wall-clock time grows much slower than linearly.

$Y \in \mathbb{R}^{B \times d_{\text{out}}}$ — output activations
FLOPs — floating-point operations; the factor of 2 accounts for multiply-accumulate (MAC) operations

KV-Cache and Memory Footprint

Autoregressive transformer serving introduces a unique cost: the key-value cache. During generation of a sequence of length $L$ with $H$ attention heads, head dimension $d_h$ , and $N$ layers, the KV-cache memory requirement is:

\quad M_{\text{KV}} = 2 \cdot N \cdot H \cdot d_h \cdot L \cdot B \cdot \text{sizeof}(\text{dtype})

Explanation: The factor of 2 accounts for both keys and values; this grows linearly with sequence length $L$ and batch size $B$ , quickly exhausting GPU VRAM for long contexts.

$N$ — number of transformer layers
$H$ — number of attention heads
$d_h = d_{\text{model}} / H$ — per-head dimension
$L$ — current sequence length (grows token by token during generation)
$\text{sizeof}(\text{dtype})$ — 2 bytes for FP16, 1 byte for INT8

For a typical 70B parameter model (e.g., Llama-3-70B) with $N=80$ , $H=64$ , $d_h=128$ , serving a batch of $B=16$ requests with $L=4096$ tokens in FP16:

\quad M_{\text{KV}} = 2 \times 80 \times 64 \times 128 \times 4096 \times 16 \times 2 \approx 107 \text{ GB}

Explanation: This single calculation reveals why KV-cache management — through techniques like PagedAttention — is the dominant memory challenge in LLM serving, often exceeding the model weights themselves.

Goodput and SLO Attainment

A production serving system is judged not by raw throughput but by goodput — the fraction of requests completed within the latency SLO:

\quad \text{Goodput} = \text{Throughput} \times P(\text{latency} \leq \text{SLO})

Explanation: A system that processes 10,000 requests/second but violates the 200ms SLO for 30% of them delivers only 7,000 useful completions per second — the SLO attainment rate is the true production metric.

$P(\text{latency} \leq \text{SLO})$ — empirical fraction of requests meeting the latency budget
This motivates tail-latency optimization, not just mean-latency optimization

researcher · Ion Stoica | b. 1970 | PagedAttention and vLLM

Ion Stoica is a professor at UC Berkeley and co-founder of Anyscale and Databricks, with foundational contributions to distributed systems including Apache Spark. He led the Berkeley Sky Computing Lab team that developed vLLM and the PagedAttention algorithm, which manages KV-cache memory using virtual memory paging principles borrowed from operating systems, enabling 24× higher throughput than naive implementations. Remarkably, his most influential AI systems work came after he was already famous in the distributed-systems community — a rare cross-domain leap that reshaped LLM deployment.

Figure

Inference Serving System Architecture

researcher · Woosuk Kwon | b. 1993 | PagedAttention Algorithm

Woosuk Kwon is a PhD student at UC Berkeley (now faculty at University of Washington) who, as the lead author of the vLLM paper (2023), designed the PagedAttention algorithm that revolutionized LLM serving. PagedAttention treats the KV-cache as a set of non-contiguous fixed-size memory blocks — analogous to virtual memory pages — eliminating fragmentation and enabling near-zero memory waste during dynamic generation. Kwon was completing his PhD when vLLM became one of the fastest-adopted open-source inference systems in history, reaching millions of daily users within months of release.

Figure

Continuous Batching vs Static Batching

Intuition

The Mental Model

Think of an inference server as a restaurant kitchen, not an assembly line. In static batching (assembly-line thinking), the kitchen waits until exactly four orders arrive, cooks them all together, then plates everything simultaneously — even if one dish was done ten minutes ago. The early diners sit waiting while their food gets cold, and new customers can't even order until the batch finishes. Continuous batching is the smart kitchen: the moment a burner frees up, a new dish goes on it, regardless of what else is cooking. Requests come and go at token-generation frequency, not at request boundary frequency. The KV-cache is your kitchen's mise en place — pre-chopped ingredients (pre-computed attention keys and values) that make each new step fast. But the mise en place takes up counter space, and when the counter is full, you have to decide which prep to throw away or temporarily store elsewhere (offloading to CPU RAM). The scheduler is the head chef, constantly deciding who gets a burner, who waits, and which prep to keep on the counter — with the goal of keeping every burner running at 100% without burning anyone's order.

Historical Development

Origins · 1 minhow this was discovered

History

2015

TensorFlow Serving released by Google Brain — the first production-grade model serving framework, introducing the concepts of versioned model management and gRPC-based prediction APIs.

2017

NVIDIA TensorRT 1.0 released — introduced graph optimization, layer fusion, and mixed-precision inference, establishing the pattern of ahead-of-time (AOT) compilation for neural networks.

2019

ONNX Runtime reaches production maturity — cross-framework inference graph with hardware-specific execution providers, enabling deployment-time model portability across CPU, GPU, and custom accelerators.

2020

Triton Inference Server (NVIDIA) open-sourced — pioneered model ensembles, dynamic batching, and multi-model concurrent execution on shared GPU resources.

2022

Orca paper (Yu et al., OSDI 2022) — introduced iteration-level scheduling (continuous batching) for LLMs, demonstrating 36× throughput improvement over static batching, fundamentally changing how autoregressive models are served.

2023

vLLM / PagedAttention released (Kwon et al., SOSP 2023) — virtual-memory-inspired KV-cache management eliminated fragmentation and enabled 24× higher effective throughput, becoming the dominant open-source LLM serving engine.

2023

Speculative decoding popularized (Leviathan et al., Miao et al.) — using a small draft model to propose tokens and a large model to verify in parallel, reducing per-token latency by 2–3× without changing output distribution.

2024

Disaggregated prefill/decode serving (Zhong et al., "DistServe") — separating the compute-intensive prefill phase from the memory-bandwidth-intensive decode phase onto different hardware, independently optimizing both SLOs.

2024

Chunked prefill and prefix caching become standard — systems like SGLang and vLLM add structured generation, multi-turn prefix reuse, and radix tree caching, making complex agent workflows economically viable.

Why It Matters

Real-World Impact

ChatGPT / GPT-4 (OpenAI)Uses a custom serving stack with continuous batching, speculative decoding, and tensor-parallel sharding across hundreds of A100s; the latency SLO of ~50 tokens/second perceived streaming is achieved by carefully overlapping prefill computation with decode phases across request cohorts.

Google Search AI OverviewsDeploys PaLM/Gemini behind a serving system that must handle millions of QPS with strict P99 latency budgets; uses speculative decoding with a smaller draft model and prefix caching for common query templates to reduce cost by an estimated 40–60%.

Llama.cppA CPU-inference serving framework enabling quantized (GGUF/GGML) LLMs on consumer hardware; uses Q4_K_M quantization to reduce a 7B model from 14GB to ~4GB with <1% perplexity loss, enabling laptop-speed inference — demonstrating that model compression and serving are inseparable.

vLLM Production Deployments (Anyscale, various enterprises)PagedAttention-based serving routinely achieves 3–8× higher throughput than naive HuggingFace `generate()` calls at the same GPU budget; used in production by major cloud providers as the backend for their LLM API endpoints.

TGI (HuggingFace Text Generation Inference)Powers the HuggingFace Inference API; implements Flash Attention 2, continuous batching, tensor parallelism, and GPTQ/AWQ quantization, serving hundreds of open models to millions of developers with < 200ms TTFT (Time to First Token).

Triton Inference Server (NVIDIA) in Medical ImagingDeployed at scale for real-time radiology AI inference; its model ensemble feature lets a preprocessing model, a segmentation model, and a post-processing model run as a pipeline with sub-50ms end-to-end latency on a single GPU server in clinical environments.

Amazon Bedrock / SageMakerUses custom routing layers that perform model-request affinity routing (sending similar prompts to the same replica to hit the prefix cache), reducing compute cost for repeated system-prompt tokens by up to 90% for high-volume enterprise customers.