Quantization of Neural Networks

Formally, a quantizer maps a real value $x \in \mathbb{R}$ to an integer $q \in \{0, \dots, 2^b - 1\}$ (for $b$-bit unsigned quantization) using an affine (uniform) quantization scheme:

with scale $s > 0$ and zero-point $z \in \mathbb{Z}$. The dequantized approximation is recovered by

When $z = 0$ the scheme is called symmetric (the real range is centered at zero, natural for weights, which are typically roughly symmetric around zero); when $z \neq 0$ it is asymmetric (useful for activations after a ReLU, which are one-sided and non-negative). The pair $(s, z)$ defines a piecewise-constant approximation of the identity function, and choosing them well is the entire game. Two design axes structure the space of methods. The first is granularity: one scale per tensor (per-tensor), one scale per output channel of a weight matrix or convolution filter (per-channel), or one scale per small group of elements (per-group, common in 4-bit LLM quantization). Finer granularity reduces quantization error because it lets each slice of the tensor use a range tailored to its own values, at the cost of storing more scale factors. The second axis is when calibration happens:

• Post-training quantization (PTQ) takes a fully trained floating-point model and computes $(s, z)$ for each tensor using statistics gathered from a small calibration set (a few hundred unlabeled examples), with no gradient updates to the weights. It is fast (minutes) and requires no labeled data or retraining infrastructure.
• Quantization-aware training (QAT) inserts fake-quantization operations into the forward pass during training (or fine-tuning), so the network experiences quantization noise on every mini-batch and its weights adapt to compensate. Gradients pass through the (non-differentiable) rounding operation via a straight-through estimator (STE). QAT is slower — it needs a training loop — but recovers accuracy that PTQ cannot when precision drops very low (4 bits or below) or when the network is unusually sensitive (small models, certain attention layers).

A third, related axis is the arithmetic that quantization targets: weight-only quantization (store weights as low-bit integers, but compute in higher precision after dequantizing on the fly — the dominant mode for memory-bound LLM inference) versus full integer quantization of both weights and activations, which additionally requires the accumulator and requantization steps described below to keep matrix multiplies entirely in integer arithmetic.

Step 1 — Choose the representable range. For a tensor $x$, PTQ observes the empirical minimum and maximum ($\min(x), \max(x)$) over the calibration set, or a tighter range chosen to minimize a distortion metric (mean-squared error or KL-divergence between the original and quantized histograms, as used in TensorRT-style calibration). Clipping the range tighter than the true min/max trades a larger error on outliers for a smaller step size $s$ on the bulk of the distribution — often a net win, since typical weight/activation distributions are heavy in the middle and outlier-diluted in the tails.

Step 2 — Compute scale and zero-point. For symmetric quantization to a signed $b$-bit range $[-2^{b-1}, 2^{b-1}-1]$:

For asymmetric quantization to an unsigned range $[0, 2^b-1]$:

Step 3 — Quantize. Apply the affine map and round-and-clip formula above to every element, producing an integer tensor plus the scalar (or per-channel vector) $(s, z)$.

Step 4 — Requantize the matmul/conv output. If both weights $W$ (scale $s_w$, zero-point $z_w=0$, symmetric) and activations $X$ (scale $s_x$, zero-point $z_x$) are quantized, an integer matmul produces a 32-bit accumulator whose value approximates $\frac{1}{s_w s_x}(W X^\top)$ up to zero-point cross-terms. This accumulator must be requantized — rescaled by $s_w s_x / s_y$ and rounded — to produce the low-bit output activation $y$ for the next layer, where $s_y$ is that next tensor's scale.

Step 5 (QAT only) — Fake-quantize in the forward pass, straight-through in the backward pass. During training, replace each weight/activation tensor $x$ with $\hat{x} = \mathrm{dequantize}(\mathrm{quantize}(x))$ inside the forward computation graph (so the network sees quantization noise and the loss reflects it), but define the backward pass as if $\hat x = x$ inside the clipping range (gradient passes straight through) and zero outside it (clipped saturation blocks gradient flow, since those inputs did not affect the rounded output). ```

# ---------- Post-Training Quantization (PTQ) ---------- function CALIBRATE(model, calib_data, bits, granularity): stats = {} # tensor_name -> (min, max) per group hook all weight tensors and activation outputs for batch in calib_data: # a few hundred examples, no labels needed run model.forward(batch) # hooks record running min/max (or histogram) for tensor_name, values in stats.items(): if granularity == PER_TENSOR: groups = [values] elif granularity == PER_CHANNEL: groups = split_by_output_channel(values) for g in groups: s, z = compute_scale_zero_point(g.min, g.max, bits, symmetric = is_weight(tensor_name)) store(tensor_name, s, z) return quant_params function QUANTIZE_TENSOR(x, s, z, bits, signed): lo, hi = quant_range(bits, signed) # e.g. (-128, 127) for int8 signed q = round(x / s + z) return clip(q, lo, hi) function DEQUANTIZE_TENSOR(q, s, z): return s * (q - z) function PTQ_DEPLOY(model, quant_params, bits): for layer in model.layers: W_q = QUANTIZE_TENSOR(layer.weight, quant_params[layer].s_w, 0, bits, signed=True) store W_q instead of layer.weight # weights quantized once, offline return IntegerInferenceModel(model, quant_params)

# ---------- Quantization-Aware Training (QAT) ---------- function FAKE_QUANTIZE(x, s, z, bits, signed): lo, hi = quant_range(bits, signed) q = clip(round(x / s + z), lo, hi) # forward: real rounding + clipping return s * (q - z) # dequantized value, same shape as x

# custom gradient (straight-through estimator):

# d(FAKE_QUANTIZE(x)) / dx = 1 if lo <= round(x/s + z) <= hi (inside range)

# = 0 otherwise (saturated -> blocks gradient) function QAT_TRAIN(model, train_data, bits, epochs): insert FAKE_QUANTIZE nodes after every weight tensor and activation tensor initialize scales s from a short calibration pass (as in PTQ) for epoch in 1..epochs: for batch in train_data: x_q = FAKE_QUANTIZE(activations, s_x, z_x, bits, ...) W_q = FAKE_QUANTIZE(weights, s_w, 0, bits, signed=True) y = layer_op(x_q, W_q) # conv / matmul in float, on fake-quantized values loss = criterion(model.forward_rest(y), labels) grads = backward(loss) # STE makes this well-defined update(weights, grads) # weights updated in full precision periodically re-estimate s from batch statistics (EMA of min/max) return model # deploy by "freezing" final scales and truly quantizing, as in PTQ_DEPLOY ``` The two algorithms share the same mathematical machinery (`QUANTIZE_TENSOR`/`DEQUANTIZE_TENSOR`, scale/zero-point computation) but differ in when the network gets to react to the rounding error: PTQ computes the map once, after the fact; QAT lets gradient descent absorb the error into the weights themselves.

Memory. A tensor with $N$ elements stored in $b$ bits instead of 32 bits occupies $N b / 8$ bytes instead of $4N$ bytes — a factor $32/b$ reduction (8× for int4, 4× for int8), plus a small constant overhead for the per-channel or per-group scale/zero-point vectors, which is $O(N/g)$ for group size $g$ and negligible when $g \gg 1$. For weight-only quantization of a model with $P$ parameters, storage drops from $4P$ bytes to $bP/8 + O(P/g)$ bytes, which is why 4-bit quantization of a 70-billion-parameter model shrinks it from roughly 280 GB to roughly 35–40 GB — the difference between "needs a multi-GPU server" and "fits on a single high-end GPU."

Time / throughput. Quantization does not reduce the number of multiply-accumulate (MAC) operations — a quantized convolution performs exactly the same $O(\text{FLOPs})$ arithmetic operations as its float counterpart. The speedup instead comes from two sources: (1) integer MACs are cheaper in silicon (smaller multiplier circuits, higher throughput per cycle on hardware with dedicated int8/int4 tensor cores) and (2) especially for memory-bound inference (small batch sizes, autoregressive decoding), the bottleneck is bytes moved from memory to compute, not FLOPs, so a 4× reduction in weight bytes can yield close to a 4× reduction in wall-clock latency even though FLOP count is unchanged. This distinguishes quantization sharply from pruning, which does reduce FLOPs.

Correctness / error analysis. The per-element quantization error of a uniform quantizer with step size $s$ satisfies $|x - \hat x| \le s/2$ (rounding to nearest introduces at most half a step of error), so the error is bounded by

This bound halves for every additional bit — the reason 8-bit quantization is nearly lossless for most vision/language models while 2-bit quantization is not: at $b=8$ over a typical weight range of $[-1,1]$, $\varepsilon \approx 1/510 \approx 0.002$, small relative to typical weight magnitudes; at $b=2$, $\varepsilon \approx 1/6 \approx 0.17$, comparable to the weight values themselves. Errors from independent quantization of many weights partially cancel in a dot product (they behave like zero-mean noise under mild assumptions), so a layer's output error grows roughly as $O(\sqrt{n}\, \varepsilon)$ for $n$-dimensional inner products rather than $O(n\varepsilon)$ — this is the informal argument (a quantization-noise-as-additive-noise model, standard in signal processing) for why quantized networks with hundreds of input features per layer degrade gracefully rather than catastrophically at moderate bit-widths.

Why per-channel/per-group scales help — an invariant argument. Consider a weight matrix where different output channels have very different dynamic ranges (channel A's weights lie in $[-0.05, 0.05]$, channel B's in $[-2, 2]$). A single per-tensor scale must cover the union of ranges, so $s$ is dictated by channel B, and channel A's small weights are quantized with a step size 40× too coarse — many of them round to exactly $0$, destroying that channel's information. Per-channel scaling restores the invariant that each channel's local quantization error is bounded by its own local step size, decoupling channels that happen to share a tensor but not a distribution. This is why per-channel (or per-group) weight quantization is close to universal in practice, while per-tensor is only used when hardware constraints force it.

Why QAT recovers accuracy that PTQ cannot. PTQ treats quantization error as a fixed perturbation applied after optimization has already found a good floating-point minimum; there is no guarantee that this minimum is robust to that perturbation. QAT instead optimizes

directly, where $\mathrm{FQ}$ is the fake-quantization operator — so gradient descent seeks a minimum that is good given the rounding noise, effectively regularizing weights toward being robust to their own discretization. The straight-through estimator is a biased gradient estimator (it is not the true gradient of the piecewise-constant `round`, which is zero almost everywhere and undefined at the jumps), but empirically it is a sufficiently good direction for optimization — the same phenomenon that makes STE work for binarized/ternary networks and for training with hard-threshold activations.

Tradeoffs, stated honestly. PTQ is cheap (no retraining, minutes of calibration) but can lose several accuracy points at 4 bits or below, and is fragile on models with large activation outliers (some transformer layers). QAT recovers most of that loss but costs a full (or partial) retraining run, needs labeled data, and adds engineering complexity (fake-quant nodes, STE, scale re-estimation schedules). Neither method reduces FLOPs; both are complementary to pruning and distillation, not substitutes for them.

Consider a single output channel of a fully-connected layer with 5 observed float32 weights, to be quantized to signed int8 ($b = 8$, range $[-128, 127]$), symmetric per-channel:

Step 1 — range. $|w|_{\max} = \max(2.35, 2.10) = 2.35$.

Step 2 — scale (symmetric, signed 8-bit uses $2^{b-1}-1 = 127$):

Step 3 — quantize each value, $q = \mathrm{round}(w/s)$, clipped to $[-128,127]$:

Step 4 — dequantize, $\hat w = s \cdot q$:

Every error is at most $s/2 = 0.00925$, exactly matching the theoretical bound derived above; the largest error (0.0090) occurs for the value with the largest fractional rounding residue ($113.49 \to 113$), as expected.

Now suppose a second channel in the same weight matrix has values $[-0.06, 0.03, 0.01, -0.02, 0.05]$. If forced to share the first channel's per-tensor scale $s = 0.0185$, every one of these values maps to $q \in \{-3, 2, 1, -1, 3\}$ — only 3–4 discrete levels are used to represent a channel whose values span barely 2% of the encoded range, and the relative error on values like $0.01 \to q=1 \to \hat w = 0.0185$ is 85%. Computing a separate per-channel scale $s_2 = 0.06/127 \approx 0.000472$ for this channel instead gives $q \in \{-127, 64, 21, -42, 106\}$, spreading the channel across nearly the full int8 range and cutting its relative error by roughly 40×. This is the concrete mechanism behind the abstract claim in the complexity section: per-channel scaling is not a minor refinement, it is often the difference between a quantized model that matches float accuracy and one that does not.

Quantization as a mathematical idea predates deep learning by decades: it is the same operation performed by every analog-to-digital converter, and its error analysis originates in classical information theory and digital signal processing. Claude Shannon's rate-distortion theory formalized the fundamental tradeoff between the number of bits used to represent a signal and the distortion incurred — the exact tradeoff neural network quantization exploits, decades later, for weights and activations instead of audio or image signals.

Every major mobile inference framework — TensorFlow Lite, PyTorch Mobile/ExecuTorch, Core ML, ONNX Runtime — ships built-in int8 post-training and quantization-aware training pipelines, because on-device vision and speech models must fit in the memory and power budget of a phone or wearable. Data-center inference servers (NVIDIA TensorRT, Triton) use int8 and, on newer hardware, fp8 arithmetic to double or quadruple throughput per GPU for latency-sensitive services like search ranking and ad prediction. In the large-language-model era, weight-only 4-bit quantization methods (GPTQ, AWQ, bitsandbytes' NF4) are the enabling technology that lets 7B–70B parameter models run on a single consumer or prosumer GPU instead of requiring a multi-GPU cluster, directly determining which organizations can afford to serve or even experiment with these models. Speech recognition on embedded devices, autonomous-vehicle perception stacks under strict latency budgets, and federated-learning clients on low-power hardware all depend on the same affine quantization machinery described here. Quantization is also a natural complement to Model Compression and Knowledge Distillation and to pruning: a production pipeline commonly distills a large teacher into a smaller student, prunes redundant structure, and finally quantizes the result, stacking three largely orthogonal compression axes (parameters, sparsity, precision) into a single deployable artifact — a workflow directly connected to broader concerns in numerical precision and hardware-aware model design.

Ask the tutor: How would you decide, for a specific model and hardware target, whether 8-bit PTQ suffices or whether you need QAT at 4 bits?