INT4 and GPTQ Quantization

INT4 and GPTQ quantization are techniques that compress neural network weights from high-precision floating-point representations (FP16/FP32) down to 4-bit integers, reducing model memory footprint by 4–8× while preserving most of the original model's accuracy. They matter enormously in modern AI because they make billion-parameter models deployable on consumer hardware — running a 70B-parameter LLM on a single GPU rather than a multi-GPU cluster. The counterintuitive fact: GPTQ achieves near-lossless compression at 4 bits by exploiting the second-order curvature of the loss landscape, meaning it uses information about how weights interact with each other, not just their individual magnitudes.

The core insight

Every weight in a neural network is stored as a 16- or 32-bit float, but the model's behavior is determined by how weights collectively transform activations — not by any single weight's exact value. GPTQ's key insight is that you can aggressively quantize individual weights as long as you immediately compensate by adjusting neighboring weights using the inverse Hessian of the layer's output error, making the layerwise reconstruction error nearly zero. This is fundamentally different from naive rounding: instead of minimizing each weight's absolute error independently, GPTQ minimizes the output error of the entire weight matrix, column by column, propagating corrections forward. The "aha moment" is realizing that a weight quantized badly can be perfectly compensated by its neighbors — quantization is a distributed error-correction problem, not a per-weight rounding problem.

Common trapCommon Misconception

The most common mistake practitioners make is assuming that INT4 quantization simply rounds each weight to the nearest 4-bit value independently — and therefore that it must degrade accuracy roughly in proportion to the rounding error per weight. This is wrong for GPTQ specifically: the algorithm uses the Hessian (computed from calibration data activations) to propagate quantization error across weights in a single row, actively canceling errors rather than accumulating them. The correct mental model is that GPTQ is performing a structured least-squares optimization over each row of the weight matrix, where quantizing one weight updates all subsequent weights in that row to compensate. Treating GPTQ output as "just rounded weights" will cause you to misunderstand why calibration data quality matters so much and why column ordering (via the Cholesky-based permutation) is critical to its success.

How It Works

Uniform INT4 Quantization

The foundation is mapping a continuous range of floating-point values into 16 discrete integer levels (since $2^4 = 16$ ). For a weight tensor $\mathbf{W}$ , we define a scale factor $s$ and optionally a zero-point $z$ :

\quad \hat{w} = s \cdot (q - z), \quad q = \text{clamp}\!\left(\left\lfloor \frac{w}{s} + z \right\rceil,\ 0,\ 2^b - 1\right)

Explanation: Each floating-point weight $w$ is mapped to a $b$ -bit integer $q$ by dividing by scale $s$ , shifting by zero-point $z$ , and rounding — then dequantized back to $\hat{w}$ for computation.

Symbol definitions:

$w$ — original FP16/FP32 weight value
$s$ — scale factor, computed as $(\max(w) - \min(w)) / (2^b - 1)$
$z$ — zero-point integer offset (0 for symmetric quantization)
$q$ — quantized integer in range $[0, 2^b - 1]$
$b$ — bit width (4 for INT4)
$\hat{w}$ — dequantized (reconstructed) weight

For symmetric quantization (common in practice), $z = 0$ and:

\quad s = \frac{\max(|\mathbf{W}|)}{2^{b-1} - 1}, \quad q = \text{clamp}\!\left(\left\lfloor \frac{w}{s} \right\rceil,\ -(2^{b-1}),\ 2^{b-1}-1\right)

Explanation: Symmetric quantization centers the integer grid at zero, using signed integers, which is preferred for weights that are roughly zero-mean.

The Quantization Error and Layerwise Objective

For a linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{d_{out} \times d_{in}}$ and input activations $\mathbf{X} \in \mathbb{R}^{d_{in} \times N}$ (from calibration data), quantization error in the output is:

\quad \mathcal{L}(\hat{\mathbf{W}}) = \|\mathbf{W}\mathbf{X} - \hat{\mathbf{W}}\mathbf{X}\|_F^2 = \|(\mathbf{W} - \hat{\mathbf{W}})\mathbf{X}\|_F^2

Explanation: GPTQ doesn't minimize weight error directly — it minimizes the Frobenius norm of the output difference, which measures how much the layer's actual predictions change after quantization. The second-order Taylor expansion of this loss around the original weights gives the Hessian:

\quad \mathbf{H} = 2\,\mathbf{X}\mathbf{X}^\top \in \mathbb{R}^{d_{in} \times d_{in}}

Explanation: The Hessian of the layerwise output error with respect to the weights is proportional to the input covariance matrix — meaning weights activated by high-variance inputs matter more and must be quantized more carefully.

The GPTQ Update Rule

After quantizing weight $w_q$ (the $q$ -th column of a row), the optimal correction to all remaining unquantized weights in that row is:

\quad \boldsymbol{\delta}_F = -\frac{w_q - \hat{w}_q}{[\mathbf{H}_F^{-1}]_{qq}} \cdot (\mathbf{H}_F^{-1})_{:,q}

Explanation: The error from quantizing column $q$ is "distributed" to all remaining columns proportionally to their Hessian cross-correlation with column $q$ , minimizing the downstream output error in a single Newton step.

Symbol definitions:

$w_q$ — original weight at column $q$
$\hat{w}_q$ — quantized weight at column $q$
$\mathbf{H}_F$ — sub-Hessian restricted to the remaining (not-yet-quantized) columns $F$
$[\mathbf{H}_F^{-1}]_{qq}$ — the $(q,q)$ diagonal element of the inverse Hessian (scalar)
$(\mathbf{H}_F^{-1})_{:,q}$ — the $q$ -th column of the inverse Hessian (vector of corrections)
$\boldsymbol{\delta}_F$ — correction vector added to all remaining unquantized weights

In practice, GPTQ uses a Cholesky decomposition of $\mathbf{H}^{-1}$ to compute all these updates efficiently in $O(d_{in}^3)$ time once, then applies them in a sequential column sweep.

Group Quantization

To improve accuracy, INT4 is almost always applied with grouped quantization: weights are divided into groups of $g$ consecutive elements (e.g., $g = 128$ ), and each group gets its own scale and zero-point: ```python

# PyTorch pseudocode: grouped INT4 quantization def quantize_grouped(W: torch.Tensor, group_size: int = 128, bits: int = 4): d_out, d_in = W.shape W_grouped = W.view(d_out, d_in // group_size, group_size)

# Compute per-group scale and zero-point w_max = W_grouped.max(dim=-1, keepdim=True).values w_min = W_grouped.min(dim=-1, keepdim=True).values scale = (w_max - w_min) / (2bits - 1) # shape: [d_out, d_in//g, 1] zero = (-w_min / scale).round().clamp(0, 2bits - 1)

# Quantize W_int = ((W_grouped / scale) + zero).round() W_int = W_int.clamp(0, 2**bits - 1).to(torch.uint8) return W_int, scale, zero # store scale/zero in FP16 for dequant ```

GPTQ Algorithm in PyTorch-Style Pseudocode

```python def gptq_quantize(W: torch.Tensor, H: torch.Tensor, bits: int = 4, blocksize: int = 128): """ W: weight matrix [d_out, d_in] H: Hessian 2*X@X.T [d_in, d_in] from calibration data """ d_out, d_in = W.shape W = W.clone().float()

# Add damping for numerical stability, then invert via Cholesky H += 1e-2 * torch.eye(d_in, device=W.device) H_inv = torch.linalg.inv(H) # in practice: Cholesky for efficiency Q = torch.zeros_like(W) # quantized weights output E = torch.zeros_like(W) # per-column quantization errors for col in range(0, d_in, blocksize): W_block = W[:, col:col+blocksize] H_inv_block = H_inv[col:col+blocksize, col:col+blocksize] for j in range(blocksize): w = W_block[:, j] # current column [d_out] d = H_inv_block[j, j] # diagonal Hessian inv element

# Quantize this column q = quantize(w, bits=bits) # round to nearest INT4 Q[:, col+j] = q

# Compute error and propagate to remaining columns in block err = (w - q) / d W_block[:, j:] -= err.unsqueeze(1) * H_inv_block[j, j:].unsqueeze(0) E[:, col+j] = err

# Propagate block error to all remaining columns W[:, col+blocksize:] -= E[:, col:col+blocksize] @ H_inv[col:col+blocksize, col+blocksize:] return Q ```

researcher · Elias Frantar | b. 1997 | Lead Author of GPTQ

Elias Frantar is a PhD student at IST Austria (Institute of Science and Technology Austria) working under Dan Alistarh, focusing on efficient deep learning and neural network compression. He is the primary author of the 2022 GPTQ paper, which adapted the Optimal Brain Surgeon framework to work on billion-parameter language models for the first time by exploiting block-diagonal Hessian structure and Cholesky-based inversion, reducing quantization time from weeks to hours. Remarkably, the original GPTQ paper quantized GPT-J (6B parameters) in approximately 4 GPU-hours on a single A100, making large-model quantization practical for ordinary researchers for the first time.

researcher · Babak Hassibi | b. 1967 | Optimal Brain Surgeon — Theoretical Foundation of GPTQ

Babak Hassibi is a professor of Electrical Engineering at Caltech, known for his work in information theory, signal processing, and neural network compression. His 1993 paper with David Stork, "Second Order Derivatives for Network Pruning: Optimal Brain Surgeon," introduced the Hessian-based weight deletion framework that GPTQ directly builds upon — the key idea that removing or quantizing a weight should be compensated by updating all other weights via the inverse Hessian. The surprising historical fact is that this foundational theory was developed in the early 1990s for tiny networks with thousands of parameters, and lay largely dormant for three decades before GPTQ showed it could scale to models with tens of billions of parameters.

Figure

GPTQ Quantization Data Flow

Figure

INT4 vs FP16 Numerical Grid