Efficiency: Quantization, LoRA & Distillation

Quantization maps a high-precision numeric representation (typically 16- or 32-bit floating point) to a lower-precision representation (typically 8-bit or 4-bit integers, or specialized float formats) for weights, and sometimes activations. Formally, a real-valued tensor $W$ is mapped to a quantized tensor $\hat{W}$ via a scale $s$ and zero-point $z$:

The scale $s$ is chosen per-tensor, per-channel, or per-block to minimize quantization error given the dynamic range of the values. Post-training quantization (PTQ) quantizes an already-trained model, sometimes using a small calibration set to choose scales well. Quantization-aware training (QAT) simulates quantization during training so the model adapts to the rounding noise.

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method. Instead of updating a pretrained weight matrix $W_0 \in \mathbb{R}^{d\times k}$ directly, LoRA freezes $W_0$ and learns a low-rank correction:

Only $A$ and $B$ are trained; $W_0$ never receives gradients. This is a direct descendant of the instruction-tuning setup, where a pretrained base model is adapted to follow instructions — LoRA asks whether that adaptation needs to touch every parameter, and answers no.

Knowledge distillation trains a smaller student network $S$ to approximate a larger teacher network $T$, typically by matching the teacher's output probability distribution (soft labels) rather than only the ground-truth hard labels. The intuition: a teacher's full softmax output over incorrect classes encodes relative similarities ("dark knowledge") that a one-hot label discards, and this richer signal is easier for a small model to learn from.

Quantization mechanics

The central operation is affine (or symmetric) mapping between float and integer ranges. For a tensor with values in $[\alpha, \beta]$ and a target $b$-bit signed integer range $[-2^{b-1}, 2^{b-1}-1]$:

Symmetric quantization (common for weights, since weight distributions are roughly zero-centered) sets $z=0$ and $s = \max(|\alpha|,|\beta|) / (2^{b-1}-1)$. Because a single scale for an entire weight matrix wastes precision when a few outlier values dominate the range, practical schemes use block-wise / group-wise quantization: partition each row (or a fixed number of consecutive weights) into blocks and store one scale per block. This is the key idea behind formats like NF4 (NormalFloat4, used in QLoRA), which additionally chooses quantization levels matched to the empirical (roughly Gaussian) distribution of pretrained weights rather than uniform spacing. ``` function QUANTIZE_TENSOR(W, bits, block_size):

# W: real-valued tensor, flattened n = length(W) num_blocks = ceil(n / block_size) scales = array of size num_blocks W_q = integer array of size n for i in 0 .. num_blocks - 1: block = W[iblock_size : min((i+1)block_size, n)] s = max(abs(block)) / (2^(bits-1) - 1) # symmetric scale scales[i] = s for each element w at position j in block: W_q[j] = round(w / s) # clamp to representable range W_q[j] = clamp(W_q[j], -2^(bits-1), 2^(bits-1) - 1) return W_q, scales function DEQUANTIZE_TENSOR(W_q, scales, block_size): W_hat = array of size length(W_q) for i in 0 .. num_blocks - 1: s = scales[i] for each element index j in block i: W_hat[j] = W_q[j] * s return W_hat ``` At inference, a quantized matmul either dequantizes weights on the fly (memory-bandwidth bound: read INT4, expand to FP16, multiply) or uses native low-precision matrix multiply units. The dominant real-world win is memory bandwidth, not FLOPs: LLM inference is largely bandwidth-bound because generation is memory-bound (one token at a time streams the whole weight matrix through the compute units), so shrinking bytes-per-weight directly shrinks latency.

LoRA mechanics

For each targeted linear layer (commonly the query/key/value/output projections in attention, sometimes the MLP projections), replace the forward pass:

where $\alpha$ is a fixed scaling hyperparameter and $r$ is the rank. $A$ is typically initialized with small random Gaussian values and $B$ initialized to zero, so that at the start of training $\Delta W = BA = 0$ and the adapted model exactly reproduces the pretrained model — training begins from a known-good point. ``` function LORA_FORWARD(x, W0, A, B, alpha, r):

# W0 is frozen; A, B are the only trainable parameters for this layer base_output = W0 @ x adapter_output = (alpha / r) * (B @ (A @ x)) return base_output + adapter_output function LORA_TRAIN_STEP(batch, model, optimizer): for each linear layer L in model.targeted_layers: freeze(L.W0) # no gradient computed or stored loss = compute_loss(model.forward(batch), batch.labels) loss.backward() # gradients flow only into A, B matrices optimizer.step() # update only A, B optimizer.zero_grad() ``` At the end of training, the adapter can either stay separate (multiple LoRA adapters can be swapped in and out of the same frozen base model, enabling multi-tenant serving) or be merged: $W = W_0 + \frac{\alpha}{r}BA$, producing a single dense matrix with zero extra inference cost.

QLoRA combines the two: the frozen backbone $W_0$ is stored in 4-bit NF4 quantized form (dequantized on the fly during the forward/backward pass), while the trainable $A,B$ matrices remain in a higher-precision format (e.g., bfloat16), and gradients are computed through the quantized weights using straight-through estimation for the rounding operation combined with paged optimizers that offload optimizer state to CPU memory during activation spikes.

Distillation mechanics

Given a trained teacher $T$ producing logits $z_T$ and a student $S$ producing logits $z_S$ over the same output vocabulary, define a temperature-softened softmax:

Higher temperature $\tau > 1$ flattens the distribution, exposing relative probabilities among non-target classes that would otherwise be near-zero. The distillation loss combines a soft-label term and (optionally) a hard-label term:

The $\tau^2$ factor rescales gradients so their magnitude is comparable across temperature settings (the gradient of the soft cross-entropy term with respect to logits scales as $1/\tau^2$, which this factor cancels). For sequence models, distillation can be applied at the token level (matching next-token distributions at every position — the standard recipe), at the sequence level (training the student to maximize the probability of teacher-generated outputs), or via on-policy distillation, where the student generates its own rollouts and is corrected using teacher feedback on those very rollouts, reducing the mismatch between training and deployment-time input distributions. ``` function DISTILLATION_TRAIN_STEP(batch, teacher, student, tau, lambda): with no_gradient: teacher_logits = teacher.forward(batch.inputs) p_teacher = softmax(teacher_logits / tau) student_logits = student.forward(batch.inputs) p_student_soft = softmax(student_logits / tau) p_student_hard = softmax(student_logits / 1.0) loss_soft = KL_divergence(p_teacher, p_student_soft) (tau tau) loss_hard = cross_entropy(batch.labels, p_student_hard) loss = lambda loss_soft + (1 - lambda) loss_hard loss.backward() # gradients flow only through student optimizer.step() ```

Quantization. Storage drops from $32$ or $16$ bits per weight to $b$ bits plus a small per-block overhead: for block size $g$ and scale stored in 16 bits, effective bits-per-weight is $b + 16/g$. With $g=64$, INT4 quantization costs about $4 + 0.25 = 4.25$ bits/weight — roughly a $3.5\times$ to $7.5\times$ reduction versus FP16/FP32. Since LLM autoregressive decoding is memory-bandwidth bound (each generated token requires streaming the entire parameter set through compute, at $O(P)$ bytes moved per token for a $P$-parameter model), and wall-clock decode time is dominated by $\text{bytes moved} / \text{bandwidth}$, quantization yields close to a linear speedup in the compression ratio for the memory-bound regime, though a floor is imposed by compute-bound operations (attention over long contexts) that don't shrink with weight quantization. Correctness argument: quantization error per weight is bounded by half the quantization step, $|w - \hat w \cdot s| \le s/2$. Because block-wise scaling adapts $s$ to the local dynamic range, the relative error per block is controlled independent of extreme values elsewhere in the tensor — this is why per-tensor quantization (one scale for the whole matrix) can catastrophically hurt accuracy when even one outlier channel exists, while per-block or per-channel quantization keeps error bounded near $2^{-b}$ relative to the local range. Empirically, transformer activations contain systematic outlier features in specific channels; algorithms such as LLM.int8() isolate these into a separate high-precision matmul, proving (empirically, not by a closed-form bound) that mixed precision recovers near-lossless accuracy at a small compute overhead.

LoRA. For a weight matrix of shape $d \times k$, full fine-tuning has $O(dk)$ trainable parameters and needs optimizer state (e.g., two Adam moment buffers) of the same size, giving total trainable memory $O(dk)$ per matrix (times a constant of 2–3 for Adam). LoRA has $r(d+k)$ trainable parameters. The reduction factor is:

For a typical $d=k=4096$ and $r=8$, this is a $\sim256\times$ reduction in trainable parameters for that layer, which directly translates into a proportional reduction in optimizer-state memory — the dominant memory cost during training. Forward/backward FLOPs increase only slightly (an extra $O(dr + rk)$ per token instead of $O(dk)$, which is smaller since $r \ll d,k$), so LoRA does not meaningfully slow training and adds no inference-time cost after merging. Correctness argument: LoRA's validity rests on the empirical claim, supported by measurements across many fine-tuning tasks, that the difference between a pretrained weight matrix and its task-adapted counterpart has low intrinsic rank — the singular value spectrum of $\Delta W = W_{\text{finetuned}} - W_0$ decays quickly, so a rank-$r$ approximation captures most of the useful update for moderate $r$ (often $r \in \{4,\dots,64\}$ suffices). This is not a universal theorem — some tasks genuinely need higher rank or updates outside the low-rank subspace spanned from a given initialization — but it holds well enough across benchmarks that LoRA typically matches full fine-tuning accuracy within a percentage point or two, while occasionally underperforming on tasks requiring large representational shifts (e.g., learning genuinely new factual knowledge at scale, versus adapting style or task format).

Distillation. Training cost is dominated by two forward passes (teacher and student) plus one backward pass (student only) per batch, so distillation training costs $O(F_T + F_S + B_S)$ per step where $F$, $B$ denote forward/backward FLOPs — cheaper than training the student from scratch on hard labels only requires, in wall-clock terms, if the teacher's forward pass is precomputed offline (logits cached). Inference cost after distillation is simply the student's own cost, $O(F_S)$ — often $10\times$ to $100\times$ smaller than the teacher's, since students are typically chosen with far fewer parameters and/or layers. Correctness argument: the soft-label KL objective provides more bits of supervision per example than a one-hot label, because it specifies an entire probability vector rather than a single class index. Under a bias–variance view of learning, matching a smoothed teacher distribution acts as a regularizer that reduces the effective hypothesis-space search the student must perform, and the temperature parameter $\tau$ trades off how much of the teacher's fine-grained (but possibly noisy) ranking information is transferred versus how close the loss stays to the ordinary cross-entropy the student would use to fit hard labels alone. Distillation cannot exceed the teacher's own competence — it is fundamentally a compression of what the teacher already knows, and its correctness is empirical: student accuracy approaches teacher accuracy asymptotically as student capacity and training data grow, but a strictly smaller student generally retains a measurable capability gap, especially on tasks requiring capabilities near the frontier of the teacher's own competence.

Consider fine-tuning a 4096×4096 attention projection matrix ($d=k=4096$) with LoRA rank $r=8$, comparing to full fine-tuning, and then quantizing the resulting deployed model to 4-bit.

Scaling this across a model with, say, 200 such matrices (a rough count for attention + MLP projections across many layers of a 7B-parameter model), full fine-tuning needs roughly $200 \times 134$ MB $\approx 26.8$ GB of optimizer state for just these matrices alone, on top of the 14 GB (FP16) or 28 GB (FP32) of weights and additional buffers for the rest of the network — easily exceeding a single 24–48 GB GPU. LoRA needs only $200 \times 0.52$ MB $\approx 104$ MB of optimizer state, making fine-tuning feasible on much smaller hardware, with the frozen 14 GB of FP16 weights as the dominant remaining memory cost. Now apply 4-bit NF4 quantization (block size 64) to the frozen base weights, as QLoRA does:

The 7B-parameter frozen backbone now needs $7\times10^9 \times 0.53 \approx 3.7$ GB instead of 14 GB — roughly a $3.8\times$ reduction — while the LoRA adapters (kept in bfloat16) add only tens of megabytes. This is precisely the regime that lets a 65B-parameter model be fine-tuned on a single 48 GB GPU: the frozen quantized backbone occupies roughly $65\times10^9 \times 0.53 \approx 34$ GB, leaving headroom for activations, the small adapter parameters, and the paged optimizer state.

Distillation trace. Suppose the teacher assigns to the next token after "The capital of France is" the distribution (restricted to the top four tokens): Paris $=0.82$, Lyon $=0.06$, a $=0.05$, London $=0.03$. A hard-label cross-entropy loss only ever tells the student "the answer is Paris, everything else is wrong equally." At temperature $\tau=2$, the softened teacher distribution (recomputed by dividing logits by $\tau$ before softmax) becomes approximately Paris $=0.55$, Lyon $=0.18$, a $=0.15$, London $=0.12$ — the student now learns that Lyon and London are meaningfully more plausible than "a" in this context, information entirely absent from the hard label, at the cost of a slightly less peaked target that the KL loss must still match closely for the dominant class.

Quantized inference runtimes (e.g., 4-bit and 8-bit weight formats supported by widely used inference libraries) are how large open-weight models run on consumer GPUs and even laptops, making local and offline deployment of capable language models practical for individual developers and privacy-sensitive applications. LoRA and its variants underpin the enormous ecosystem of community-shared "adapters" — small files, often tens of megabytes, that reconfigure a shared base model for a new domain, style, or language without redistributing the full weights, and are the standard technique behind rapid, low-cost customization offered by many model-hosting platforms. Distillation is how smaller, cheaper, faster model variants are produced from larger flagship models for latency-sensitive applications such as real-time chat assistants, on-device summarization, and search-result reranking, where the cost or latency of the largest model would be prohibitive at scale. Because these techniques make adaptation and deployment dramatically cheaper, they also lower the barrier to modifying a model's behavior after its original safety training — a LoRA adapter can, in principle, be trained to undo alignment properties introduced during Instruction Tuning just as easily as it can add a helpful new skill. This tension between efficient customization and controllable safety is exactly the terrain explored under Safety & Red-Teaming, where questions of how easily fine-tuning can strip away guardrails, and how to detect or prevent that, become central.

Ask the tutor: How would you decide, for a specific deployment budget, whether to prioritize quantization, LoRA, or distillation first?