Anchor Boxes

An anchor box is a fixed-size, fixed-aspect-ratio rectangle placed at a specific location in the image, used as a reference from which the detector predicts a refined bounding box. Anchors are not learned positions — they are a deterministic geometric grid, defined once from a small set of hyperparameters: a set of scales $\{s_1, \dots, s_m\}$ (areas, typically expressed as $\sqrt{\text{area}}$) and a set of aspect ratios $\{r_1, \dots, r_k\}$ (width-to-height ratios such as $1{:}1$, $1{:}2$, $2{:}1$). At every location on a convolutional feature map, the detector instantiates one anchor per (scale, ratio) combination, giving $m \times k$ anchors per location and, over an $H \times W$ feature map, $H \cdot W \cdot m \cdot k$ anchors in total. Each anchor is treated as a candidate detection slot. During training, every anchor is assigned a label — positive (matched to some ground-truth object), negative (background), or ignored (ambiguous overlap) — based on its Intersection-over-Union (IoU) with ground-truth boxes. Positive anchors additionally receive a regression target: the parameterized offset needed to transform the anchor into the matching ground-truth box. The network head, at that same spatial location, predicts an objectness/class score and a 4-vector of offsets; the loss compares predicted offsets to targets only for positive (and sampled negative) anchors. At inference time, the offsets are decoded by applying them to the anchor geometry, producing the final predicted box, which is then filtered by score threshold and non-maximum suppression. The key conceptual shift is that the search space of possible boxes is discretized in advance by the anchors, and the learning problem becomes local refinement plus classification. This is why anchors are often described as encoding a strong geometric prior: rather than a network having to learn, from pixels alone, that "objects near this pixel tend to be roughly 40×80 pixels," the anchor grid supplies that scale explicitly, and the network only needs to learn the residual correction and the presence/absence decision.

Step 1 — Anchor generation. For a feature map of size $H \times W$ with stride $s$ (the downsampling factor from image to feature map), each cell $(i, j)$ corresponds to an image-space center

For every combination of scale $a$ (anchor area, or a base size times a scale factor) and aspect ratio $r$, the anchor width and height are

(one common convention; SSD/Faster R-CNN vary slightly in exact formula but the idea — area fixed, width/height set by ratio — is the same). The anchor is the box centered at $(c_x, c_y)$ with that width and height.

Step 2 — Matching / assignment. Given ground-truth boxes $\{G_k\}$ for an image, compute IoU between every anchor $A_i$ and every $G_k$. A standard two-threshold rule (used in Faster R-CNN, SSD):

• If $\max_k \text{IoU}(A_i, G_k) \ge \tau_{\text{pos}}$ (e.g., 0.7), label $A_i$ positive, assigned to $\arg\max_k \text{IoU}(A_i, G_k)$.
• If $\max_k \text{IoU}(A_i, G_k) < \tau_{\text{neg}}$ (e.g., 0.3), label $A_i$ negative (background).
• Otherwise, label $A_i$ ignored (excluded from the loss) to avoid confusing ambiguous cases.
• To guarantee every ground-truth box has at least one positive anchor, additionally force the single best-matching anchor for each $G_k$ to be positive, regardless of its absolute IoU.

Step 3 — Regression target parameterization. For each positive anchor $A = (x_a, y_a, w_a, h_a)$ (center-x, center-y, width, height) matched to ground truth $G = (x_g, y_g, w_g, h_g)$, the regression target is:

This parameterization is scale-invariant: translation offsets are normalized by anchor size, and size offsets are logarithmic ratios, so a target for a huge object and a target for a tiny object live on comparable numeric scales — this is essential for a single regression head with one loss weighting to train stably across scales.

Step 4 — Loss. The network predicts, per anchor, a class score (or objectness) $\hat{p}$ and offsets $(\hat t_x, \hat t_y, \hat t_w, \hat t_h)$. The total loss sums a classification loss (cross-entropy or focal loss) over all sampled anchors, and a regression loss (typically smooth-L1 / Huber) over positive anchors only:

Step 5 — Decoding at inference. Given a predicted offset $(\hat t_x,\hat t_y,\hat t_w,\hat t_h)$ for anchor $A$, invert the parameterization:

Boxes are then filtered by class score threshold and passed through non-maximum suppression. ```

# ANCHOR GENERATION function generate_anchors(feature_h, feature_w, stride, scales, ratios): anchors = [] for i in 0 .. feature_h - 1: for j in 0 .. feature_w - 1: cx = (j + 0.5) stride cy = (i + 0.5) stride for a in scales: for r in ratios: w = a sqrt(r) h = a / sqrt(r) anchors.append(Box(cx, cy, w, h)) # center-form return anchors # length = feature_h feature_w |scales| |ratios|

# MATCHING (TRAINING TIME) function match_anchors(anchors, gt_boxes, pos_thresh, neg_thresh): labels = array(len(anchors), fill = IGNORE) targets = array(len(anchors), fill = None) assigned_gt = array(len(anchors), fill = -1) iou_matrix = compute_iou(anchors, gt_boxes) # |anchors| x |gt|

# (a) best anchor per ground-truth box is forced positive for k in 0 .. len(gt_boxes) - 1: i_best = argmax_i( iou_matrix[:, k] ) labels[i_best] = POSITIVE assigned_gt[i_best] = k

# (b) threshold-based assignment for the rest for i in 0 .. len(anchors) - 1: best_k = argmax_k( iou_matrix[i, :] ) best_iou = iou_matrix[i, best_k] if labels[i] == POSITIVE: continue # already forced positive if best_iou >= pos_thresh: labels[i] = POSITIVE assigned_gt[i] = best_k else if best_iou < neg_thresh: labels[i] = NEGATIVE

# else: stays IGNORE

# (c) compute regression targets for positives for i in 0 .. len(anchors) - 1: if labels[i] == POSITIVE: targets[i] = encode(anchors[i], gt_boxes[assigned_gt[i]]) return labels, targets function encode(anchor, gt): tx = (gt.cx - anchor.cx) / anchor.w ty = (gt.cy - anchor.cy) / anchor.h tw = log(gt.w / anchor.w) th = log(gt.h / anchor.h) return (tx, ty, tw, th)

# DECODING (INFERENCE TIME) function decode(anchor, t): x = t.tx anchor.w + anchor.cx y = t.ty anchor.h + anchor.cy w = anchor.w exp(t.tw) h = anchor.h exp(t.th) return Box(x, y, w, h) ```

Time complexity of anchor generation. Generating anchors is a fixed enumeration: $O(H \cdot W \cdot m \cdot k)$ boxes, each built in $O(1)$ time from closed-form arithmetic. This is cheap — for a typical feature map $H=W=50$ with $m=3$ scales and $k=3$ ratios, that is $50 \times 50 \times 9 = 22{,}500$ anchors, generated once (they don't depend on the image content, only on the feature map size, so they can be precomputed and cached for a fixed input resolution).

Time complexity of matching. Computing the full IoU matrix between $N_a$ anchors and $N_g$ ground-truth boxes costs $O(N_a \cdot N_g)$ if done densely. Since $N_g$ (objects per image) is typically small (single digits to low hundreds) while $N_a$ can be tens of thousands, this is effectively linear in the number of anchors, $O(N_a)$ for constant $N_g$. Assignment (argmax over rows/columns) is also $O(N_a \cdot N_g)$. This is affordable per training image and is why anchor-based detectors can densely supervise every location without a combinatorial search over box proposals.

Space complexity. Storing $N_a$ anchors (4 floats each) is $O(N_a)$; storing per-anchor labels and regression targets is likewise $O(N_a)$. Since anchors are geometry-only (no learned parameters), the memory cost is a small constant-factor overhead compared to the feature maps and network weights themselves.

Why the scale-invariant parameterization is the correct choice, not an arbitrary one. If regression targets were raw pixel differences $(x_g - x_a)$, the loss magnitude — and hence the gradient scale — would depend on absolute object size and image resolution: a 3-pixel error on a 10-pixel object is catastrophic, but a 3-pixel error on a 300-pixel object is negligible, yet an unnormalized L2/L1 loss would weight them identically in raw pixels. Dividing by anchor width/height ($t_x, t_y$) makes translation error relative to object scale; using $\log$ ratios for width/height ($t_w, t_h$) makes size error symmetric and additive under composition (multiplying width by 2 and predicting the wrong direction costs the same as dividing by 2), and keeps the target finite and centered near 0 when anchor and ground truth are close — a first-order Taylor argument: for $w_g \approx w_a$, $\log(w_g/w_a) \approx (w_g - w_a)/w_a$, so near a good match the parameterization reduces to a simple normalized difference, matching the translation terms' form and giving a well-conditioned loss landscape across all four coordinates.

Why the matching rule is correct (recall guarantee). The "force the best anchor per ground truth to be positive" rule (Step 2a) is not an optimization heuristic; it is a coverage guarantee. Without it, a small or unusually shaped ground-truth object whose best IoU with any anchor falls below $\tau_{\text{pos}}$ would receive zero positive anchors and could never be learned no matter how the regression head is trained — the gradient for that object would simply never appear in the loss. Forcing the arg-max anchor to be positive guarantees every ground-truth instance contributes at least one training signal, closing this gap; the IoU-threshold rule on top of it lets multiple anchors match large or elongated objects, giving redundancy that helps recall at inference (several anchors "fire" for the same object, and non-maximum suppression consolidates them).

Tradeoffs. Anchors introduce hyperparameters (how many scales, ratios, thresholds) that must be tuned or learned from data statistics (e.g., k-means over ground-truth box shapes, as in YOLOv2). Too coarse a set of scales/ratios under-covers unusual object shapes, hurting recall; too fine a set inflates $N_a$ and slows training/inference and increases class imbalance (the vast majority of anchors are negative, requiring hard-negative mining or focal loss to keep the loss from being swamped by easy background). This design burden is exactly what anchor-free detectors later tried to remove, at the cost of different assignment heuristics.

Consider a $4\times4$ feature map (16 cells) from an image of size $128\times128$, so stride $s = 32$. Use one scale $a = 64$ (area-scale in pixels) and two aspect ratios $r \in \{1, 2\}$ (square and 2:1 wide), giving $2$ anchors per cell, $32$ anchors total. Take the cell at $(i,j) = (1,2)$ (0-indexed): center $c_x = (2+0.5)\cdot 32 = 80$, $c_y = (1+0.5)\cdot 32 = 48$.

• Anchor A (ratio 1): $w = 64\sqrt{1}=64$, $h = 64/\sqrt{1}=64$ → box center $(80,48)$, size $64\times64$.
• Anchor B (ratio 2): $w = 64\sqrt{2}\approx 90.5$, $h = 64/\sqrt{2}\approx 45.3$ → box center $(80,48)$, size $90.5\times45.3$.

Suppose the ground truth is a car: $G = (x_g,y_g,w_g,h_g) = (85, 50, 70, 50)$ (center-form).

IoU check. Anchor A: $(80,48,64,64)$ → corners $[48,16]$ to $[112,80]$. $G$ corners: $[50,25]$ to $[120,75]$. Intersection: x-range $[\max(48,50),\min(112,120)] = [50,112]$ → width 62; y-range $[\max(16,25),\min(80,75)] = [25,75]$ → height 50. Intersection area $= 62\times50 = 3100$. Union $= 64\cdot64 + 70\cdot50 - 3100 = 4096+3500-3100 = 4496$. IoU $\approx 3100/4496 \approx 0.69$. With $\tau_{\text{pos}}=0.7$, this narrowly misses the threshold — but suppose across the whole anchor set this is still the single highest-IoU anchor for $G$; by the forced-positive rule, Anchor A is still labeled positive for $G$ regardless.

Encoding the regression target for Anchor A:

The network head at cell $(1,2)$, anchor-slot A, is trained to regress toward $(0.0781, 0.0313, 0.0896, -0.2469)$ and to classify this anchor as "car" (positive).

Decoding (inference-time check). Suppose at test time the network predicts $\hat t = (0.07, 0.03, 0.09, -0.24)$ for this anchor. Decode:

The decoded box $(84.48, 49.92, 70.03, 50.34)$ is nearly identical to the true $G=(85,50,70,50)$ — the small offset the network had to learn was tiny precisely because Anchor A was already a good geometric prior for this object.

Anchor boxes as a named, systematized mechanism entered mainstream computer vision through region-proposal and single-shot detection research in the mid-2010s, though the underlying idea — multi-scale, multi-aspect-ratio template windows — traces back further to sliding-window and part-based detection work.

Anchor-based design is the backbone of two major detector families taught alongside this topic. In the two-stage family — The R-CNN Family — Faster R-CNN's Region Proposal Network uses anchors to generate object proposals cheaply from a shared convolutional feature map, replacing external proposal algorithms like Selective Search and enabling end-to-end training. In the one-stage family — Single-Shot Detectors such as SSD and YOLOv2/v3 — anchors (sometimes called "default boxes" or "priors") are tiled across multiple feature-map resolutions so that shallow, high-resolution layers handle small objects and deep, low-resolution layers handle large ones, all within a single forward pass with no separate proposal stage. Production systems in autonomous driving perception stacks, retail shelf analytics, aerial and satellite imagery pipelines, and video surveillance historically relied on anchor-based detectors for their favorable accuracy/speed tradeoff and mature tooling; many current-generation detectors have shifted toward anchor-free or query-based (transformer) designs, but understanding anchor mechanics remains essential for interpreting model configs, benchmark comparisons, and the design decisions of frameworks like Faster R-CNN, SSD, and RetinaNet that still anchor (pun intended) much of the field's baseline literature and remain widely deployed.

Ask the tutor: How do the anchor scale and aspect-ratio choices in SSD differ across its multiple feature-map resolutions, and why?