Single-Shot Detectors (YOLO/SSD)

A single-stage or single-shot detector predicts every bounding box and class label from one forward pass of a convolutional network, with no intermediate step that proposes candidate regions and re-processes them. This is the essential structural distinction from two-stage detectors (e.g., Faster R-CNN), which first run a region proposal network to generate a few hundred to a few thousand candidate boxes, then crop and re-classify each candidate with a second network stage. The mechanism that makes a single pass possible is dense prediction over a fixed output structure. The network's final feature map has spatial dimensions $H' \times W'$ (much smaller than the input, due to downsampling), and each of the $H' \times W'$ locations is treated as a predictor site. Each site is responsible for detecting objects whose center (or best-matching anchor) falls in its receptive field. At each site, the network predicts a small, fixed-length vector: box offsets (typically 4 numbers), an objectness/confidence score, and a probability distribution over $C$ classes. Multiplying this out across the whole grid and (in SSD's case) across several grids at different resolutions, a single forward pass yields many thousands of candidate detections, most of which are immediately suppressed as low-confidence or redundant. Two design vocabulary items carry the rest of the article. An anchor box (also called a default box or prior box) is a pre-defined box shape — a specific width and height, sometimes tied to a specific aspect ratio — placed at a grid location; the network predicts offsets relative to the anchor rather than absolute coordinates, which makes the regression problem easier because the anchor already encodes a reasonable scale/aspect-ratio prior. <a class="wiki-link" href="/wiki/cs-cv-non-maximum-suppression" data-track="wiki_article.body.wiki_link">Non-Maximum Suppression</a> is the post-processing step that takes the dense flood of overlapping candidate boxes and, for each class, greedily keeps the highest-confidence box and discards other boxes that overlap it above an IoU threshold. Single-shot detectors are inseparable from NMS: the density that gives them speed and recall also produces massive redundancy that must be pruned at the end.

YOLO's mechanism (grid cells, no explicit anchors in the original version). The input image is divided into an $S \times S$ grid (YOLOv1 used $S=7$). Each grid cell is responsible for detecting any object whose ground-truth bounding-box center falls inside that cell. Each cell predicts $B$ bounding boxes (YOLOv1 used $B=2$), each as $(x, y, w, h, \text{confidence})$, plus one set of $C$ class probabilities shared across the boxes in that cell. The output tensor has shape $S \times S \times (B \cdot 5 + C)$. At test time, the class score for a given box is the product of that cell's class probability and the box's confidence.

SSD's mechanism (multi-scale anchors, no grid-cell class sharing). SSD attaches detection heads to several feature maps of decreasing resolution taken from different depths of a backbone network (e.g., a VGG or ResNet trunk with extra convolutional layers appended). At each spatial location in each feature map, SSD places several anchor boxes with different aspect ratios and a scale appropriate to that feature map's resolution — coarse feature maps (deep, low resolution, large receptive field) use large anchors for large objects; fine feature maps (shallow, high resolution) use small anchors for small objects. Each anchor gets its own box-offset prediction and its own $C+1$-way class prediction (including "background"). This directly answers the single hardest weakness of YOLOv1 — poor detection of small and multi-scale objects — by making scale explicit in the architecture rather than implicit in one grid.

Training-time matching. Both families need a rule that decides, for every ground-truth object, which predictor (grid cell in YOLO; anchor at a location in SSD) is responsible for regressing to it.

• YOLO: the grid cell containing the ground-truth box's center is responsible; among that cell's $B$ predicted boxes, the one with highest IoU to the ground truth at training time is assigned to it (the others are penalized only on confidence, pushed toward zero).
• SSD: an anchor is matched to a ground-truth box if their IoU exceeds a threshold (commonly $0.5$), and additionally each ground-truth box is guaranteed at least one match (its best-IoU anchor), following the standard Anchor Boxes matching strategy used in most modern detectors.

Loss function. Both use a multi-task loss combining localization and classification, evaluated only where a predictor is "responsible" (matched) plus a term that suppresses false positives elsewhere:

YOLOv1 uses sum-squared error for everything, with a coordinate-loss weight $\lambda_{\text{coord}} = 5$ (boost the signal from box regression, which is otherwise drowned out by the many "no object" cells) and a weight $\lambda_{\text{noobj}} = 0.5$ (down-weight the many background confidence terms). SSD uses smooth-L1 loss for box regression and softmax cross-entropy for classification, with hard negative mining: because background anchors vastly outnumber object anchors, only the hardest-scoring negatives (highest confidence background is wrong) are kept, at a fixed ratio (commonly 3:1) to positives, so the loss is not overwhelmed by trivially-easy negatives.

Inference pipeline. After the forward pass, the network has emitted thousands of (box, class-score) pairs. NMS is applied per class: sort candidates by confidence, keep the top box, remove all remaining boxes of the same class whose IoU with it exceeds a threshold (e.g., 0.45), repeat. ```pseudocode

# ---- Single-shot detector: training-time target assignment ---- function ASSIGN_TARGETS(ground_truths, anchors, iou_threshold):

# anchors: list of (location, box_shape) pairs, one per predictor site matches = empty map from anchor -> ground_truth (or "background") for a in anchors: matches[a] = "background"

# 1. Ensure every ground-truth gets at least one anchor (best-IoU match) for g in ground_truths: best_anchor = argmax_a IoU(a.box, g.box) matches[best_anchor] = g

# 2. Additionally match any anchor whose IoU with some gt exceeds threshold for a in anchors: if matches[a] == "background": g_best, iou_best = argmax_and_value_g IoU(a.box, g.box) if iou_best >= iou_threshold: matches[a] = g_best return matches

# ---- Single-shot detector: forward pass + loss (one image) ---- function FORWARD_AND_LOSS(image, ground_truths, anchors, network, iou_threshold): feature_maps = network.backbone(image) # one or several resolutions predictions = [] # (offset, class_scores) per anchor for fmap in feature_maps: for location in fmap.spatial_positions: for anchor_shape in anchor_shapes_at(fmap): offset, class_scores = network.head(fmap, location, anchor_shape) predictions.append((location, anchor_shape, offset, class_scores)) targets = ASSIGN_TARGETS(ground_truths, anchors, iou_threshold) loc_loss, cls_loss = 0, 0 positives, negatives = [], [] for (loc, shape, offset, class_scores), a in zip(predictions, anchors): if targets[a] != "background": g = targets[a] loc_loss += SMOOTH_L1(offset, encode(g.box, a)) # encode: box -> anchor-relative offset cls_loss += CROSS_ENTROPY(class_scores, g.label) positives.append(a) else: negatives.append((a, class_scores))

# Hard negative mining: keep only the highest-confidence-wrong negatives negatives.sort_by(confidence_of_background_being_wrong, descending=True) kept_negatives = negatives[: 3 len(positives)] for (a, class_scores) in kept_negatives: cls_loss += CROSS_ENTROPY(class_scores, "background") return loc_loss + LAMBDA cls_loss

# ---- Inference: decode + per-class Non-Maximum Suppression ---- function DETECT(image, network, anchors, score_threshold, nms_iou_threshold): predictions = network.forward(image) # dense (offset, class_scores) per anchor boxes_by_class = map from class -> list of (box, score) for (loc, shape, offset, class_scores), a in zip(predictions, anchors): box = decode(offset, a) # anchor + offset -> absolute box for c in classes: score = class_scores[c] if score >= score_threshold: boxes_by_class[c].append((box, score)) final_detections = [] for c in classes: kept = NON_MAX_SUPPRESSION(boxes_by_class[c], nms_iou_threshold) final_detections.extend([(box, score, c) for (box, score) in kept]) return final_detections ```

Time complexity of inference. The forward pass is a fixed convolutional network applied once to the image; its cost is the usual $O(\text{FLOPs of the CNN})$, independent of the number of objects in the scene. This is the crucial structural difference from two-stage detectors, whose region-classification stage costs $O(R \cdot f)$, where $R$ is the number of proposed regions (hundreds to thousands) and $f$ is the cost of classifying one cropped region — a cost that scales with scene complexity. A single-shot detector's total predictor count is fixed in advance:

summed over feature-map scales $k$ with spatial size $H_k \times W_k$ and $A_k$ anchor shapes each — a constant determined by network architecture, not by the image content. Decoding all predictions is $O(N_{\text{anchors}})$. NMS, applied per class with $n$ candidates after confidence thresholding, costs $O(n \log n)$ for the sort plus $O(n^2)$ worst case for pairwise IoU suppression checks (each surviving box is compared against remaining candidates); in practice $n$ is small after thresholding, and spatial data structures or vectorized IoU on a GPU make this cost negligible relative to the CNN forward pass. Net effect: single-shot detection is $O(1)$ forward passes versus a two-stage detector's proposal generation plus $O(R)$ classification passes, which is the formal reason single-shot detectors are faster by roughly the ratio $R$ (often 100–2000$\times$ in proposal count) modulo the fact that per-region classification networks in two-stage detectors are typically smaller than the shared backbone.

Space complexity. Memory is dominated by the feature-map activations, $O\!\left(\sum_k H_k W_k D_k\right)$ for feature depth $D_k$, plus the dense prediction tensor $O(N_{\text{anchors}} \cdot (4 + C))$. Both are fixed at design time and do not grow with the number of true objects in the image, unlike two-stage detectors whose proposal buffer and per-region feature crops scale with $R$.

Correctness argument — why dense grid/anchor prediction can find arbitrary objects. The core claim to justify is: a fixed, dense set of predictors can still detect a variable, unbounded number of objects of varying position, scale, and aspect ratio. The argument has two parts. 1. Coverage. Every anchor is anchored to a specific location and a specific scale/aspect-ratio prior. As long as the union of anchor shapes across all locations and scales sufficiently tiles the space of plausible object geometries — this is a design/hyperparameter guarantee, verified empirically by measuring the best achievable IoU (the "recall upper bound") of the anchor set against a dataset's ground-truth box distribution — every ground-truth object has some anchor with IoU high enough to be matched and refined by regression. SSD's use of multiple feature-map resolutions directly targets this: without it, a single grid resolution cannot simultaneously give large receptive fields for big objects and fine spatial resolution for small ones, since receptive field size and spatial resolution are coupled in a plain CNN. 2. Assignment consistency (no collision problem in expectation). Two ground-truth objects rarely need the same anchor at the same location, because the "responsible predictor" is chosen by spatial containment (YOLO) or IoU-maximizing match (SSD), and real images rarely have two object centers in the same small grid cell with similar scale. When collisions do happen — the classical failure mode being small, densely packed objects (e.g., a flock of birds) — this is an honest limitation, not a bug: a network is architecturally limited to fewer detections than predictor sites of the appropriate scale in the same cell, since one predictor conventionally reports one box. This is precisely why YOLOv1's coarse $7\times7$ grid struggled with clusters of small objects, and why later versions increased predictor density and added anchors, and why some limitation of this kind persists in any pure grid-based formulation.

Why the loss makes learning well-posed. The regression targets are always defined relative to an anchor (offset encoding, not raw pixel coordinates), which keeps the numeric range of regression targets small and roughly zero-centered regardless of object scale — this is exactly the anchor-relative parameterization introduced generically for Anchor Boxes, and it is what makes a single shared regression head trainable across wildly different object sizes. Hard-negative mining (SSD) and the loss re-weighting (YOLO's $\lambda_{\text{noobj}} = 0.5$) address a genuine numerical pathology: with tens of thousands of anchors and typically only a handful of true objects per image, an unweighted loss is dominated almost entirely by easy background gradients, which would otherwise wash out the useful localization and foreground-classification signal — a straightforward class-imbalance argument, later formalized more sharply by focal loss in RetinaNet.

Honest tradeoffs. Single-shot detectors gain speed and simplicity (no proposal network to train separately, no RoI cropping/pooling machinery) at a real cost: coarser localization for small or overlapping objects, sensitivity to the anchor design (poor anchor shapes silently cap recall no matter how well the network trains), and — especially for YOLOv1's shared per-cell class prediction — an inability to detect multiple objects of different classes whose centers land in the same grid cell.

Consider a simplified single-scale SSD-style head on a $3 \times 3$ feature map (so $S=3$), with $2$ anchor shapes per location (a "wide" and a "tall" box), $C = 2$ classes (person, car), image size $300 \times 300$, so each grid cell corresponds to a $100 \times 100$-pixel stride region.

Setup. Ground truth: one car, box $[120, 130, 260, 210]$ (in $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$ pixels), center at $(190, 170)$.

Step 1 — locate responsible grid cell. Cell size is $100\times100$; the center $(190,170)$ falls in cell $(\text{col}=1, \text{row}=1)$ (0-indexed), since $100 \le 190 < 200$ and $100 \le 170 < 200$.

Step 2 — compute IoU with both anchors at that cell. Anchor centers are placed at the cell center, $(150, 150)$ in pixel coordinates (cell (1,1) center). Suppose:

• Anchor A ("wide"): $80 \times 40$ → box $[110, 130, 190, 170]$
• Anchor B ("tall"): $60 \times 100$ → box $[120, 100, 180, 200]$

Ground-truth area is $140 \times 80 = 11200$; recompute unions carefully: Anchor A area $=3200$, union $=3200+11200-2800=11600$, IoU $=2800/11600 \approx 0.241$. Anchor B area $=6000$, union $=6000+11200-4200=13000$, IoU $=4200/13000\approx0.323$.

Step 3 — matching. Neither anchor clears a typical $0.5$ threshold on its own, but Anchor B has the higher IoU, so under the "best-match guarantee" rule, ground truth is matched to (cell (1,1), Anchor B) regardless of threshold. This cell/anchor pair becomes a positive training example; all other 17 (cell, anchor) pairs at this scale are treated as background unless matched to some other ground truth.

Step 4 — regression target (encoding). SSD encodes offsets relative to the anchor:

With ground-truth center $(190,170)$, width/height $(140,80)$, and anchor B center $(150,150)$, width/height $(60,100)$:

The network at (cell (1,1), anchor B) is trained via smooth-L1 loss to output $(0.667, 0.2, 0.847, -0.223)$, and via cross-entropy to output class "car" with high probability.

Step 5 — inference and NMS. At test time the network emits predictions at all $9$ cells $\times$ $2$ anchors $=18$ sites (plus, in a real SSD, several more scales). Suppose after decoding, three overlapping boxes near the true car all score highly for "car": $0.91$, $0.78$, $0.55$, with pairwise IoUs of $0.7$, $0.62$, and $0.3$ respectively relative to the top box. With an NMS IoU threshold of $0.45$: sort by score → keep $0.91$; suppress $0.78$ (IoU $0.7 > 0.45$) and $0.55$'s comparison to the top box (IoU $0.62>0.45$, suppressed) — result: a single retained detection at the highest-confidence box, matching the intuition that NMS collapses the dense, redundant grid output into one detection per object.

Single-shot detection emerged from a research push, roughly 2015–2016, to make deep object detectors fast enough for real-time and embedded use, building directly on the two-stage R-CNN lineage and on the Anchor Boxes mechanism first popularized by Faster R-CNN's region proposal network.

Real-time video analytics systems — traffic monitoring, retail shelf analysis, sports analytics — depend on single-shot detectors precisely because throughput at video frame rate is a hard requirement, not a nice-to-have. Autonomous vehicles and drones use single-shot or closely related dense detectors as one component of perception stacks where latency budgets are measured in milliseconds. Mobile and embedded vision (phone camera apps, AR filters, on-device object recognition) rely on lightweight single-shot architectures (e.g., MobileNet-SSD variants) because there is no accelerator budget for a two-stage pipeline's proposal-and-crop machinery. The core anchor-matching and NMS post-processing machinery developed for YOLO/SSD also carries forward, largely unchanged, into later dense one-stage detectors like RetinaNet and FCOS, and the general question of how to structure dense per-location predictions resurfaces — with attention replacing convolutional locality — in detection heads built on top of Vision Transformers.

Ask the tutor: How does the anchor-matching strategy in SSD generalize to the set-prediction matching used in DETR-style Vision Transformer detectors?