The R-CNN Family
A self-driving car's camera sees a cluttered street: pedestrians, cyclists, parked cars, a traffic light. The system must not only say "there are cars here" — it must draw a box around each one and label it correctly. This is Object Detection: localization plus classification, applied to an unknown number of objects per image. Classic Transfer Learning for Vision gives us powerful image classifiers. But a classifier alone answers "what is the dominant object in this image?" It cannot answer "where are all the objects, and what are they?" The R-CNN family solved this by decomposing detection into two stages. First, propose a manageable set of candidate regions that might contain an object. Then classify and refine each region with a convolutional network. This "two-stage" strategy dominated object detection benchmarks from 2014 to roughly 2017. It remains the conceptual backbone of many production systems today. These include parts of medical imaging pipelines, satellite analysis tools, and early autonomous-vehicle perception stacks. The story is also a case study in systems thinking applied to deep learning. R-CNN (2014) was accurate but absurdly slow — minutes per image. Fast R-CNN (2015) shared computation across region proposals inside a single forward pass. Faster R-CNN (2015) replaced the last non-differentiable, CPU-bound piece — the region proposal algorithm. It swapped in a learned neural network, the Region Proposal Network (RPN). This made the entire detector nearly end-to-end trainable and dramatically faster. Each step removes a bottleneck the previous step exposed. This article walks through all three architectures. It shows precisely what problem each one fixes. It includes pseudocode, a diagram of the RPN's anchor mechanism, complexity analysis, and a worked numerical trace. It assumes familiarity with Anchor Boxes and Non-Maximum Suppression. It also sets up the ideas needed for Single-Shot Detectors and Instance Segmentation with Mask R-CNN.
- R-CNN: run selective search for ~2000 region proposals, warp and run each through a CNN — accurate but slow (~47s/image).
- Fast R-CNN: run the CNN once per image; use RoI Pooling to extract fixed-size features per proposal from shared feature maps — 25x+ faster than R-CNN at test time.
- Faster R-CNN: replace external proposal generation (selective search) with a learned Region Proposal Network (RPN) sharing the same backbone — end-to-end trainable, near real time.
- The RPN scores and refines a dense grid of predefined Anchor Boxes at every spatial location of a feature map.
- All stages rely on Non-Maximum Suppression to collapse overlapping detections into one box per object.
- The lineage — R-CNN → Fast R-CNN → Faster R-CNN — is a textbook example of iteratively removing a pipeline's slowest, least-differentiable component.
Object detection becomes tractable by separating "where might an object be" (region proposal) from "what is it" (classification), and each R-CNN generation made that separation cheaper and more differentiable.
The idea
A two-stage detector first generates a sparse set of candidate regions ("region proposals" or "regions of interest," RoIs) that are likely to contain objects, then classifies each candidate and regresses a tighter bounding box around it. This contrasts with single-stage detectors, which predict boxes and classes densely in one shot — a design explored in Single-Shot Detectors. The two-stage approach trades some speed for higher accuracy, because the second stage only has to reason carefully about a few hundred to a few thousand well-chosen locations rather than every pixel. Three terms recur throughout this family. A region proposal is a rectangular candidate box, produced either by a hand-designed algorithm (Selective Search, used in R-CNN and Fast R-CNN) or by a learned network (the RPN in Faster R-CNN). RoI Pooling (and its refinement, RoIAlign, used later in Mask R-CNN) converts a variable-sized rectangular region of a convolutional feature map into a fixed-size feature vector, so that a shared fully-connected classifier can be applied to every proposal regardless of its shape. An anchor box is a reference box of fixed size and aspect ratio, placed at every location of a feature map; the RPN predicts adjustments (regression offsets) relative to these anchors rather than predicting raw pixel coordinates, which stabilizes training — this is the same idea introduced generally in Anchor Boxes. The historical arc of the family is a sequence of bottleneck removals. R-CNN's bottleneck: a separate, un-shared forward pass through a full CNN for each of ~2000 proposals per image. Fast R-CNN's fix: compute the CNN feature map once per image, then pool from it. Fast R-CNN's remaining bottleneck: proposal generation itself (Selective Search) ran on the CPU, took ~2 seconds per image, and was not learned jointly with the network. Faster R-CNN's fix: replace Selective Search with a small convolutional Region Proposal Network sharing the backbone's features, so proposal generation costs almost nothing extra and is trained end-to-end with the detector.
How it works
Stage 0 (all variants): a shared backbone
Every variant starts with a convolutional backbone (e.g., a VGG or ResNet pretrained on ImageNet, per Transfer Learning for Vision) that turns an image into a feature map. What differs is when the backbone is applied relative to region proposals, and how proposals are produced.
R-CNN (2014)
``` function RCNN_DETECT(image): proposals = SelectiveSearch(image) # ~2000 candidate boxes, class-agnostic detections = [] for box in proposals: patch = crop_and_warp(image, box, size=227x227) # warp to fixed CNN input size features = CNN_forward(patch) # full forward pass PER BOX for each class c: score_c = SVM_c(features) # one linear SVM per class if max(score_c) > threshold: refined_box = BBoxRegressor(features, box) # class-specific linear regression detections.append((refined_box, argmax_c, max(score_c))) detections = NonMaxSuppression(detections, iou_threshold=0.3) # per class return detections ``` R-CNN trains three separate components: the CNN (fine-tuned for classification on warped region crops), a bank of one-vs-rest linear SVMs (one per class, trained on cached CNN features), and class-specific bounding-box regressors (trained on CNN features to correct the proposal box toward the ground truth). Because these three pieces are trained independently on cached features rather than jointly end-to-end, R-CNN is accurate for its time (mAP 53.7% on PASCAL VOC 2010) but slow to train and slow at inference: roughly 2000 CNN forward passes per image, each on a warped 227×227 patch, giving roughly 47 seconds per image on a GPU of that era.
Fast R-CNN (2015)
Fast R-CNN's key insight: don't rerun the CNN per proposal. Run the convolutional backbone once on the whole image, and pool features for each proposal directly from the resulting feature map. ``` function FAST_RCNN_DETECT(image, proposals): # proposals still from Selective Search feature_map = CNN_forward(image) # ONE forward pass for the whole image detections = [] for box in proposals: roi_feat = RoIPool(feature_map, project(box), output_size=7x7) # fixed-size, e.g. 7x7xC fc_feat = FullyConnected(roi_feat) class_scores = Softmax(FC_cls(fc_feat)) # (K+1)-way softmax, incl. background bbox_deltas = FC_reg(fc_feat) # per-class (dx, dy, dw, dh) detections.append((box, class_scores, bbox_deltas)) refined = [apply_deltas(box, bbox_deltas[c]) for best class c per detection] refined = NonMaxSuppression(refined, iou_threshold=0.3) return refined ``` `project(box)` maps the proposal's coordinates in the original image to coordinates in the (lower-resolution) feature map, accounting for the backbone's total stride. `RoIPool` then divides that projected region into a fixed `output_size` grid (e.g., 7×7) and max-pools within each grid cell, regardless of the region's original size — this is the trick that lets one fully-connected head process proposals of arbitrary shape. Fast R-CNN also unifies training: a single network with two sibling heads (classification and bounding-box regression) is trained jointly with a multi-task loss: ``` L(p, u, t, t) = L_cls(p, u) + λ·[u ≥ 1]·L_loc(t, t) ``` where `p` is the predicted class distribution, `u` is the ground-truth class (0 = background), `t` is the predicted box offset for the ground-truth class, `t*` is the target offset, `L_cls` is log loss, and `L_loc` is a smooth L1 loss (robust to outlier proposals, unlike raw L2). The indicator `[u ≥ 1]` disables the localization loss for background proposals, which have no meaningful ground-truth box. Because gradients now flow from both heads back through the shared convolutional feature map, the whole network (aside from the external proposal generator) is trained end to end. This yields both higher accuracy and a roughly 25× inference-time speedup over R-CNN's classification stage, though Selective Search proposal generation remains a separate, non-learned, CPU-bound step (~2s/image) — the next bottleneck.
Faster R-CNN (2015) and the Region Proposal Network
Faster R-CNN replaces Selective Search with a small convolutional network — the Region Proposal Network (RPN) — that slides over the same shared feature map Fast R-CNN already computes, and outputs proposals directly, with negligible added cost. This is the crucial unification: one backbone, two heads that share features (RPN and the detection head), trained jointly. The RPN works over a dense grid of anchors: at each spatial location of the feature map, it considers `k` reference boxes of different scales and aspect ratios (typically `k = 9`: 3 scales × 3 ratios), centered at the pixel position that location corresponds to in the original image. ``` function RPN_FORWARD(feature_map): shared = Conv3x3(feature_map, channels=256, activation=ReLU) # shared 3x3 conv for each spatial location (i, j) in shared: for each of k anchors at (i, j): obj_score = Conv1x1_cls(shared[i,j]) # 2 numbers: object vs background (per anchor) bbox_delta = Conv1x1_reg(shared[i,j]) # 4 numbers: (dx, dy, dw, dh) relative to anchor return all (anchor, obj_score, bbox_delta) triples function GENERATE_PROPOSALS(feature_map, image_size): triples = RPN_FORWARD(feature_map) boxes = [apply_deltas(anchor, delta) for (anchor, score, delta) in triples] boxes = clip_to_image(boxes, image_size) boxes = remove_tiny(boxes) ranked = sort_by(boxes, key=obj_score, descending=True) top_N = ranked[:12000] # pre-NMS cap (typical training value) kept = NonMaxSuppression(top_N, iou_threshold=0.7) return kept[:2000] # post-NMS cap fed to detection head ``` Anchors are labeled during training by their Intersection over Union with ground-truth boxes: an anchor is a positive example if its IoU with some ground-truth box exceeds 0.7 (or is the single best-matching anchor for that box), and negative if its IoU with all ground-truth boxes is below 0.3; anchors in between are ignored. This gives the RPN a well-posed binary classification target (object vs. not) plus a regression target (box offset) at every anchor, without ever running a full classifier CNN over the raw pixels. The full pipeline is: ``` function FASTER_RCNN_DETECT(image): feature_map = Backbone_CNN(image) # shared by RPN and detection head proposals = GENERATE_PROPOSALS(feature_map, image.size) detections = [] for box in proposals: roi_feat = RoIPool(feature_map, project(box), output_size=7x7) fc_feat = FullyConnected(roi_feat) class_scores = Softmax(FC_cls(fc_feat)) bbox_deltas = FC_reg(fc_feat) detections.append((box, class_scores, bbox_deltas)) refined = [apply_deltas(box, bbox_deltas[c]) for best class c per detection] refined = NonMaxSuppression(refined, iou_threshold=0.3) # per class return refined ``` Training alternates (in the original scheme) or jointly optimizes (in later, simpler schemes) two multi-task losses — one for the RPN (objectness + anchor regression) and one for the detection head (classification + box regression) — while sharing the backbone's convolutional weights. Because the RPN reuses the same feature map already needed for detection, generating ~2000 high-quality proposals costs only a few extra small convolutions, replacing a ~2-second CPU algorithm with a ~10 millisecond GPU computation.
Complexity & correctness
R-CNN. Let `P ≈ 2000` be the number of proposals and `C_cnn` the cost of one CNN forward pass on a warped patch. Total inference cost is `O(P · C_cnn)`, dominated entirely by redundant convolution: nearby proposals overlap heavily, so the same image content is convolved from scratch many times over. This is why R-CNN takes tens of seconds per image even though a single CNN forward pass takes tens of milliseconds. Training cost is similarly `O(P · C_cnn)` per image, multiplied further by the disjoint SVM and regressor training stages, and storage cost is high because cached CNN features for all proposals must be written to disk (hundreds of gigabytes for PASCAL VOC).
Fast R-CNN. The backbone runs once per image: `O(C_cnn)`. Each of the `P` proposals then costs only `O(RoIPool + FC)`, where `RoIPool` is a small, cheap fixed-size pooling operation and `FC` is one or two fully connected layers — orders of magnitude cheaper than a full convolutional pass. Total cost is `O(C_cnn + P · C_fc)`, and since `C_fc ≪ C_cnn`, this is close to `O(C_cnn)`: the cost is dominated by a single backbone pass, independent (to first order) of `P`. This is precisely why Fast R-CNN is roughly 25× faster than R-CNN at test time despite examining the same number of proposals — the redundant convolution is eliminated by construction, not by approximation.
Faster R-CNN. Proposal generation itself is folded into the shared feature map: the RPN adds a `3×3` conv plus two `1×1` convs evaluated once over the `h × w` feature map, costing `O(h · w · k)` for `k` anchors per location — negligible next to `C_cnn`. Total inference cost remains `O(C_cnn + P · C_fc)`, but now the previous external `O(SelectiveSearch)` term (≈2 seconds, CPU-bound, unparallelized with the network) disappears, replaced by microseconds of extra GPU convolution. This is the precise sense in which Faster R-CNN "removes the last bottleneck": every remaining stage is a GPU tensor operation, and the RPN's `O(h·w·k)` cost scales with the image resolution rather than with a hand-tuned segmentation heuristic.
Why the region-based approach is correct (proposal recall argument). The whole two-stage strategy is only sound if proposals reliably cover true objects — i.e., proposal recall at a reasonable IoU threshold (say 0.5) must be high, or later stages can never recover a missed object. Selective Search achieves high recall by combining multiple segmentation strategies (color, texture, size, fill) so different object types are captured by at least one strategy. The RPN achieves comparable or better recall by covering the image with a dense, multi-scale bank of anchors — since anchors span a range of scales and aspect ratios at every location, some anchor is guaranteed to overlap any reasonably-sized object with IoU exceeding the positive threshold, which is exactly the property the anchor-labeling rule (IoU ≥ 0.7 → positive) is designed to exploit. Formally, if the anchor grid has stride `s` and the anchor scales are chosen so consecutive scales differ by a bounded ratio `r`, then for any ground-truth box within the image's scale range, there exists an anchor whose center is within `s/√2` of the box center and whose scale is within a factor `r` of the box's — bounding the achievable IoU away from zero and justifying empirically observed high recall (>96% at IoU 0.5 on standard benchmarks with typical anchor configurations).
Why the multi-task loss trains a coherent detector. The joint loss `L_cls + λ·L_loc` is correct in the sense that both terms are proper statistical objectives: `L_cls` is the negative log-likelihood of the true class under the predicted softmax (a well-founded classification loss), and `L_loc` is smooth L1 on normalized offsets `(dx, dy, dw, dh)` — parameterizing box regression as a scale-invariant transform of the anchor/proposal rather than raw pixel coordinates. This normalization is what makes a single shared regressor network usable across objects of wildly different sizes: gradients for a small object's box and a large object's box are on comparable numeric scales, which is essential for stable joint training and is the same normalization trick underlying Anchor Boxes in general.
Tradeoffs, honestly stated. Two-stage detectors are more accurate than early single-stage detectors on small and densely packed objects, because the first stage explicitly filters the search space before fine-grained classification — but they are slower than modern single-stage detectors and have more moving parts (anchor design, IoU thresholds, two loss functions) to tune. NMS, used at every stage, is itself a greedy heuristic: it can wrongly suppress a correct detection of a heavily occluded second object of the same class if the two boxes overlap enough, a limitation inherited from Non-Maximum Suppression rather than fixed by anything in this family.
Worked example
Consider a `600 × 600` input image processed by a backbone with total stride 16, producing a `37 × 37` feature map (`600/16 ≈ 37.5`, rounded down by the network's padding/pooling arithmetic). With `k = 9` anchors per location (3 scales × 3 aspect ratios), the RPN evaluates:
| Stage | Quantity | Value |
|---|---|---|
| Feature map locations | `37 × 37` | 1369 |
| Anchors per location | `k` | 9 |
| Total raw anchors | `1369 × 9` | 12321 |
| After clipping to image bounds | (boundary anchors removed) | ≈ 9500 (typical) |
| Kept after objectness ranking (pre-NMS cap) | top-N | 12000 (cap not binding here) |
| Kept after NMS at IoU 0.7 | — | ≈ 2000–6000 |
| Proposals passed to detection head | post-NMS cap | 2000 |
Suppose one ground-truth box is a pedestrian at image coordinates `(x=310, y=200, w=40, h=110)`. An anchor centered at feature-map cell `(19, 12)` — which maps back to image coordinates `(19×16, 12×16) = (304, 192)` — with scale `128` and aspect ratio `1:2` (width 64, height 128) computes IoU with the ground truth:
- Intersection: overlap in x is `min(304+32, 310+20) - max(304-32, 310-20) = min(336,330) - max(272,290) = 330 - 290 = 40`
- Overlap in y is `min(192+64, 200+55) - max(192-64, 200-55) = min(256,255) - max(128,145) = 255 - 145 = 110`
- Intersection area ≈ `40 × 110 = 4400`
- Union = anchor area (`64×128=8192`) + gt area (`40×110=4400`) − intersection (`4400`) = `8192`
- IoU ≈ `4400 / 8192 ≈ 0.537`
This anchor's IoU (0.537) is below the 0.7 positive threshold, so during training it would be labeled ignored unless it happens to be the single highest-IoU anchor for this ground-truth box (the "best anchor" rule), which guarantees every ground-truth object has at least one positive anchor even if none crosses 0.7 outright. Suppose a neighboring anchor at the same location with scale `90`, ratio `1:2.5` (width 40, height 100) achieves IoU 0.81 — this one is labeled positive, its objectness target is 1, and its regression target is: ``` tx = (gt_x - anchor_x) / anchor_w = (310 - 304) / 40 = 0.15 ty = (gt_y - anchor_y) / anchor_h = (200 - 192) / 100 = 0.08 tw = log(gt_w / anchor_w) = log(40 / 40) = 0.0 th = log(gt_h / anchor_h) = log(110 / 100) = 0.095 ``` At inference, the RPN predicts `(dx, dy, dw, dh)` for this anchor from the shared feature map; applying the inverse transform (`x = anchor_x + dx·anchor_w`, etc.) recovers a proposal box close to `(310, 200, 40, 110)`. This proposal, ranked highly by the objectness score, survives NMS, gets pooled via RoI Pooling into a `7×7` grid, and is classified by the detection head — ideally emitting `"pedestrian"` with high confidence and a further-refined box. This trace shows concretely how a raw anchor at a fixed grid location becomes a tight, class-labeled detection through two rounds of learned offset regression (RPN, then detection head) rather than one.
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Forgetting to clip/ignore anchors crossing image borders | Border anchors have ill-defined ground-truth overlap and destabilize training | Clip anchors to image bounds or exclude them from loss during training |
| Using raw pixel-coordinate regression instead of anchor-relative offsets | Large scale variance across objects makes gradients numerically unstable | Regress normalized `(dx, dy, dw, dh)` relative to anchor/proposal size |
| Applying one global NMS threshold across all classes at once | Boxes of different classes can spuriously suppress each other | Run NMS independently per class |
| Treating RoI Pooling's quantization as harmless | Rounding proposal coordinates to the feature grid misaligns features, hurting small-object accuracy | Use RoIAlign (bilinear interpolation) for precision-sensitive tasks, foreshadowing Instance Segmentation with Mask R-CNN |
| Setting the positive/negative IoU thresholds too close together | Leaves too few negative anchors or mislabels ambiguous anchors, causing training imbalance | Use a gap (e.g. positive ≥0.7, negative <0.3) and ignore the middle band |
| Sampling all anchors per image for the RPN loss | Extreme class imbalance: negatives vastly outnumber positives | Sample a fixed mini-batch (e.g. 256 anchors, balanced positive/negative) per image |
Pioneers & history
Origins · 1 minRoss Girshick and how this was discovered
Ross Girshick led the development of R-CNN, Fast R-CNN, and later Mask R-CNN, establishing the region-based deep detection paradigm that dominated the field for years. His work exemplifies a recurring pattern in applied deep learning: identify the actual computational bottleneck in a working system, and design the next architecture specifically to remove it, rather than redesigning from scratch.
Where this matters
Two-stage detectors from this family (or close descendants) have shipped in real production systems: early versions of autonomous-vehicle perception stacks used Faster R-CNN variants for pedestrian and vehicle detection; medical imaging tools use region-based detectors to localize tumors or lesions in radiology scans, where recall matters more than raw speed; satellite and aerial imagery analysis pipelines use them to detect small, densely packed objects (vehicles, buildings) where two-stage precision historically outperformed single-stage alternatives. The RPN's core idea — a lightweight network predicting objectness and offsets over a dense anchor grid — is reused directly inside Instance Segmentation with Mask R-CNN, which adds a mask prediction branch and replaces RoI Pooling with RoIAlign for pixel-accurate segmentation. The broader proposal-then-classify pattern also foreshadows the contrast explored in Single-Shot Detectors, which fuse both stages into one dense prediction pass at some cost to accuracy on small or crowded objects.
Ask the tutor: How would the anchor-matching and loss design in the RPN need to change if objects can appear at extreme scale variation, like both 5-pixel and 500-pixel objects in the same image?
Why is Faster R-CNN considered "end-to-end trainable" while R-CNN is not?
In R-CNN, region proposals come from Selective Search, a fixed, non-differentiable, hand-designed algorithm that is not updated during training; the CNN, SVMs, and box regressor are also trained as separate stages on cached features rather than jointly. In Faster R-CNN, the Region Proposal Network is itself a small neural network sharing the backbone's convolutional weights, so gradients from both the RPN's objectness/regression loss and the detection head's classification/regression loss can flow back through the shared feature map in a single joint (or alternating, in early training schemes) optimization. This does not mean literally every component is a differentiable function of every other (NMS, for instance, remains a non-differentiable post-processing step in both), but the core proposal-generation-plus-classification pipeline becomes trainable with backpropagation rather than requiring separately-trained modules on cached features.
What exactly does the "R" in R-CNN stand for, and what problem does the region-based design solve?
"R-CNN" stands for "Regions with CNN features." Before R-CNN, applying a CNN classifier to every possible window of an image (a sliding-window approach) was computationally infeasible because the number of candidate windows across all scales and aspect ratios is enormous. R-CNN's insight was to first use a cheap, class-agnostic algorithm (Selective Search) to propose a much smaller set — around 2000 — of promising regions likely to contain objects, based on low-level cues like color and texture similarity, and only then run the expensive CNN classifier on that reduced set. This region-based filtering is what made deep CNN-based detection computationally tractable in the first place.
How does RoI Pooling differ from RoIAlign, and why does it matter?
RoI Pooling divides a proposal's projected region on the feature map into a fixed grid (e.g., 7×7) and applies max pooling within each cell, but because the projected region's boundaries rarely align exactly with the integer feature-map grid, RoI Pooling rounds (quantizes) coordinates twice — once when mapping the box onto the feature map, and once when dividing that box into grid cells. This quantization introduces small misalignments between the pooled features and the actual object location, which matters little for large objects but can noticeably hurt localization for small objects or pixel-precise tasks. RoIAlign, introduced with Mask R-CNN, avoids both quantization steps by using bilinear interpolation to sample feature values at exact (non-integer) coordinates, improving both detection and, especially, mask-prediction accuracy.
Why does the RPN use multiple anchor scales and aspect ratios rather than just one box size per location?
Real-world objects vary enormously in scale (a nearby car versus a distant pedestrian) and shape (a tall pedestrian versus a wide bus). A single anchor shape per location could only achieve high IoU with objects of roughly that same shape and size, meaning many true objects would never get a well-matching anchor — collapsing proposal recall. By placing several anchors of different scales and aspect ratios at every location, the RPN approximates a multi-scale, multi-shape sliding-window search efficiently within one convolutional pass, so that essentially every plausible object shape has at least one reasonably-matching anchor nearby, which is what the recall argument in the Complexity & Correctness section formalizes.
Are two-stage detectors like Faster R-CNN still used today, or have they been fully replaced?
Two-stage detectors are less common as the default choice in latency-critical applications, where single-stage detectors and transformer-based detectors have become popular, but they remain relevant where accuracy on small or densely packed objects matters more than raw speed — satellite imagery, medical imaging, and certain industrial inspection systems still use Faster R-CNN or its descendants. More importantly, the architectural ideas — shared backbone features, anchor-based dense prediction, RoI-based feature pooling — persist inside many modern systems, including Mask R-CNN for instance segmentation and various keypoint and 3D detection pipelines, so understanding this family is foundational even where the exact original architecture is no longer deployed.
What is the role of Non-Maximum Suppression in this pipeline, and where exactly is it applied?
Non-Maximum Suppression is applied at least twice in Faster R-CNN: once inside the RPN, to collapse thousands of overlapping raw anchor-based proposals down to a manageable, mostly non-redundant set (e.g., 2000) before passing them to the detection head, and once again after the detection head produces final class-specific refined boxes, to collapse multiple overlapping detections of the same object into a single final box per object per class. Without NMS, the RPN's dense anchor grid would produce dozens of near-duplicate proposals for every real object, and the final output would show many redundant, overlapping bounding boxes for a single physical object instead of one clean detection.
Why is bounding-box regression trained with smooth L1 loss instead of ordinary squared error?
Squared error (L2) loss grows quadratically with the size of the prediction error, so a handful of very poor proposals — proposals whose offset from the ground truth is unusually large, which happens fairly often early in training or for oddly-shaped objects — can dominate the gradient and destabilize learning for the rest of the batch. Smooth L1 loss behaves like squared error for small errors (giving well-behaved gradients near the optimum) but transitions to linear (L1-like) behavior for large errors, capping how much any single bad example can dominate the total gradient. This makes bounding-box regression training substantially more robust to the occasional badly-matched proposal or outlier ground-truth annotation.