Instance Segmentation with Mask R-CNN
A self-driving car sees two cyclists standing shoulder to shoulder. A bounding box tells you where they are. But it doesn't tell you whose wheel is whose. Semantic segmentation labels every pixel "cyclist" but merges them into one blob. What the car actually needs is different. It needs a pixel-accurate outline for each individual cyclist. It also needs a category label and a confidence score. This is instance segmentation: detect every object, and simultaneously carve out its exact silhouette. Mask R-CNN solves this by extending the The R-CNN Family object-detection pipeline. It adds a third output branch that predicts a binary mask for every detected instance. The key engineering insight is deceptively small. It replaces a quantized pooling operation with a continuous, aligned one. Yet this is the difference between blurry, misregistered masks and crisp, pixel-accurate ones. This single fix, called RoIAlign, turned a plausible idea into a production-grade algorithm. It is now adopted across robotics, medical imaging, aerial mapping, and video editing. The article builds the story in three layers. First, the detection backbone inherited from Faster R-CNN. Second, the geometric fix (RoIAlign) that makes per-pixel prediction possible. Third, the mask head that turns aligned features into a spatial mask. We connect this to Semantic Segmentation, which supplies the per-pixel prediction machinery. We also connect it to the two-stage detection framework, which supplies the "propose then refine" machinery. By the end, you will be able to derive the RoIAlign bilinear-interpolation formula. You will also trace a full forward pass by hand. Finally, you will explain precisely why decoupling mask prediction from classification is what makes the whole system work. The chapter closes by pointing toward architectures that later challenged this convolutional, region-based recipe. These include attention-based detectors and Vision Transformers. Such architectures replace hand-designed RoI pooling with global attention over image patches.
- Mask R-CNN = Faster R-CNN (region proposals + box regression + classification) plus a parallel mask-prediction branch.
- RoIAlign replaces RoI Pooling's harsh coordinate rounding with bilinear interpolation, eliminating misalignment between features and the input image.
- The mask branch predicts one small binary mask per class per RoI; classification decides which class's mask to keep.
- Loss is multi-task: classification loss + box regression loss + a per-pixel binary mask loss, added together and trained jointly.
- Decoupling "what class is this" from "which pixels belong to it" avoids inter-class competition and sharply improves mask quality.
- Complexity is dominated by the shared backbone convolution plus a small, fixed-size head per proposal — near-linear in the number of RoIs.
Mask R-CNN turns detection into segmentation by adding a tiny fully convolutional mask head per region, made accurate only because RoIAlign preserves pixel-to-feature spatial correspondence exactly.
The idea
Instance segmentation requires three things simultaneously for every object in an image: a class label, a bounding box, and a pixel-level mask that separates the object from everything else, including other instances of the same class. This is strictly harder than object detection (box only) and strictly harder than semantic segmentation (per-pixel class, no instance separation). Mask R-CNN's central design decision is to treat these as three parallel prediction heads attached to a common set of region proposals, rather than trying to solve segmentation and detection as one entangled problem. The architecture builds directly on the two-stage detector paradigm from the R-CNN family: a backbone convolutional network extracts a feature map from the whole image; a Region Proposal Network (RPN) scans that feature map and proposes candidate object regions (Regions of Interest, RoIs); and for each RoI, per-region heads predict a class, a refined box, and now also a mask. The novelty is twofold. First, an existing RoI feature-extraction step, RoI Pooling, is replaced by RoIAlign, which uses bilinear interpolation to sample features at exact, non-quantized locations. Second, a small fully convolutional network — the mask head — is attached alongside the classification and box-regression heads, predicting an `m × m` binary mask independently for each class. The crucial architectural choice is decoupling mask prediction from class prediction. Instead of asking the mask head to produce one mask and separately deciding what class it belongs to (which invites competition between classes for the same pixels), Mask R-CNN predicts `K` masks per RoI — one for every possible class — and at inference time simply selects the mask corresponding to whichever class the classification branch already chose. Training uses a per-pixel binary cross-entropy loss applied only to the ground-truth class's mask channel, so masks for different classes never compete for the same output.
How it works
Stage 1 — Backbone and Feature Pyramid
An input image is passed through a convolutional backbone (e.g. ResNet-50/101, typically combined with a Feature Pyramid Network, FPN) to produce multi-scale feature maps. FPN merges high-resolution, low-semantic features with low-resolution, high-semantic ones via a top-down pathway with lateral connections, giving strong features at every scale — essential since instances range from a few pixels to filling the frame.
Stage 2 — Region Proposal Network (RPN)
A small convolutional network slides over each feature map level, and at every spatial location predicts, for a set of pre-defined anchor boxes (multiple scales/aspect ratios): 1. an "objectness" score (object vs. background), and 2. four box-regression offsets that adjust the anchor into a tighter proposal. The top-scoring proposals (after non-maximum suppression) become RoIs — typically a few hundred to two thousand per image at inference, and up to 2000 at training time before subsampling.
Stage 3 — RoIAlign
Each RoI, expressed in original-image coordinates, must be converted into a small, fixed-size feature grid (e.g. 7×7 for classification/box, 14×14 for masks) so it can be fed to fully connected or convolutional heads. RoI Pooling (from Fast/Faster R-CNN) did this by rounding the RoI boundaries and bin boundaries to the nearest feature-grid integer — introducing spatial misalignment of up to half a feature-map cell, harmless for coarse box classification but catastrophic for pixel-accurate masks. RoIAlign fixes this by: 1. Not rounding the RoI's coordinates when mapping it into the feature map's coordinate system. 2. Dividing the RoI into `k × k` bins (still using floating-point boundaries). 3. Sampling a fixed number of points inside each bin (commonly 4, arranged as a 2×2 sub-grid) using bilinear interpolation on the surrounding four feature-map cells. 4. Aggregating the sampled points per bin (average pooling) to produce the bin's output value. Bilinear interpolation at a real-valued location `(x, y)` uses the four nearest integer grid points: ``` f(x, y) ≈ Σ over 4 neighbors (i, j) of w_ij · F[i, j] where w_ij = (1 - |x - i|) · (1 - |y - j|) ``` This is a continuous, differentiable function of `x` and `y`, which means gradients can be backpropagated through the exact sampling location — something rounding cannot support.
Stage 4 — Per-RoI Heads
Each aligned RoI feature (e.g. 7×7×256) feeds three heads:
- Classification head: fully connected layers → softmax over `K+1` classes (including background).
- Box regression head: fully connected layers → 4 offsets per class, refining the RoI into a tight box.
- Mask head: a small fully convolutional network (typically 4 conv layers + 1 deconvolution/upsample) operating on the finer 14×14 RoIAlign output, producing a `K × m × m` tensor — one `m×m` binary mask per class (commonly `m = 28`).
Stage 5 — Loss and Training
The total loss for each sampled RoI is: ``` L = L_cls + L_box + L_mask ```
- `L_cls`: standard multi-class cross-entropy over the RoI's class.
- `L_box`: smooth-L1 regression loss, only for foreground RoIs.
- `L_mask`: average binary cross-entropy over the `m×m` grid, computed only on the mask channel matching the ground-truth class — other class channels are ignored entirely for that RoI.
Stage 6 — Inference
At test time: 1. Run backbone + RPN to get proposals. 2. RoIAlign + classification/box heads → final class and refined box per surviving RoI (after NMS). 3. Run the mask head, but read out only the mask channel for the predicted class. 4. Resize that `m×m` soft mask back to the box's size in the original image, threshold (commonly 0.5) to get the binary instance mask. ```pseudocode function MASK_RCNN_INFERENCE(image): features = BACKBONE(image) # e.g. ResNet+FPN proposals = RPN(features) # objectness + box deltas per anchor proposals = NMS(proposals, threshold=0.7) proposals = TOP_N(proposals, N=1000) detections = [] for roi in proposals: aligned_feat_cls = ROI_ALIGN(features, roi, out_size=7) aligned_feat_msk = ROI_ALIGN(features, roi, out_size=14) class_scores, box_deltas = HEAD_CLS_BOX(aligned_feat_cls) class_id = ARGMAX(class_scores) score = class_scores[class_id] if class_id == BACKGROUND or score < SCORE_THRESH: continue refined_box = APPLY_DELTAS(roi, box_deltas[class_id]) mask_logits = MASK_HEAD(aligned_feat_msk) # shape: K x m x m mask_km = SIGMOID(mask_logits[class_id]) # m x m soft mask, this class only mask_full = RESIZE_TO_BOX(mask_km, refined_box) # back to image resolution binary_mask = mask_full > 0.5 detections.append((class_id, score, refined_box, binary_mask)) detections = NMS_ON_BOXES(detections, threshold=0.5) return detections
function ROI_ALIGN(features, roi, out_size):
# roi given in original image coordinates; map to feature-map scale (no rounding!) (x1, y1, x2, y2) = roi / feature_stride bin_w = (x2 - x1) / out_size bin_h = (y2 - y1) / out_size output = zeros(out_size, out_size) for r in 0..out_size-1: for c in 0..out_size-1: samples = [] for (dy, dx) in SUBGRID_OFFSETS(sample_ratio=2): # e.g. 2x2 = 4 points py = y1 + (r + dy) bin_h px = x1 + (c + dx) bin_w samples.append(BILINEAR_INTERPOLATE(features, px, py)) output[r, c] = AVERAGE(samples) return output ```
Complexity & correctness
Time complexity. Let the image have resolution producing a backbone feature map computed once in `O(backbone cost)` — independent of the number of regions, and typically the dominant cost for images with modest object counts. For each of `N` proposals (after NMS, `N` is typically ≤ a few hundred at inference, up to a couple thousand at training before sampling), RoIAlign performs `O(k² · s)` work per RoI (`k×k` output bins, `s` samples per bin, each a constant-time bilinear interpolation), and each head does a constant amount of work on a fixed-size input (7×7 or 14×14 feature grid). So the per-region cost is `O(N · c)` for a constant `c` independent of image size, and total inference cost is: ``` O(backbone(H, W)) + O(N · c) ``` which is near-linear in image size (through the backbone) plus linear in the number of proposals — far cheaper than the naive alternative of running a full segmentation network independently per candidate crop at full resolution. This is the same complexity structure that made Fast/Faster R-CNN efficient: share the expensive convolutional computation across all regions, and keep per-region computation cheap and fixed-size.
Space complexity. Feature maps for the backbone/FPN dominate memory, `O(H · W · C)` for the deepest shared tensor, plus `O(N · k² · C)` for the batch of RoI-aligned features held simultaneously during a forward/backward pass — again linear in the number of proposals, not the image resolution, for the head-level computation.
Why RoIAlign is correct where RoI Pooling is not. RoI Pooling maps a real-valued RoI boundary `x_roi / stride` to a feature-grid coordinate and rounds twice: once when locating the RoI's corners, once when locating each bin's corners. Each rounding step can introduce misalignment up to `0.5` feature-map cells, which at a typical stride of 16 or 32 corresponds to 8–16 original-image pixels — comparable to the width of a thin object part (a bicycle spoke, a finger). Classification is tolerant of this because it only needs the gist of the region; per-pixel mask prediction is not, because a systematic few-pixel shift directly corrupts the boundary of the predicted mask everywhere. RoIAlign removes both rounding steps and instead evaluates the underlying feature function via bilinear interpolation, which is: 1. Exact at the sample points that do align to the grid (interpolation reduces to the grid value when `x, y` are integers), and 2. Continuous and differentiable everywhere else, so gradients flow correctly to backbone features even from non-integer sample locations, unlike max-pooling with hard argmax indices that shift discontinuously under rounding. This is why RoIAlign is not merely "higher resolution" — it removes a systematic bias, and this bias-removal is what the empirical mask-AP gains trace back to.
Why decoupling masks from classes is correct. If the network instead predicted a single `m×m` mask followed by a softmax over classes at each pixel (as in a semantic-segmentation-style head), objects of different classes overlapping the same RoI would compete for probability mass at shared pixels, and the mask head would need to simultaneously solve "what" and "where." By predicting one full mask per class and letting a separately-trained classification head choose which channel to trust, the mask head only ever has to answer the strictly easier question, "given that this is a dog, which pixels are dog," never "is this a dog or a cat." The multi-task loss formalizes this: the mask loss for a training RoI is computed only for the ground-truth class's channel, so gradients for other channels are simply zero for that RoI — there is no cross-class competition signal at all.
Tradeoffs. Mask R-CNN's masks are computed at a fixed low resolution (`m×m`, e.g. 28×28) and then upsampled to the box size, which blurs fine boundary detail on large objects — a known limitation later addressed by boundary-refinement extensions. Its two-stage design (propose, then classify/segment) is also slower than single-stage instance segmentation methods, trading some speed for accuracy and conceptual modularity.
Worked example
Consider an image at 800×800 pixels, backbone stride 16 (so the feature map is 50×50), and a proposed RoI in image coordinates `[x1, y1, x2, y2] = [97, 190, 215, 370]` (a person, roughly 118 px wide, 180 px tall).
Step 1 — map to feature-map coordinates (no rounding): ``` x1' = 97/16 = 6.0625 y1' = 190/16 = 11.875 x2' = 215/16 = 13.4375 y2' = 370/16 = 23.125 ``` Compare to RoI Pooling, which would round these to `(6, 12, 13, 23)` — already a shift of up to 0.4 cells (6.4 pixels) at each edge before any binning happens.
Step 2 — bin the RoI for a 7×7 output (classification branch): ``` bin width = (13.4375 - 6.0625) / 7 = 1.0536 bin height = (23.125 - 11.875) / 7 = 1.6071 ``` For output cell `(row=3, col=2)`, sample the 2×2 sub-grid centers, e.g. one sample point: ``` py = 11.875 + (3 + 0.25) 1.6071 = 17.100 px = 6.0625 + (2 + 0.25) 1.0536 = 8.434 ```
Step 3 — bilinear interpolation at (8.434, 17.100): Neighbors: `(8, 17), (9, 17), (8, 18), (9, 18)` with feature values, say, `F[8,17]=0.20`, `F[9,17]=0.35`, `F[8,18]=0.10`, `F[9,18]=0.05`. ``` dx = 0.434, dy = 0.100 w_00 = (1-0.434)(1-0.100) = 0.5094 → 0.20 w_10 = (0.434)(1-0.100) = 0.3906 → 0.35 w_01 = (1-0.434)(0.100) = 0.0566 → 0.10 w_11 = (0.434)(0.100) = 0.0434 → 0.05 value = 0.50940.20 + 0.39060.35 + 0.05660.10 + 0.04340.05 = 0.1019 + 0.1367 + 0.0057 + 0.0022 = 0.2465 ``` Average this with the other three sub-grid samples in the same bin to get output cell `(3,2) = 0.24` (say). This value is a smooth function of the RoI's exact position — nudge the RoI by half a pixel and this output shifts smoothly too, instead of jumping discretely as it would under RoI Pooling's rounding.
Step 4 — heads. The 7×7×256 aligned tensor feeds the classification head → say, `person: 0.94`, background: 0.02, others low. The box head refines `[97,190,215,370]` to `[101,188,213,366]`. The 14×14×256 aligned tensor (separately computed with finer bins) feeds the mask head, which outputs `K × 28 × 28` logits; we read out only the "person" channel, apply sigmoid, threshold at 0.5, and resize the resulting 28×28 mask to the refined box's `112 × 178` pixel footprint using standard image resizing — yielding the final silhouette.
| Stage | Output |
|---|---|
| RPN | ~1000 proposals, one is our RoI |
| RoIAlign (7×7) | aligned feature grid, no quantization error |
| Classification head | class = person, score = 0.94 |
| Box head | refined box `[101,188,213,366]` |
| RoIAlign (14×14) | finer aligned grid for mask branch |
| Mask head | 28×28 soft mask, "person" channel only |
| Resize + threshold | binary silhouette pasted into the 112×178 box |
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Confusing RoIAlign with "just RoI Pooling with more bins" | Both produce fixed-size grids; the difference (no rounding, bilinear sampling) is subtle but is the actual source of accuracy gains | Always check whether coordinates are quantized anywhere in the pipeline before pooling |
| Using one mask channel with softmax across classes | Feels natural by analogy to semantic segmentation | Predict `K` independent sigmoid masks; select by class, don't blend across classes |
| Training mask loss on all `K` channels per RoI | Naive implementation of multi-class output | Mask loss must be computed only on the ground-truth class channel per RoI |
| Applying NMS on masks instead of boxes | Assuming segmentation should dedupe by mask overlap | Standard Mask R-CNN dedupes proposals by box IoU; mask quality is a downstream concern, not the dedup criterion |
| Ignoring class imbalance in RPN training | Vast majority of anchors are background | Subsample a fixed foreground:background ratio (e.g. 1:3) per training batch |
| Assuming mask resolution scales with object size | `m×m` is fixed regardless of RoI size | Expect blocky/blurry masks on very large objects; consider higher-res mask heads or refinement stages |
Pioneers & history
Mask R-CNN sits at the end of a lineage that runs from region-based detection through fully convolutional segmentation. The R-CNN family (R-CNN → Fast R-CNN → Faster R-CNN) established the propose-then-classify paradigm and the shared-convolution efficiency trick; Fully Convolutional Networks established that segmentation could be framed as dense per-pixel prediction using the same convolutional machinery as classification. Mask R-CNN's contribution was recognizing that these two lines of work could share almost the entire computational graph, differing only in how RoI features are extracted (aligned, not pooled) and in adding one small extra head.
Origins · 1 minKaiming He and how this was discovered
Where this matters
Instance segmentation is the workhorse behind systems that must reason about individual objects, not just regions or classes. Autonomous driving stacks use it to separate adjacent pedestrians and vehicles for tracking and trajectory prediction. Robotic manipulation pipelines use per-instance masks to compute precise grasp points on cluttered bins of objects. Medical imaging tools use instance segmentation to delineate individual cells, nuclei, or tumors in microscopy and radiology scans, where overlapping instances of the same tissue type would defeat plain semantic segmentation. Aerial and satellite imagery pipelines use it to count and outline individual buildings, ships, or trees. Photo and video editing tools (background replacement, object removal, "cut out this person") rely on exactly the same mask-per-instance output. Mask R-CNN's modular head design also directly seeded panoptic segmentation (unifying "stuff" semantic segmentation with "thing" instance segmentation) and remains a standard strong baseline against which newer detection and segmentation architectures — including transformer-based detectors that lead toward Vision Transformers — are benchmarked.
Ask the tutor: How would you modify Mask R-CNN's mask head to produce higher-resolution boundaries on large objects without changing its overall complexity class?
Why can't we just increase RoI Pooling's resolution instead of switching to RoIAlign?
Increasing the pooling grid resolution (say from 7×7 to 28×28) reduces the size of each bin, but it does not remove the fundamental cause of misalignment: RoI Pooling rounds the RoI's own boundary coordinates and each bin's boundary coordinates to the nearest integer feature location before pooling. That rounding error is a fixed offset determined by where the RoI happens to fall relative to the feature grid, not by how many bins you use inside it. Finer bins subdivide a misaligned region more finely, but the region itself is still shifted from where it should be. RoIAlign instead avoids rounding the RoI or bin coordinates at all, and samples the underlying continuous feature function via bilinear interpolation, which removes the systematic bias rather than merely averaging over it.
Does Mask R-CNN require ground-truth masks during training, or can it be trained with boxes only?
Mask R-CNN's mask branch requires per-instance ground-truth masks during training, since its loss is a per-pixel binary cross-entropy comparing predicted masks to annotated silhouettes. The classification and box-regression branches, inherited from Faster R-CNN, only require boxes and class labels. In practice this means the mask head can be trained only on the subset of data with mask annotations, while the detection branches can still benefit from box-only datasets, but full instance segmentation performance depends on having genuine mask supervision for at least the categories of interest.
How does Mask R-CNN handle overlapping instances of the same class?
Because each RoI is processed independently through its own RoIAlign features and its own mask head forward pass, two heavily overlapping proposals (e.g. two adjacent people) each produce their own independent 28×28 mask conditioned only on their own RoI's features. The masks are not forced to be mutually exclusive at the pixel level; each instance's mask is predicted "in isolation" relative to its own bounding box crop of features. Overlaps are resolved implicitly, since each RoI's aligned features are dominated by that instance's appearance within its box, and non-maximum suppression on boxes ensures near-duplicate proposals for the same physical instance are merged before their masks are even computed.
What is the m×m mask resolution, and why not predict a full-resolution mask directly?
The mask head outputs a small, fixed-size grid, commonly 28×28, regardless of the RoI's actual pixel size, and this grid is later resized up to the refined box's footprint in the image. Predicting a fixed small size keeps the mask head's computational cost constant per RoI, independent of object size, which preserves the near-linear complexity in the number of proposals described above. The tradeoff is that very large objects lose fine boundary detail after upsampling from a coarse grid, and very small objects can be prone to aliasing; this limitation motivated later refinements that predict boundary detail at higher resolution or iteratively refine mask edges.
How does Mask R-CNN relate to panoptic segmentation?
Panoptic segmentation unifies two traditions: semantic segmentation of "stuff" classes (sky, road, grass — uncountable, no individual instances) and instance segmentation of "thing" classes (cars, people — countable, separable instances). Mask R-CNN directly supplies the "thing" half of a panoptic system: its detection and mask branches produce individual instance masks with class labels and confidence scores, which are then merged with a separate semantic segmentation branch for "stuff" classes, with conflicts at overlapping pixels resolved by simple priority or confidence rules. Many panoptic architectures literally reuse a Mask R-CNN-style instance branch as one of their two parallel heads.
Why does the article mention Vision Transformers as a foreshadowed direction?
Mask R-CNN's design leans heavily on convolutional inductive biases and hand-engineered geometric operations, such as anchor boxes and RoIAlign's spatial sampling, to connect region proposals to pixel-accurate features. Vision Transformer-based detectors and segmenters replace much of this hand-designed geometry with learned global attention over image patches, letting the network itself learn which regions of the image are relevant to a given query, without explicit anchors or hand-tuned RoI sampling. Understanding why RoIAlign's precise geometric alignment mattered so much for Mask R-CNN helps clarify what transformer-based approaches are implicitly trying to learn instead.
Is Mask R-CNN still used in practice, or has it been superseded?
Mask R-CNN remains a widely used and heavily benchmarked baseline, particularly valued for its modularity, mature open-source implementations, and reliable accuracy on datasets like COCO. Many production systems, especially those needing well-understood, debuggable two-stage pipelines, still deploy Mask R-CNN or close variants (with FPN backbones, Cascade heads, or improved mask heads). Newer single-stage and transformer-based instance segmentation methods often achieve higher accuracy or speed, but Mask R-CNN's core ideas — decoupled mask prediction and precise RoI feature alignment — persist as building blocks even inside many of those newer architectures.