The first time I read a detection paper after only knowing classification, it felt like a different field. Anchor boxes, non-max suppression, feature pyramids — none of it resembled the ResNet-and-softmax pipeline I'd learned first. It took building a few of these systems end to end to notice that it wasn't actually a different field. It was the same five-stage pipeline with a different last stage bolted on.
The five stages
Input → Preprocessing → Backbone → Neck (optional) → Task Head → Post-processingInput. An image, or a batch of frames, arrives in whatever resolution and format the source produces.
Preprocessing. Resize to a fixed input shape, normalize pixel values, and — during training only — apply augmentation (crops, flips, color jitter) so the model doesn't memorize the exact pixel statistics of the training set.
Backbone. A convolutional network (ResNet, EfficientNet) or a vision transformer (ViT) turns pixels into a feature map: a compressed spatial representation where nearby patterns in the image show up as similar activations. This is the expensive part, and it's almost always pretrained — trained once on a huge dataset like ImageNet, then reused, because learning good visual features from scratch on a small task-specific dataset rarely works as well.
Neck. Some architectures insert a stage that combines features at multiple scales — a Feature Pyramid Network, for instance — so the model can see both fine detail and broad context at once. Classification often skips this; detection and segmentation usually need it, because objects show up at wildly different sizes in the same image.
Task head. This is the only stage that actually differs between "tasks." A classification head is a single dense layer mapping pooled features to class scores. A detection head predicts a box location and a class per candidate region. A segmentation head predicts a class per pixel. Everything upstream of the head is shared machinery.
Post-processing. Detection heads emit far more candidate boxes than there are real objects, so non-max suppression collapses overlapping predictions into one. Segmentation masks often get cleaned up with thresholding. Classification just takes the top score.
Why this framing matters
Once you see it this way, three things that used to look unrelated collapse into the same idea:
- Transfer learning is reusing the backbone and swapping the head — because the backbone learned generally useful visual features, and the head is where task-specific behavior lives.
- Multi-task models are one backbone feeding several heads at once — a single feature map doing double duty for, say, classification and segmentation, because the expensive part of the network doesn't need to be duplicated per task.
- Foundation vision-language models (CLIP and its descendants) are what you get when the "head" stops being a fixed classifier and becomes a comparison against text embeddings instead — which is exactly why they can classify categories they were never explicitly trained on. The backbone-to-features idea didn't change; what changed is what sits on top of it.
A practical takeaway
When you're evaluating a new vision architecture or paper, the fastest way to understand it isn't to read the whole thing top to bottom — it's to identify which of the five stages it's actually innovating on. Most papers change exactly one stage (a new backbone, a new neck, a new head) and leave the rest standard. Once you know which slot the contribution goes in, the rest of the paper reads as implementation detail rather than a new mental model.
I write about the ideas underneath the AI tooling everyone's shipping, in my newsletter, AI Shipped. New issue every week.