DETR vs. ViT: Two Different Questions, Not One Architecture Plus a Head

The common shorthand is that DETR is "a Vision Transformer with a detection head bolted on." It's a convenient sentence, and it's wrong in a way that matters once you try to actually reason about either architecture. The two models aren't the same backbone answering different questions at the head — they're built to answer different questions from the start, and that difference shows up well before the transformer ever sees a token.

What a Vision Transformer asks

ViT asks a single question: what is this image?

Image
  ↓
Patch Embedding
  ↓
Transformer Encoder
  ↓
CLS Token (summary of entire image)
  ↓
Classification Head
  ↓
1,000 logits

The image is resized to a fixed square (224×224), cut into fixed-size patches, and each patch is embedded as a token. A learned CLS token rides alongside the patch tokens through the encoder and, through self-attention, absorbs a summary of the whole image. By the final layer, that one 768-dimensional vector is treated as a compressed description of everything in the frame.

The math at the head is a single matrix multiply:

(1 × 768) × (768 × 1000) = (1 × 1000)

One image goes in. One prediction comes out. There is exactly one thing to be right or wrong about.

What DETR asks

DETR asks a different question: find every object in this image.

Image
  ↓
CNN Backbone (ResNet)
  ↓
Feature Map
  ↓
Flatten + Positional Encoding
  ↓
Transformer Encoder
  ↓
~800–1,000 image features (varies by image size)
  ↓
Transformer Decoder  ←  100 object queries
  ↓
100 object embeddings
  ↓
Classification Head + Bounding Box Head

Instead of one summary vector, DETR carries 100 learned "object queries" through a transformer decoder that cross-attends to the encoded image. Each query is free to attach itself to a different region and, ideally, a different object: one might converge on a dog, another on a person in the background, most on nothing at all — DETR explicitly predicts a "no object" class for queries that don't find anything worth reporting.

The classification head is shared across all 100 queries, but it runs once per query, not once per image:

(100 × 256) × (256 × 81) = (100 × 81)

Each of the 100 outputs carries its own class prediction and its own bounding box. Where ViT produces one answer, DETR produces a set of 100 candidate answers and lets a bipartite matching loss during training decide which queries are supposed to be responsible for which ground-truth objects.

Two details worth being precise about

The two architectures don't share an "image to tokens" step. ViT patchifies the raw image directly and feeds those patches straight into a transformer encoder. DETR runs a CNN backbone first and only flattens the resulting feature map into a sequence afterward. The divergence isn't at the head — it's at the very first stage, before either model does anything a transformer would recognize as attention.

ViT's patch count is fixed; DETR's token count isn't. ViT resizes every input to the same 224×224 square, so it always produces the same 196 patches, full stop — it's a constant baked into the preprocessing step, not a property of the architecture. DETR keeps roughly the original aspect ratio and downsamples through the CNN backbone at a stride of 32, so the number of tokens fed into the transformer is just the feature map's height times width for that specific image. A wide image and a square image of similar resolution will hand the transformer noticeably different sequence lengths.

The mental model that made it click

ViT is one student who studies the entire image and turns in a single final answer.

DETR is 100 investigators searching the same scene at once, each responsible for finding at most one object — most of them turning up empty, and that's by design, not failure.

Once I stopped treating DETR as "ViT plus another output layer" and started treating it as a fixed set of parallel object hunters operating on a CNN's feature map, the architecture stopped requiring memorization and started following from the question it was built to answer.

I write about the ideas underneath the AI tooling everyone's shipping, in my newsletter, AI Shipped. New issue every week.