Picture this. You show an image to a smart assistant and say "find every car." A human glances once and points to all of them in parallel — eyes flick to each car instantly, no waiting in line. But the way most vision-language models do it in 2025 is closer to a stenographer dictating coordinates out loud, one digit at a time: "first car, x equals… three… two… seven… comma… y equals…" — and only after that whole number is spelled out can it start on the next box.
That spelling-out is the bottleneck. LocateAnything, a paper out of NVIDIA (with collaborators from HK PolyU, Princeton, Nanjing University, and UIUC), kills it. Their trick is called Parallel Box Decoding (PBD), and it lets the model emit a whole bounding box as one atomic unit and many boxes at once. The result is a grounding model that is up to 10× faster than competitors while also being more accurate — which is the rare combination that makes a paper worth squeezing.
"LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding" (arXiv:2605.27365, 2026; corresponding author Zhiding Yu) is the subject. Let's unpack what grounding is, why the old way is slow, what PBD actually changes, and why the numbers are convincing.
First — what is "grounding"?
Grounding is the bridge between language and pixels. You give the model an image plus a text query ("the dog on the left", "all traffic lights", "the Submit button"), and it returns where in the image those things are — usually as bounding boxes $(x_1, y_1, x_2, y_2)$, sometimes as points.
It is the workhorse behind a lot of things you already use: object detection, GUI agents that click buttons, document layout parsing, OCR that knows where each word sits, and any agent that needs to "look and act." If a model can ground reliably and quickly, it can drive a screen, read a form, or count objects in a warehouse photo.
Grounding = "given words, return coordinates." It is the perception layer that every vision agent stands on. Make it fast and accurate, and everything built on top gets faster and more reliable.
The old way, and why it crawls
Modern grounding models are mostly built on top of large language models. An LLM only knows how to do one thing: predict the next token. So to make it output a box, people turned the box into text. The model literally writes out the coordinates as a sequence of tokens — digit by digit, comma by comma — and you parse the string afterward.
This is called autoregressive (AR) decoding, or next-token prediction
(NTP). Each token depends on the one before it, so the model must generate them strictly in
order. To emit one box like (327, 88, 410, 250) you might burn a dozen tokens.
For a dense scene with 200 objects, that's thousands of sequential steps. The GPU spends
most of its time waiting on itself.
The paper points out this design has two distinct problems, not one:
- The inference bottleneck. Serializing 2D geometry into a 1D token stream is inherently sequential. More objects = linearly more steps. Dense detection is painfully slow.
- The geometric mismatch. A bounding box is a 2D object with four interrelated numbers. Forcing it through a 1D left-to-right token pipeline is unnatural — the model predicts $x_1$ with no knowledge of $y_2$, even though they describe the same box. The structure of the data fights the structure of the decoder.
Different prior works attacked these from different angles. Rex-Omni quantizes coordinates into special tokens (fewer tokens per box, but still sequential). Qwen3-VL decodes coordinates as plain text. Both still pay the one-box-after-another tax. LocateAnything's claim is that you can drop the tax entirely.
The core idea: Parallel Box Decoding
Here's the shift. Instead of treating a box as a string of tokens, PBD treats each box (or point) as a single atomic unit. At one decoding step, the model predicts the entire box — all four coordinates $(x_1, y_1, x_2, y_2)$ — at once. And it can predict many boxes in parallel in the same pass.
The analogy: the old way is a single cashier ringing up items one digit at a time. PBD is opening every checkout lane simultaneously — each box gets its own lane, and all the coordinates within a box pop out together rather than being spelled.
Fig 1 — AR decoding spells each box out token-by-token and serializes boxes; PBD emits each box as an atomic unit and many boxes in parallel.
By making the box atomic, PBD also fixes the geometric mismatch. The four coordinates are predicted jointly, so the model reasons about the box as a 2D shape rather than as four unrelated text symbols arriving one at a time.
optional read — AR vs PBD factorisation
Autoregressive decoding factorises a set of $N$ boxes (flattened into a token sequence $t_1 \dots t_M$) as a strict chain:
$$p(t_1, \dots, t_M \mid I) = \prod_{i=1}^{M} p(t_i \mid t_{<i}, I)$$
Every token waits on every prior token — $M$ is large because each box costs many tokens. PBD instead predicts each box $b_j = (x_1, y_1, x_2, y_2)_j$ as a single decoding target and conditions the boxes on the image and a set of parallel query slots rather than on each other:
$$p(b_1, \dots, b_N \mid I) \approx \prod_{j=1}^{N} p(b_j \mid I, q_j)$$
The product now runs over boxes, not tokens, and the terms are computed in parallel. The number of sequential steps stops scaling with coordinate digits and stops scaling linearly with object count in the same brutal way.
The architecture, piece by piece
LocateAnything is not exotic plumbing. It is three well-understood blocks wired together, with PBD as the decoding head on top.
1. Moon-ViT — the eyes
The vision encoder is Moon-ViT, which handles native resolution. That matters more than it sounds. Many encoders force every image down to a fixed square (say 448×448), which smears small objects and tiny UI elements into mush. Native resolution means a 4K screenshot or a dense document keeps its detail, so small buttons and fine text survive into the model's representation. This is a big reason the GUI and document numbers come out strong.
2. MLP projector — the translator
A small multi-layer perceptron maps the visual features into the language model's embedding space. Standard glue, but necessary — the language decoder only speaks "embedding," so the image features have to be translated into that vocabulary.
3. Qwen2.5 — the brain
The language decoder is Qwen2.5. It interprets the text query, fuses it with the visual tokens, and drives the PBD head to produce boxes. Using a strong off-the- shelf LLM means LocateAnything inherits good language understanding for free — it can parse messy referring expressions ("the third person from the left who's wearing red").
Three speeds: Fast, Slow, Hybrid
The clever bit is that LocateAnything doesn't force you to pick parallel-or-sequential forever. It ships three inference modes, and the default is adaptive.
- Fast mode (parallel / MTP): full Parallel Box Decoding. Emit everything at once. This is where the 10× speed lives. Great for clean, well-separated objects.
- Slow mode (autoregressive / NTP): fall back to the classic one-at-a-time decoding. Slower, but the sequential context can resolve tricky cases — heavy overlap, ambiguous instructions, weird formats.
- Hybrid mode (default): run Fast by default, and fall back to Slow only when the model detects trouble — format irregularities or spatial ambiguity. You get fast speed almost always, and the accuracy safety net exactly when it's needed.
Hybrid is the headline product. It's the "fast unless it's hard, then careful" policy — which is exactly how you'd want a perception system to behave. The ablation shows Hybrid hits 51.6 F1 on COCO at 13.2 boxes/sec: near the accuracy of the slow path, near the speed of the fast path.
The data: LocateAnything-Data
A model is only as good as what it's fed, and grounding needs a lot of boxes. The authors built a giant unified corpus to train on.
- 138M queries
- 12M images
- 785M bounding boxes
And crucially it's a mixture of task types, not just COCO-style detection. The composition:
| Task type | Share | What it teaches |
|---|---|---|
| Detection | 66.9% | find all instances of a category |
| GUI | 16.5% | locate buttons, icons, UI elements |
| Referring | 7.3% | resolve "the X that is Y" |
| OCR | 3.6% | find and read text regions |
| Layout | 3.5% | parse document structure |
| Point | 2.2% | pinpoint single locations |
That diversity is why one model can do detection, click GUIs, read documents, and resolve natural-language references — instead of needing a separate specialist for each. The detection-heavy mix anchors the core skill; the smaller slices broaden it.
Do the numbers hold up?
Speed and accuracy usually trade off. The interesting claim here is that LocateAnything wins on both. Start with speed.
Speed (boxes per second on an H100)
- LocateAnything: 12.7 BPS
- Rex-Omni: 5.0 BPS — LocateAnything is 2.5× faster
- Qwen3-VL: 1.1 BPS — LocateAnything is ~10× faster
The denser the scene, the bigger the win — the paper reports a 2–6× speedup on images with 20–300 boxes, because that's exactly where the AR baselines drown in sequential steps and PBD doesn't.
Accuracy — and especially at tight overlap
The headline accuracy result is on LVIS (a long-tail detection benchmark) at a strict IoU = 0.95 — meaning a box only counts as correct if it overlaps the ground truth almost perfectly. LocateAnything scores 31.1 mean F1 versus 20.7 for the comparison, a +3.8% mean F1 jump where it's hardest to score. That's a sign PBD's joint coordinate prediction produces geometrically tighter boxes, not just roughly-right ones.
optional read — what IoU = 0.95 actually demands
Intersection-over-Union measures box overlap:
$$\text{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|}$$
At the common IoU = 0.5 threshold, a box can be quite sloppy and still "count." At IoU = 0.95 the predicted box must share 95% of its area with the truth — almost pixel perfect. Models that predict coordinates loosely (digit-by-digit text, quantized buckets) tend to fall apart here. LocateAnything gaining ground at 0.95 is the strongest evidence that predicting all four coordinates jointly yields genuinely tighter geometry.
The rest of the scoreboard
| Benchmark | What it tests | Score |
|---|---|---|
| COCO | classic detection | +1.8% over Rex-Omni |
| ScreenSpot-Pro | GUI grounding | 60.3 (SOTA) |
| DocLayNet | document layout | 76.8 |
| M6Doc | document layout | 70.1 (+14.5) |
| TotalText | OCR localisation | 43.3 |
| HumanRef | referring on people | 78.7 |
| Dense200 | very dense scenes | 58.7 |
| VisDrone | aerial / tiny objects | 39.9 |
The standout is M6Doc +14.5 — a big jump on document layout, which is exactly the kind of dense, small-element task where native-resolution Moon-ViT plus tight boxes should pay off. ScreenSpot-Pro SOTA means it's the best published GUI grounder, which matters a lot for screen-driving agents.
The ablations — does PBD itself earn its keep?
It's fair to ask whether the wins come from PBD or just from the giant dataset. The ablations isolate PBD.
- PBD in slow mode scores 52.1 F1 on COCO — beating 1D serialization. Same backbone, same data; the only change is treating the box as an atomic unit. It wins. So the geometric framing helps accuracy, not just speed.
- Hybrid hits 51.6 F1 at 13.2 BPS — almost the slow-mode accuracy at near top speed. That's the mode you'd actually ship.
- Coordinate ordering matters: predicting corners in an X-Y order came out optimal. A small detail, but it confirms the model is learning real spatial structure rather than memorising a token string.
The ablation that matters most: PBD beats 1D serialization even in slow mode, where there's no speed advantage. That means PBD is a better way to represent boxes, full stop — the speed is a bonus on top.
How it stacks against the competition
- Rex-Omni — quantizes coordinates into discrete tokens. Fewer tokens than plain text, but still sequential box-by-box. LocateAnything beats it on COCO (+1.8%) and is 2.5× faster.
- Qwen3-VL — decodes coordinates as raw text. Most flexible, slowest (1.1 BPS). LocateAnything is ~10× faster.
- GUI-Owl-32B — a much larger GUI specialist, yet LocateAnything (a 3B model) takes SOTA on ScreenSpot-Pro. Small + well-designed beats large + general here.
Where it's honest about limits
A few things worth keeping in mind before you assume this solves grounding forever:
- Parallel decoding can struggle with heavy overlap. When many boxes pile on top of each other, the parallel slots can collide or duplicate — which is precisely why the Slow/Hybrid fallback exists. Hybrid is a mitigation, not a proof that parallel always wins.
- It leans on a massive proprietary-ish dataset. 785M boxes is a lot of curation. Some of the gain is data, and reproducing the full recipe without that corpus is non-trivial (the dataset release was still pending at paper time).
- 3B is small for a reason, but still a VLM. Native-resolution encoding of 4K images is not free; the speedups are measured on an H100. Real-time on edge hardware is a different story.
Why this paper is worth your time
Most grounding papers chase a benchmark with a bigger model or more data. LocateAnything instead questions a design assumption everyone inherited from LLMs: that a 2D box should be spelled out as a 1D token string. Once you reject that, both problems — speed and geometry — improve at the same time. That's the mark of a good idea: it doesn't trade one thing for another, it dissolves the false choice.
The practical upshot: if you're building a screen agent, a document parser, or anything that needs to localise many things quickly, a 3B model running at 12.7 boxes/sec with SOTA GUI accuracy is a very attractive perception layer. Fast enough to sit in an agent loop, accurate enough to trust the boxes.
Project page · arXiv:2605.27365 — LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding. Code on NVlabs/Eagle, weights as nvidia/LocateAnything-3B on Hugging Face, with a live demo on HF Spaces.