NVIDIA LocateAnything: Parallel Box Decoding Makes Visual Grounding a One-Step Decode

A Grounding Specialist That Treats the Box, Not the Token, as the Unit of Generation

Most vision-language models bolt detection on as an afterthought: you ask where the red mug is, and the model writes out coordinates one digit at a time while your request times out. LocateAnything, released by NVIDIA's research labs with collaborators from PolyU, Princeton, Nanjing University, and UIUC on May 26, 2026, refuses that framing. It is a 3.83-billion-parameter VLM — Qwen2.5-3B-Instruct as the language core, MoonViT-SO-400M as the vision encoder, an MLP projector between them — whose entire output formulation is built around generating a complete bounding box in a single decoding step. The paper is on arXiv as 2605.27365, the code lives in the Embodied directory of NVlabs/Eagle under Apache 2.0, and the weights ship as nvidia/LocateAnything-3B on Hugging Face under NVIDIA's non-commercial research license. In June the work was accepted to ECCV 2026.

I evaluate models like this from one specific seat: the perception component of production agents. A GUI agent that clicks things needs to know where the search button is on every step of its loop. A labeling pipeline needs to propose ten thousand boxes overnight. In both cases, grounding latency and grounding quality set the ceiling on the whole system, and a generalist chat VLM serializing coordinates digit by digit is a bad fit for both. LocateAnything attacks exactly that seam. The model card also notes it has already been folded into NVIDIA's Nemotron 3 Nano Omni and the computer-use and visual-grounding features of Nemotron and Cosmos, so this is a paper plus a shipped component, not a lab curiosity.

The task surface is one set of weights covering six families: open-vocabulary and dense object detection, phrase and referring-expression grounding, GUI element grounding, OCR text localization, document layout grounding, and point-based localization. Same weights, one prompt template per task, one output grammar to parse. That consolidation is the operational appeal: instead of maintaining a detector for products, a GUI-pointer model, an OCR boxer, and a layout parser, you run one 3B model with different prompts and one parser.

The Serial Bottleneck Hiding Inside Every VLM Detector

The problem LocateAnything targets is architectural, not a tuning issue. Standard VLMs formulate grounding as coordinate-token generation: a 2D box becomes a sequence of 1D tokens, each learned and decoded largely independently, one after another. That design mismatches the geometry it is trying to express — x2 is only meaningful relative to x1, y2 relative to y1 — and it imposes a strictly sequential dependency chain at inference time. Every coordinate digit is a decoder step, and dense images multiply the sentence length mercilessly.

The paper's throughput table makes the cost concrete. Measured in boxes per second (BPS) on a single H100 at batch size 1, textual-coordinate decoding in Qwen3-VL manages 1.1 BPS, quantized-coordinate decoding in Rex-Omni reaches 5.0 BPS, and LocateAnything's hybrid mode hits 12.7 BPS. To translate: at 1.1 BPS, a 200-box aerial frame costs roughly three minutes of pure decoding; at 12.7 BPS it is about sixteen seconds. That gap is my arithmetic on their numbers, but the ratio is the entire product pitch, and it is the difference between a nightly batch job and an interactive loop.

The less obvious claim is that serialization also hurts accuracy, and the strongest evidence shows up at high IoU thresholds. On LVIS at IoU 0.95 — meaning predicted boxes must overlap ground truth almost perfectly — LocateAnything scores 31.1 F1 against Rex-Omni's 20.7. Token-by-token generation lets errors accumulate across a box's span; predicting the box as one atomic unit keeps its internal geometry consistent by construction. Speed and precision improvements here come from the same root cause, which is rare and worth internalizing before you reach for the usual "fast but sloppier" mental model.

Parallel Box Decoding: One Forward Pass per Box

The core mechanism, Parallel Box Decoding (PBD), changes the unit of generation. Each bounding box — or point — becomes an atomic unit of constant length, and the model emits the full coordinate set (x1, y1, x2, y2) in one parallel step. There is no arbitrary chunking of a coordinate into digits, no sequential handoff inside a box.

The output grammar makes this concrete. A detection looks like <ref>label</ref><box><x1><y1><x2><y2></box>, where coordinates are integers in [0, 1000] that you divide by 1000 for relative positions. A point is <box><x><y></box>, and an empty result is the literal <box>none</box>. Internally, outputs are organized into fixed-length blocks of six — Semantic, Box, Negative, and End blocks, with unused positions padded by <null> tokens — so the decoder always knows exactly how many positions constitute one geometric unit. The authors also ablated box ordering and found X-Y corner order gives the highest F1 among the four spatial orderings they tried, which tells you the block layout is a first-class design surface, not an encoding afterthought.

One subtlety I want to flag because it defeats a lazy interpretation of the paper: the win is not "multi-token prediction go brrr." The ablation compares PBD against a structure-agnostic MTP method (SDLM-B6) that also predicts multiple tokens per step but without box-aligned structure. PBD reaches 16.9 BPS against 5.5 BPS while improving F1. The box-aligned supervision is what unlocks the parallelism; generic MTP on a serialized stream just produces irregular distributions. And jointly training both formulations raises the ceiling of the autoregressive path itself from 50.1 to 52.1 F1 on COCO, so the Slow mode of this model is not merely a legacy fallback — it is the best pure-decoding accuracy they measured.

The training side follows a four-stage pipeline: initial multimodal knowledge adaptation on captioning, VQA, and OCR data, then grounding and dense-scene localization fine-tuning, with next-token prediction and multi-token prediction optimized jointly. The released fine-tuning entry point is a continual SFT script with block_size 6, learning rate 2e-5, 25,000 steps, ZeRO stage 2, and 16K sequence length under Magi attention, plus a LoRA path (default LLM rank 64, projector trainable, backbones frozen) for cheaper adaptation.

Hybrid Mode: Parallel Speed With an Autoregressive Safety Net

Three inference modes ride on the same weights. Fast Mode is pure MTP: every box in one step, maximum throughput, intended for latency- and compute-constrained settings like robotics loops. Slow Mode is pure NTP: autoregressive coordinate decoding, maximum stability, aimed at high-precision labeling and offline evaluation. Hybrid Mode — the default, and what all headline benchmarks use — runs Fast Mode and falls back to Slow Mode exactly where parallel output looks unreliable.

The fallback triggers are two named failure classes. Format irregularity is malformed syntax at category boundaries — think a broken <ref> or <box> scaffold. Spatial ambiguity is an intermediate coordinate landing between densely arranged objects. When the verifier detects either, the compromised block is discarded, generation rewinds to the last verified prefix, and the NTP path regenerates the problematic block token by token before MTP resumes. In the COCO ablation, Hybrid holds 51.6 F1 at 13.2 BPS against Slow Mode's 52.1 F1 — you give up half an F1 point and keep most of the speed. The model card's own recommendation is max_new_tokens=8192 with generation_mode="hybrid" to avoid truncated responses, which I read as the authors telling you where the production default belongs.

sequenceDiagram
    participant App as locateanything_worker
    participant Dec as Qwen2.5 decoder in Hybrid Mode
    participant Ver as Block verifier
    participant Ntp as NTP re-decoder
    App->>Dec: visual tokens plus task prompt
    Dec->>Ver: parallel block, full box in one step
    Ver-->>Dec: verified, continue in MTP
    Dec->>Ver: next parallel block
    Ver-->>Ntp: format irregularity, reject block
    Ntp->>Ntp: rewind to last verified prefix
    Ntp->>Ver: regenerate failed block token by token
    Ver-->>Dec: block verified, resume MTP
    Dec-->>App: complete box stream
    App->>App: parse ref and box tokens, scale 0-1000 coords to pixels

What I like about this design is that it treats decoding failures as a routing problem rather than a sampling problem. Instead of tuning temperature and hoping, it detects the two ways parallel decoding actually breaks and pays the autoregressive cost only on the blocks that need it. That is the same shape as a retry budget on a flaky service, and it means your worst-case latency is bounded by the fraction of blocks that fail verification, not by the whole response.

From Image to Parsed Boxes: The Full Inference Path

End to end, an inference looks like this. The image goes in at native resolution — the model card lists production resolution up to 2.5K — and MoonViT extracts visual tokens preserving fine-grained spatial detail, with prompts supported up to 24K tokens. The MLP projector injects those tokens into the Qwen2.5-3B decoder stream, which emits the length-6 block sequence under the verifier's watch. What comes out the other side is ordinary text in a rigid grammar, which is the integration gift here: your wrapper is a regex, not a vision pipeline.

flowchart TD
    A["RGB image at native resolution, up to 2.5K"] --> B["MoonViT-SO-400M vision encoder"]
    Q["Text query: categories, phrase, GUI instruction, or pointing request"] --> C["MLP projector"]
    B --> C
    C --> D["Qwen2.5-3B decoder emits length-6 blocks: Semantic, Box, Negative, End"]
    D --> E{"Verifier: syntax complete and geometry plausible?"}
    E -- "yes" --> F["Emit next full box in one MTP step"]
    E -- "no" --> G["Discard block, rewind to last verified prefix"]
    G --> H["NTP re-decodes the failed block"]
    H --> F
    F --> I["Parser over ref/box grammar, coordinates 0-1000"]
    I --> J["Scaled boxes, points, or none"]

The released worker API keeps this boring, in the good sense. worker.detect(img, ["person", "car"]) for category detection, worker.ground_multi(img, "people wearing red shirts") for phrase grounding, worker.detect_text(img) for scene text, worker.ground_gui(img, "the search button", output_type="point") for GUI targets, worker.point(img, "the traffic light") for pointing, and worker.detect_batch([...]) on the batched runtime with a pipeline scheduler. Parsing is a five-line regex over <box><d><d><d><d></box> matches divided by 1000 and scaled to pixel dimensions — the README ships exactly that snippet, and I would treat it as the stable contract to build a typed wrapper around: text in, list of pixel boxes out, everything else your code.

For throughput deployments there is a separate batch release: batch_infer.py as a JSONL/image-query CLI, batch_utils with the hybrid MTP/NTP scheduler, and kernel_utils providing the la_flash attention path built on FlashAttention varlen sparse range plans — no custom C++/CUDA extension to compile. One honest limitation to note in the same breath: coordinates are quantized to integers in [0, 1000], so on a 3840-pixel-wide frame each quantization step is about 3.84 pixels. For GUI clicking and document layout that is ample; for sub-pixel measurement tasks it is not the right tool, and no amount of prompt engineering changes quantization granularity.

The Data Engine Behind 785 Million Boxes

Model architecture gets the headlines, but the training corpus is doing as much work. LocateAnything-Data — public on Hugging Face as NVEagle/LocateAnything-Data — spans 12 million unique images, 138 million language queries (the model card rounds to roughly 140M), and 785 million boxes across natural scenes, robotics, driving, GUI interaction, and documents. The task mix is deliberately lopsided:

Task	Share of queries	Share of boxes	What it buys the model
General object detection	66.9%	83.1%	Dense coordinate supervision, the core skill
GUI element grounding	16.5%	not broken out	Click-target precision for computer-use agents
Referring comprehension	7.3%	not broken out	Mapping free-form phrases to specific regions
Text localization (OCR)	3.6%	not broken out	Tight boxes around glyphs and lines
Layout grounding	3.5%	not broken out	Document and scene structure reasoning
Point-based localization	2.2%	not broken out	Sub-box precision for pointing

All remaining box mass lives in the non-detection rows — detection's 83.1% share means everything else shares 16.9% of 785M. The label provenance matters if you plan to fine-tune: annotations are a hybrid of human and open-source origins plus model-assisted and synthetic labels generated with Qwen3-VL, Molmo, SAM 3, and Rex-Omni, with automated post-verification. That is a detector-distillation pattern — the best available predictors teach the next one — and it means the failure modes you should probe on your own domain are inherited teacher biases: systematic misses on teacher-blind categories, box-style artifacts on text and GUI elements where synthetic labels dominate, and English-centric phrasing, since the card describes queries as typically English.

For anyone building labeling automation, the useful takeaway is that the corpus composition tells you where the model is strongest out of the box. It has seen an order of magnitude more generic detection supervision than GUI or pointing supervision, so expect its density and high-IoU behavior to transfer best, and budget fine-tuning budget for niche pointing or unusual document layouts rather than assuming the 3B parameters generalize everywhere.

What the Benchmark Numbers Actually Say

The headline results, all under Hybrid Mode on a single H100 at batch size 1, from the paper and the repository's results table:

Benchmark	Metric	LocateAnything-3B	Best baseline in the paper's comparison
Throughput (H100)	BPS	12.7	Rex-Omni 5.0, Qwen3-VL 1.1
LVIS	F1, mean	50.7	+3.8 vs. Rex-Omni
LVIS at IoU 0.95	F1	31.1	vs. Rex-Omni 20.7
COCO	F1, mean	54.7	+1.8 vs. Rex-Omni
Dense200	F1, mean	58.7	+0.4 vs. Rex-Omni
VisDrone	F1, mean	39.9	+1.4 vs. Grounding DINO Swin-T
DocLayNet	F1, mean	76.8	+6.1 vs. Rex-Omni
M6Doc	F1, mean	70.1	+14.5 vs. Rex-Omni
TotalText (OCR)	F1, mean	43.3	+2.7 vs. Rex-Omni
ScreenSpot-Pro	mean F1	60.3	+2.3 vs. GUI-Owl-32B
HumanRef	F1 at 0.95	68.8	+3.4 vs. Rex-Omni
RefCOCOg val	F1, mean	76.7	+2.0 vs. Qwen3-VL-8B

Three observations from reading this sheet rather than skimming it. First, the high-IoU rows are the real story: beating a same-size competitor by 3.8 mean F1 on LVIS is good, but 31.1 versus 20.7 at IoU 0.95 is the difference between boxes you can crop with and boxes you need to post-process. High-IoU quality is exactly what matters for auto-labeling pipelines where a human reviews rather than redraws.

Second, density scaling behaves the way the architecture predicts. As target boxes grow from 20 to 300, NTP methods hit a severe latency wall while PBD sustains a 2x-to-6x speedup, with throughput climbing from 12 to roughly 25 BPS in dense scenes. More objects should mean more work, and here more objects make parallel decoding look better, not worse. Combined with VisDrone 39.9 and Dense200 58.7 mean F1, drone and aerial imagery is a natural fit.

Third, the GUI row is the one that matters for the agent category this blog tracks: 60.3 mean F1 on ScreenSpot-Pro, above both the 32B-specialist GUI-Owl-32B and generalists like Qwen3-VL-30B-A3B, with icon-based queries the strongest suit. Document work shows the same pattern at larger margins — M6Doc plus 14.5 over Rex-Omni — because layout boxes reward exactly the geometric coherence PBD preserves.

Two caveats I hold while reading all of this. The evaluation framework credits the Rex-Omni team, so baselines were run in an adapted harness rather than each model's own — reasonable, but it means I would re-verify any number I build a business case on. And accuracy is F1 at IoU 0.5 and 0.95 plus mean IoU over roughly 48K box-eval and 35K point-eval images; point success is containment-based (point falls inside the ground-truth mask or box), which is a weaker bar than box overlap. Neither caveat is damning; both belong in your notes before you quote the sheet upward.

Running It: Hardware, Licenses, and the Gaps in the Release

The practical envelope, all from the repo and model card. Supported hardware spans Ampere, Hopper, Blackwell, and Lovelace — A100, H100, L40, RTX 4090 — on Linux only. Runtime is plain Transformers with BF16 and KV cache; TensorRT, TensorRT-LLM, and Triton are explicitly not yet supported, so there is no serving-stack shortcut yet. Dependencies are pinned (transformers==4.57.1, deepspeed==0.15.4, accelerate==1.5.2, peft==0.12.0, liger_kernel==0.3.1, among others), and the stock long-context attention path (MagiAttention v1.0.5, for 16K–32K+ contexts) only builds on Hopper and Blackwell.

If you are not on Hopper, the la_flash batch runtime is the path, and its numbers are the most operationally interesting figure in the whole release. On their A100 probe — a real 3840x2160 street image, batch size 4, 25,600 input-token limit, hybrid MTP — the stock dense SDPA path took 8.26 seconds with 35.12 GB peak reserved memory, while la_flash took 8.03 seconds with 11.71 GB. Same latency, a third of the memory, no custom CUDA build. That 11.71 GB figure is what puts this model on a 24 GB consumer or workstation card for real 4K inputs, and it is the difference between a demo and something your team can actually run. Note the boundary: la_flash is for inference and evaluation only; training stays on the standard model code path.

The license split needs to be on every planning doc. Code is Apache 2.0. The model is NVIDIA License: academic and non-profit research use only, commercial use not permitted except for NVIDIA and its affiliates, and fine-tuned derivatives inherit the constraint. A labeling service or a shipped product feature built on these weights is off-limits without an arrangement with NVIDIA, regardless of how the code is licensed.

Two functional gaps in the current release. The public nvidia/LocateAnything-3B weights do not support visual-prompt inference out of the box — the fine-tuning script for visual prompts ships, but official visual-prompt-capable weights are promised for a future version. So if your use case is "show a crop, find all instances like it," you fine-tune yourself today or wait. And embedded deployment, say on NVIDIA Thor for a robot, is described as possible only with additional quantization, compression, or distillation — the Fast Mode pitch mentions on-device robotics, but the released 3B BF16 checkpoint is not itself an embedded artifact.

Where I'd Point LocateAnything First — and Where I Wouldn't

Situation	My pick	Why
GUI agent needs click targets every loop iteration	LocateAnything Hybrid, point output	60.3 mean F1 on ScreenSpot-Pro with icon queries the strong suit; block-parallel decode keeps the perception step off the critical path
Bulk pre-labeling of 100K+ images	Hybrid or Slow Mode in batch on H100/A100 with la_flash	Highest-F1 mode where review budget matters, 11.71 GB peak on 4K inputs elsewhere; box output parses straight into your schema
Dense aerial or drone frame analysis	Hybrid	VisDrone 39.9 and Dense200 58.7; throughput scales 2x–6x as box count grows instead of collapsing
Document ingestion: layout routing plus OCR localization	Hybrid	DocLayNet 76.8 and M6Doc 70.1; one model covers both layout and text boxes
Occasional "where is X" inside a consumer app	A hosted generalist VLM	At low volume, a 3B specialist's ops overhead buys you nothing
Commercial product shipping detection	Wait or negotiate with NVIDIA	Weights are non-commercial; Apache 2.0 covers code, not checkpoints
Find-more-like-this-crop queries	Not yet supported	Released weights lack visual-prompt inference; use the shipped fine-tune script or wait for the promised weights

Concretely, the three builds I would actually attempt with this: first, a computer-use agent's perception service — point output on GUI frames behind a small cache, hybrid mode, batched when the agent prefetches; the benchmark profile says this is the intended center of mass. Second, a warehouse-scale pre-labeler for detection datasets that writes <ref>label</ref><box>... parses directly into annotation-tool format and flags low-confidence boxes for human review instead of full redraws — the high-IoU numbers are the economic argument there. Third, a document pipeline that uses layout grounding to route pages (invoice versus contract versus form) and OCR localization to hand cropped text regions to a recognition model; one model, two jobs it beats the same-size alternative on by double digits.

The failure modes I'd instrument from day one: quantization-aware tolerance in any pixel-exact downstream step, English-only phrasing in query handling, the visual-prompt gap if stakeholders assume crop-based search exists, and decoder-step latency telemetry so you can see what fraction of blocks fall back to NTP in production — if that fraction climbs on your domain, your real throughput is drifting toward Slow Mode and your capacity plan should know before your users do.

Resources

• LocateAnything project page, NVIDIA Research
• LocateAnything paper, arXiv:2605.27365
• NVlabs/Eagle, Embodied directory (code, Apache 2.0)
• nvidia/LocateAnything-3B model card, Hugging Face