TrianguLang

Abstract

Localizing objects and parts from natural language in 3D space is essential for robotics, AR, and embodied AI, yet existing methods face a trade-off between the accuracy and geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically plausible but geometrically inconsistent matches without requiring ground-truth poses. Validated on seven benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided segmentation and localization, reducing user effort from O(N) clicks to a single text query. The model processes each frame at 1008×1008 resolution in ~58ms (~17 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications.

TrianguLang in the Wild

Explore Gaussian splat reconstructions and highlight objects by text query. Select a scene, click an object name to see TrianguLang's 3D relevancy overlay. Use mouse to orbit and scroll to zoom.

How It Works

TrianguLang takes uncalibrated RGB images and a single text query (e.g. "the red mug") and produces per-view segmentation masks plus a 3D centroid in world coordinates. No camera poses, no SLAM, no per-scene optimization.

The pipeline composes two frozen foundation models with a lightweight trained decoder: SAM3 (frozen, 848M) extracts text-conditioned semantic features, MapAnything (frozen, 1.23B) jointly estimates metric depth, intrinsics, and extrinsics from the input images, and the GASA decoder, together with a fine-tuned SAM3 pixel decoder and instance head (trained, 15.4M total), fuses them with geometry-aware cross-view attention. Only 0.74% of the 2.09B total parameters are trained. The geometry model is a swappable front-end — DA3-NESTED and π³ reach comparable accuracy.

Overview of the TrianguLang architecture

Standard cross-attention matches features by semantic similarity alone, producing false correspondences between visually similar but spatially distant regions (e.g. two identical mugs). GASA introduces a geometric veto: each token carries a 3D position from depth unprojection, and a learned distance kernel penalizes attention between tokens that are far apart in metric space. If two features look similar but are meters apart, the geometric bias suppresses that match.

We also replace standard 2D positional encodings with world-space positional encoding: each pixel is unprojected to 3D using the predicted depth and camera parameters, then encoded with sinusoidal functions. The same physical point gets the same embedding regardless of viewpoint.

Spatial qualifiers like "nearest chair" or "mug left of the keyboard" are handled through direct geometric computation on depth-derived 3D positions, no LLM required. The pipeline parses spatial keywords, generates mask candidates, computes 3D centroids via depth unprojection, and selects the candidate satisfying the constraint. This runs in ~60ms, compared to 1-10+ seconds for LLM-based spatial reasoning.

Spatial language demo (static/images/spatial_demo.png)

Results

On ScanNet++, a single text query achieves 62.4% mIoU, beating MV-SAM's 51.0% with 12 click prompts (+11.4 points). The same ScanNet++-trained model then transfers zero-shot, outperforming SAM3 on every open-vocabulary benchmark we test — 86.8% on uCO3D, 78.8% on LERF-OVS, 94.2% on 3D-OVS, and 79.5% on Mip-NeRF360 — and reaching state-of-the-art zero-shot segmentation on LERF-OVS and Mip-NeRF360, all at ~58ms per frame with no per-scene optimization (three orders of magnitude faster than per-scene language-field methods).

Mask selection is not a bottleneck. Comparing predicted masks against an oracle that always picks the ground-truth-best candidate, TrianguLang loses just 1 point (63.4% → 62.4%), whereas SAM3 loses 30 points (79.9% → 49.8%) because it generates hundreds of candidates per scene and cannot reliably rank them. Despite a much higher oracle ceiling, SAM3 ends up 12.6 points below TrianguLang in practice.

Model	Setting	Train → Eval	mIoU ↑	mAcc ↑
MV-SAM	In-domain	ScanNet++ → ScanNet++	0.510	0.694
	In-domain	uCO3D → uCO3D	0.910	0.965
	Cross-domain	uCO3D → ScanNet++	0.194	0.251
	Cross-domain	ScanNet++ → uCO3D	0.322	0.517
	Large-scale	SA-1B → ScanNet++	0.489	0.635
	Large-scale	SA-1B → uCO3D	0.877	0.950
TrianguLang (Ours)	In-domain	ScanNet++ → ScanNet++	0.624	0.774
	In-domain	uCO3D → uCO3D	0.946	0.983
	Cross-domain	uCO3D → ScanNet++	0.279	0.685
	Cross-domain	ScanNet++ → uCO3D	0.868	0.875

Zero-Shot Open-Vocabulary Generalization

Trained only on ScanNet++, TrianguLang transfers zero-shot to object-centric (uCO3D), tabletop (3D-OVS), and unbounded outdoor (Mip-NeRF360) scenes that look nothing like its indoor training data, using raw text queries and no per-scene optimization. It outperforms SAM3 on every one.

Method	uCO3D	LERF-OVS	3D-OVS	Mip-NeRF360
SAM3	82.4	71.0	87.4	77.7
TrianguLang	86.8	78.8	94.2	79.5

Zero-shot mIoU (%) from the ScanNet++-trained model, text-only prompts, no per-scene optimization.

3D Open-Vocabulary Segmentation

Evaluating directly in 3D rather than on 2D masks, we lift per-frame predictions onto the ScanNet++ laser-scan mesh and score per-vertex semantic IoU. TrianguLang reaches 27.9% f-mIoU, on par with SceneSplat (28.4%), the strongest generalizable language-Gaussian method, and far above per-scene methods. The difference is what each method needs as input: TrianguLang requires neither per-scene optimization nor a Gaussian reconstruction.

Method	f-mIoU	Per-scene Opt.	Needs 3DGS
LangSplat	1.3	Yes	Yes
Feature-3DGS	8.0	Yes	Yes
SceneSplat	28.4	No	Yes
TrianguLang	27.9	No	No

Foreground per-vertex f-mIoU (%) on ScanNet++, excluding wall/floor/ceiling. SceneSplat and ours use the full ~50-scene validation set; LangSplat and Feature-3DGS are the 10-scene mini-val as benchmarked by SceneSplat++.

Ablations

Each component matters. Removing the GASA kernel drops mIoU by 5.3 points; removing world-space PE drops it by 5.4. Removing both costs 8.4 points, less than the sum of the two, so the components carry partly overlapping geometric information. Swapping the learned distance kernel for a fixed RBF costs 10.7 points. The model runs at ~58ms per frame (1008×1008, single A100), while per-scene methods require 30-60 minutes per new scene.

Against alternative geometric encodings under a matched GASA recipe, our world-frame 3D PE is comparable to Plücker coordinates and clearly ahead of the relative-camera encodings PRoPE and RayRoPE, which inject intrinsics and extrinsics into attention.

Positional Encoding	mIoU
World-frame 3D PE (ours)	41.4
Plücker	40.2
PRoPE	36.8
RayRoPE	23.0

Held-out ScanNet++ (50 scenes, 8 views, 504 resolution), all using the GASA decoder.

GASA derives its 3D pointmaps from a frozen visual geometry model, and it is not tied to a particular one. Swapping the backbone while keeping the decoder fixed changes accuracy by less than a point, so the geometric reasoning transfers across depth models rather than depending on any single one.

Geometry Backbone	ScanNet++ mIoU
MapAnything	63.0
π³-VO	62.7
DA3-NESTED (reported system)	62.4

In-domain ScanNet++ mIoU (%) across geometry backbones under a matched recipe. All three land within 0.6 points.

Segmentation results on ScanNet++ and uCO3D

Text-guided segmentation on ScanNet++ (left) and uCO3D (right). Each row, left-to-right: RGB input, predicted depth, ground truth, TrianguLang prediction, overlay.

Qualitative comparison on LERF-OVS. LERF and LangSplat-V2 produce diffuse activations while TrianguLang tightly focuses on the target (~58ms vs. 10–45 min).

Zero-shot open-vocabulary segmentation on Mip-NeRF360 and 3D-OVS

Zero-shot open-vocabulary segmentation on Mip-NeRF360 (left) and 3D-OVS (right), from the ScanNet++-trained model with raw text queries and no per-scene optimization.

Segmentation on SPIn-NeRF (91.4% mean IoU). TrianguLang produces view-consistent masks using only text prompts.

Spatial disambiguation on NVOS. “dino” (97.6% IoU) vs. “leftmost dino” (95.8% IoU) using depth-derived spatial reasoning.

Citation

@misc{grant2026triangulanggeometryawaresemanticconsensus,
      title={TrianguLang: Geometry-Aware Semantic Consensus for Pose-Free 3D Localization},
      author={Bryce Grant and Aryeh Rothenberg and Atri Banerjee and Peng Wang},
      year={2026},
      eprint={2603.08096},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.08096},
}