Plann3r: Predicting Planning Costs Grounded in 3D

Abstract

Planning paths or trajectories for robot navigation requires understanding scene geometry and traversability. Classic approaches relied on accurate 3D maps to define occupancy based planning costs. Learning based alternatives predict planning costs either in terms of distance to goal or temporal distance between images. However, the former overfits to scene layout and the latter lacks geometric understanding. Moreover, most of these methods estimate image-level scalar costs, which are not sufficiently informative to effectively guide the robot. In this paper, we propose Plann3r as a 3D grounded method to predict pixel-level planning costs in terms of geodesic distances for any given set of images with an arbitrary subgoal pixel. We integrate Plann3r within a navigation pipeline, dubbed VGGT-Nav, in two unique ways: the offline mapping cum global planning phase iteratively uses Plann3r to generate subgoals and reference image global costmaps; and in the execution phase Plann3r performs simultaneous localization and local planning to generate planning costmaps to directly condition a learnt control policy. We show that the proposed Plann3r and VGGT-Nav largely outperform baselines respectively on the planning and navigation tasks on an existing prior-traversal navigation benchmark. We further show real world demonstrations that highlight sim-to-real deployment.

Walkthrough

VGGT-Nav, step by step

Two HM3D validation episodes, rendered from the logged runs. Poses, images, costmaps and paths come from the logs. The 3D mesh is only for visualization, since Plann3r works on the reference images in a sliding window.

Scene A

Goal: chair, alternate goal: refrigerator

Scene B

Goal: plant, alternate goal: refrigerator cabinet

Key ideas

What's new

Plann3r, a method to predict planning costs in the form of pixel-level geodesic distances for a query image to an arbitrary subgoal pixel given within a submap.
VGGT-Nav, a single-camera visual navigation pipeline that leverages 3D grounded backend for mapping, localization and planning, which enables improved control prediction and navigation success rates across multiple tasks of a prior-traversal (seen-environment) navigation benchmark.
Simultaneous localization and planning. Plann3r enables simultaneous mapping/localization and planning as the query costmap is implicitly based on query-to-submap cross-attention; this forgoes the need for explicit pairwise feature matching which is more prone to failures.

Method

Plann3r architecture

We build Plann3r on top of a pre-trained VGGT model, fine-tuning it to predict geometrically consistent planning costs in terms of geodesic distances from an image pixel to a (sub)goal pixel given within a submap of images.

The VGGT-Nav pipeline

VGGT-Nav pipeline: offline mapping/global planning and online localization/local planning.

We use the proposed Plann3r, based on VGGT, to predict planning costs in terms of geodesic distances from query image pixels to an arbitrary (sub)goal pixel given within a submap. Plann3r is used iteratively both during the offline mapping cum global planning phase and the online execution phase of localization cum local planning, where the output costmaps from the latter are used to predict egocentric trajectory.

Offline: mapping and global planning

Functional map of costs

Plann3r predicts local planning costs on a submap around the goal image. The lowest-cost patch of the image shared with the next submap is the next subgoal, and its cost is added as an offset to the next window. Repeating this back to the start gives every reference image a costmap of global planning costs to the goal, without a 3D occupancy grid or a topological graph.

Online: localization and local planning

Cost-conditioned control

The robot retrieves a submap of 8 reference images, and the subgoal pixel is the minimum cost in their precomputed costmaps. Plann3r takes the query image, the submap and the subgoal together, so its global attention localizes the query without explicit feature matching. A controller with the architecture and training of ObjectReact turns the query costmap into 5 future waypoints in the local BEV space.

Why geodesic

Dense geodesic costmaps

Recent 3D mapping methods do not directly tell whether a place can be reached. That needs reasoning about traversable regions in a geodesic space, not a Euclidean one. Plann3r gives each pixel the geodesic distance from its closest floor point to the subgoal.

Qualitative comparison of planning costmaps: Ground Truth, Plann3r, Euclidean, ObjectReact.

The goal is outside the room and we observe that the side of the wall inside the room gets a high cost (shades of red) in both the Ground Truth and Plann3r generated costmaps, whereas the same wall either gets a low cost (shades of blue) when directly using 3D Euclidean distances or misses a cost gradient in ObjectReact.

Try it

Demo

Pick a scene and goal, then scrub the traverse to see Plann3r's predicted geodesic costmap per frame. Blue is a low cost to the goal and red is a high cost. The second scene is from the video game Sekiro, which the model never saw in training, run with no fine-tuning.

Cost to goal along the traverse

Overlay

Loading demo data...

Each goal is a pixel selected in one reference frame (white-and-red marker). The costmaps come straight from Plann3r's offline propagation pass over this scene. Global normalizes color across the whole traverse, so it shows the absolute cost to the goal. Relative normalizes each frame on its own, which shows local structure even far from the goal.

Demonstrations

Mapping and execution in action

Real-world recordings of the VGGT-Nav pipeline.

Offline mapping

Mapping to an alternate goal

Costs propagate from the alternate goal through overlapping local submaps of the traverse.

Execution view

Query, submap, costmap, and rollout

The query image, retrieved submap, predicted costmap and controller waypoints, beside the robot's run to the alternate goal.

Offline mapping

Mapping to the final goal

The same traverse, with costs propagated from the final goal.

Real-world rollout

Reaching the alternate goal

The robot follows the controller's waypoints in closed loop to the alternate goal.

Results

Navigation performance

Method	Imitate		Alt Goal		Shortcut		Reverse		Average
Method	SPL	SSPL	SPL	SSPL	SPL	SSPL	SPL	SSPL	SPL	SSPL
GNM	78.79	82.95	8.70	15.44	15.38	31.74	3.33	6.11	26.55	34.56
GNM (HM3D)	81.82	86.38	0.00	10.91	15.38	24.57	13.28	20.77	27.62	35.66
PixNav	42.42	46.75	26.09	31.66	7.69	22.29	16.16	25.56	23.09	31.57
RoboHop	57.56	64.99	30.43	38.23	30.77	40.87	9.98	16.92	32.19	40.25
ObjectReact	60.60	68.51	21.74	26.68	23.08	39.64	30.00	42.01	33.36	44.71
Plann3r + GT-trained Control (Ours)	61.30	75.53	58.88	69.42	45.36	61.10	0.00	9.80	41.39	53.96
Plann3r + Control (Ours)	78.43	87.69	65.92	77.78	55.68	70.82	0.00	11.99	50.01	62.07

State-of-the-art comparison on four navigation tasks of the IIN-val benchmark (HM3D), in SPL and Soft-SPL (SSPL). Best scores in bold, and the shaded rows are ours. GT-trained Control denotes a controller trained on simulator ground-truth (GT) costmaps but deployed with Plann3r-predicted costmaps.

Costmap quality

Costmap Source	All Pixels Below Threshold					Within Cost Range
Costmap Source	5%	15%	30%	50%	100%	0-5%	5-15%	15-30%	30-50%	50-100%
Euclidean Distance	2.284	2.533	2.617	2.561	2.232	2.236	2.403	2.412	2.296	2.645
ObjectReact	3.650	3.600	3.483	3.260	2.837	3.731	3.312	2.975	2.619	3.304
Plann3r (Ours)	2.398	2.424	2.369	2.299	1.986	2.387	2.262	2.091	2.043	2.318

Mean Absolute Rank Difference (MARD) between ground truth and predicted costs (sweeping over min-k% cost pixels or k-based bins). Lower is better. Left: cumulative MARD over all pixels below the given cost threshold. Right: MARD restricted to pixels within each cost bin. Euclidean distance is more accurate in the lowest 5% of costs around the subgoal, and Plann3r has the lowest error at every other threshold and bin.

Qualitative navigation overview

Qualitative overview of the navigation pipeline: submap and query costmaps, subgoal/goal pixels, predicted waypoints, and a top-down view.

Qualitative overview of the navigation pipeline. The query costmaps remain geodesically consistent with the submap costmaps and correctly direct the robot at both the timesteps, regardless of the visibility of the final goal in the current image at timestep t1.

BibTeX

@inproceedings{plann3r2026,
  title     = {Plann3r: Predicting Planning Costs Grounded in 3D},
  author    = {Anonymous Authors},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
  note      = {Project page: https://plann3r.github.io/}
}