Evaluation on NavSpace

Evaluation Environment and Metrics: NavSpace takes Habitat 3.0 as the simulator to conduct the evaluation. Evaluation scenes are selected from the HM3D datasets. At each step, the agent can only select one action. Following previous instruction navigation benchmarks, we employ the following widely used evaluation metrics: Navigation Error (NE), Oracle Success Rate (OS), Success Rate (SR). We conduct a comprehensive evaluation of existing multimodal large models and navigation models. These models can be categorized into the following five types:

• Chance Level Baselines
• Open-source MLLMs
• Proprietary MLLMs
• Lightweight Navigation Models
• Navigation Large Models

Performances on NavSpace: NavSpace poses a major challenge for current models. Open-source MLLMs achieve less than 10% success, near chance level, while proprietary MLLMs perform slightly better, with GPT-5 leading but still under 20%. This indicates that existing MLLMs are far from reliable navigation agents for spatial intelligence tasks. Lightweight navigation models such as BEVBert and ETPNav also fail on NavSpace, whereas larger navigation models like NaVid and StreamVLN outperform GPT-5 and show preliminary spatial reasoning ability. Our proposed model, SNav, surpasses both state-of-the-art navigation models and advanced MLLMs, establishing a strong new baseline. Ablation results further demonstrate that its instruction-generation pipeline is key to enhancing spatial intelligence performance.

Our SoTA Model——SNav: The architecture of the SNav incorporates three fundamental components: the Vision Encoder v(·), the Projector p(·), and the Large Language Model (LLM) f(·). We follow the previous work to conduct navigation finetuning through co-training with three tasks. These tasks include Navigation Action Prediction, Trajectory-based Instruction Generation, and General Multimodal Data Recall. After this, we obtain the vanilla SNav model.

Spatial Intelligence Enhancement:

• Cross-floor Navigation: We identify R2R trajectories likely to cross floors by thresholding height differences between start and end. For each candidate, the agent follows a shortest-path planner in Habitat while recording RGB frames. A trajectory is labeled floor-crossing if GPT-5 detects stairs in ≥3 frames. Using methods from HOV-SG, we assign floor labels to start/end points and combine them with Habitat's floor count for vertical-space annotations. GPT-5 then restyles raw instructions (e.g., “Walk up the stairs ...”) into floor-specific ones (e.g., “Walk up to the third floor ...”).
• Precise Movement: We sample start-goal pairs in MP3D scenes and compute shortest paths in Habitat. After filtering, trajectories of target length (20-60 steps) are retained, with discrete actions recorded (turn ±30°, move forward 0.25m, stop). Consecutive actions of the same type are merged into concise descriptions (e.g., “move forward 3 m, turn right 60°, move forward 2 m”). GPT-5 paraphrases these into natural-language instructions.
• Environment State Inference: We extract start-end point pairs with navigation instructions from the R2R dataset and generate trajectories using a shortest-path planner, saving RGB frames along each path. GPT-5 is then queried with the first and last frames to infer three elements: observable objects, unobservable objects, and scene descriptions. After observing the structure of this category, we design five template categories that integrate multimodal observations with original instructions. Two representative patterns are: (1) Original_instruction; if [visible_object in last frame] then stop at [last-frame location], otherwise go to [scene description from first frame], and (2) If [fabricated_object in first frame] then stop, otherwise follow Original_instruction and stop at [last-frame location]. GPT-5 rewrites instructions according to these templates, producing high-quality training data for environment state inference.
• Spatial Relationship: We filter R2R instructions with regular expressions for ordinal phrases (“first room/door”, “second room/door”, etc.) and detect multi-object relations using keywords like “between”, “along”, and “across”.

Real World Tests: In the real-world test, we test our model SNav across three different environments: office, campus, and outdoor environments. The test covers five categories of spatially intelligent navigation instructions (excluding vertical perception). Our experimental platform is the AgiBot Lingxi D1 quadruped, which is equipped with a monocular RGB camera and motion-control APIs.

@misc{yang2025navspacenavigationagentsfollow,
                title={NavSpace: How Navigation Agents Follow Spatial Intelligence Instructions}, 
                author={Haolin Yang and Yuxing Long and Zhuoyuan Yu and Zihan Yang and Minghan Wang and Jiapeng Xu and Yihan Wang and Ziyan Yu and Wenzhe Cai and Lei Kang and Hao Dong},
                year={2025},
                eprint={2510.08173},
                archivePrefix={arXiv},
                primaryClass={cs.RO},
                url={https://arxiv.org/abs/2510.08173}, 
          }