Talk2Escape: Conversational Grounding for Vision-and-Language Navigation

TL;DR

Navigation agents shouldn't wander until they fail. They should ask for help.

Single-turn VLN agents run open-loop: one instruction, no way to verify progress or recover once sensor noise, odometry drift, or perceptual aliasing push them into a lost state. Talk2Escape (T2E) is a model-agnostic dialogue intervention module. A lightweight kinematic monitor watches for localized looping and trajectory divergence; when a trigger fires, a vision-language translator turns raw egocentric views into a concise grounded query, and a corrective hint from an oracle or human guide is injected back into the agent's prompt. No fine-tuning, no learned dialogue policy — pure zero-shot MLLM reasoning, on simulated benchmarks and on a physical Unitree Go2.

66.0%

SR on R2R-CE, above the best supervised SOTA — fully zero-shot

+17.2

absolute SR gain over the best open-loop zero-shot method (GTA)

84.6%

SR on VLNVerse with a vanilla NavGPT agent (up from 19.3%)

62.0%

real-world SR on a Unitree Go2 quadruped

Motivation

Open-loop execution is an architectural failure mode

When people get lost in an unfamiliar building, they don't keep wandering — they describe what they see and ask a guide. This instinctive error-recovery loop is exactly what standard VLN frameworks lack. Prior dialogue-driven VLN work treats conversation as supplementary planning context inside simulators that assume perfect communication and zero physical execution error. T2E instead treats dialogue as an operational safety net for real deployment.

Paradigm shift from single-turn to failure-aware dialogue-driven VLN — **Paradigm shift.** Left: single-turn agents execute one-off instructions open-loop, architecturally brittle to physical uncertainty. Right: T2E closes the loop with proactive, kinematic-aware help-seeking.

Method

Monitor → trigger → ask → hint → re-plan

Overview of the Talk2Escape framework — **The Talk2Escape framework.** A kinematic monitor evaluates the base agent's spatial state at every step; on failure it pauses the robot, grounds the current observation into an active query, and injects the guide's corrective hint into the MLLM prompt as a Navigation Assistance block.

Results

Zero-shot, above the supervised ceiling

On R2R-CE Val-Unseen (100-episode protocol of Open-Nav), T2E + GTA reaches 66.0% SR — past the strongest supervised method (Efficient-VLN, 64.2%) without a single in-domain training trajectory. Even a vanilla NavGPT base jumps to 60.0% SR.

Method	Type	NE ↓	OSR ↑	SR ↑	SPL ↑
Efficient-VLN	Supervised	4.18	73.7	64.2	55.9
NavFoM	Supervised	4.61	72.1	61.7	55.3
GTA	Zero-shot	4.95	56.2	48.8	41.8
VLN-Zero	Zero-shot	5.97	51.6	42.4	26.3
Talk2Escape + NavGPT	Zero-shot	4.84	60.0	64.0	46.4
Talk2Escape + GTA	Zero-shot	4.80	66.0	72.0	49.4

R2R-CE Val-Unseen, 100 sampled episodes following the Open-Nav protocol. Full RxR-CE and complete baselines in the paper.

Dialogue beats internal reasoning by a wide margin

On the high-fidelity VLNVerse benchmark, passive Chain-of-Thought lifts NavGPT by only 9.4 points; active dialogue lifts it by 65.3.

Agent	NE ↓	SR ↑	OSR ↑	SPL ↑	nDTW ↑
NavGPT	6.10	19.30	61.50	8.13	52.60
NavGPT + CoT	5.25	28.70	50.50	9.91	44.10
NavGPT + T2E	2.92	84.60	88.30	35.20	60.50
MapGPT	5.62	25.53	55.32	7.57	43.38
MapGPT + CoT	4.51	42.19	72.40	11.32	45.20
MapGPT + T2E	4.14	67.02	74.47	30.46	53.62

Fine-grained instruction task on VLNVerse.

Real-World Deployment

From simulator to a Unitree Go2

We deploy T2E on a Go2 quadruped with a servo-mounted depth camera sweeping orthogonal views, decoupling MLLM cognition from high-frequency locomotion. Under a strict human-in-the-loop protocol mirroring the deviation trigger, T2E reaches 62.0% SR against 40.0% for the strongest open-loop baseline.

Real-world execution of the Talk2Escape paradigm — **Rescue in the wild.** The agent misreads the instruction and heads for the wrong door; a grounded hint via dialogue corrects the route and the robot reaches the intended target.

Method	Type	SR ↑ (%)	NE ↓ (m)
VLN-BERT	Supervised	16.0	5.36
RDP	Supervised	20.0	5.45
SmartWay	Zero-shot	32.0	4.85
GTA	Zero-shot	40.0	3.66
Talk2Escape + GTA	Zero-shot	62.0	3.31

Key Findings

When to speak matters as much as what to say

strategic sparsity

The deviation trigger is the best autonomous strategy (SR 66.0, SPL 49.4). The hybrid trigger fires too often (23.1% of steps) and drops to SR 54.0 — excessive querying disrupts the agent's contextual memory and causes "intervention fatigue".

agent capability

Weaker agents degrade linearly as guidance gets sparser; a strong agent like GTA shows an inverted U-shape — too-frequent polling interferes with its internal planning, too-sparse polling deprives it of rescue.

modality mismatch

MapGPT's dense absolute 3D coordinates conflict with the oracle's egocentric hints; a clean, action-centric history aligns far better with natural-language corrections than over-engineered geometric prompts.

asymmetric context

Human guides receiving only static frames struggle to reconstruct the robot's true orientation — the "asymmetric context" bottleneck. Future work: spatio-temporal video buffers and multi-view memory in the dialogue state.