ERA Lab USF · Embodied Robotics & Autonomy
Accepted to IROS 2026

CORAL: COntextual Reasoning And Local Planning in A Hierarchical VLM Framework for Underwater Monitoring

Zhenqi Wu, Yuanjie Lu, Xuesu Xiao, Xiaomin Lin

University of South Florida & George Mason University

Paper Video Code
CORAL system overview: hierarchical VLM planner over MDDP low-level controller for underwater reef monitoring
CORAL decouples high-level semantic reasoning (VLM planner) from low-level reactive control (MDDP), validated in simulation and on a BlueROV2 in a controlled pool.

Abstract

Oyster reefs are critical ecosystem species that sustain biodiversity, filter water, and protect coastlines, yet 85% have been lost globally. Restoring these ecosystems requires regular underwater monitoring to assess reef health, a task that remains costly, hazardous, and limited when performed by human divers. Autonomous underwater vehicles (AUVs) offer a promising alternative, but existing AUVs rely on geometry-based navigation that cannot interpret scene semantics. Recent vision-language models (VLMs) enable semantic reasoning for intelligent exploration, but existing VLM-driven systems adopt an end-to-end paradigm, introducing three key limitations: frequent inference delays, weak dynamics awareness, and limited self-correction.

We propose CORAL, a framework that decouples high-level semantic reasoning from low-level reactive control. The VLM provides high-level exploration guidance by selecting waypoints, while a dynamics-based planner handles low-level collision-free execution. A geometric verification module validates waypoints and triggers replanning when needed. Compared with the previous state-of-the-art, CORAL improves coverage by 14.28 percentage points, reduces collisions by 100%, and requires 57% fewer VLM calls.

Method

CORAL separates the navigation stack into three tightly integrated layers: a persistent occupancy map with centroid chain extraction, a VLM high-level planner invoked only at meaningful decision points, and a dynamics-aware MDDP low-level controller running continuously at 10 Hz.

  • Perception & Centroid Chain — Depth and segmentation images are fused into a persistent 2D occupancy map. Cluster centroids are extracted and organized into an ordered chain with cross-frame exponential smoothing.
  • VLM High-Level Planner — A smart trigger invokes the VLM only on goal arrival, stuck detection, or planner failure. A geometric verifier rejects backward or off-trend waypoints with structured corrective feedback.
  • MDDP Low-Level Controller — Decremental Dynamics Planning augments MPPI with variable-fidelity dynamics — dense near the robot, progressively simplified farther along — enabling long-horizon planning at zero collisions.

Simulation

We evaluate CORAL across five reef-topology environments of increasing complexity. Each environment is shown below with its coverage result and a playback of the full mission.

L-Shape reef simulation environment
L-Shape — 100%
S-Shape reef simulation environment
S-Shape — 92.86%
O-Shape reef simulation environment
O-Shape — 100%
K-Shape reef simulation environment
K-Shape — 83.33%
E-Shape reef simulation environment
E-Shape — 100%

Simulation videos

Trajectories

Coverage trajectories across all five environments, comparing DREAM, CORAL w/o Low-Level, and CORAL (Full).

L-Shape coverage trajectory comparison
L-Shape
S-Shape coverage trajectory comparison
S-Shape
O-Shape coverage trajectory comparison
O-Shape
K-Shape coverage trajectory comparison
K-Shape
E-Shape coverage trajectory comparison
E-Shape

Real-World Deployment

CORAL deployed on a BlueROV2 Heavy in a controlled pool (12 ft diameter, 5 ft depth) with oyster shells and floating obstacle balls. Perception runs on-board in real time; VLM queries reach cloud API endpoints asynchronously.

Overhead view

First-person view

Results

We evaluate CORAL against two baselines across 10 simulation environments of varying topology complexity. All metrics are averaged over all environments.

Method Coverage (%) ↑ Time (s) ↓ Collisions ↓ VLM Calls ↓
DREAM 80.00 1513.2 9 1261
CORAL w/o Low-Level 91.40 687.0 24 770
CORAL (Ours) 94.28 642.0 0 547

Headline gains over the previous state-of-the-art (DREAM):

  • +14.28 pp coverage rate (80.00% → 94.28%)
  • −100% collisions (9 → 0)
  • −57% VLM calls (1261 → 547 per mission)

BibTeX

@misc{wu2026coralcontextualreasoninglocal,
  title         = {CORAL: COntextual Reasoning And Local Planning in A Hierarchical VLM Framework for Underwater Monitoring},
  author        = {Zhenqi Wu and Yuanjie Lu and Xuesu Xiao and Xiaomin Lin},
  year          = {2026},
  eprint        = {2603.14786},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2603.14786}
}

Back to Research