APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

Emily Jin, Joy Hsu, Yiqing Xu, Weiyu Liu†, Nick Haber†, Jiajun Wu†

Stanford University · †Equal advising

Paper arXiv Code BibTeX

APIVOT is a VLM-based planner that combines complementary modes of reasoning: language for semantic structure, vision for geometric feasibility. By adaptively interleaving language reasoning with imagined visual future states, APIVOT explicitly reasons about geometric constraints and produces more feasible long-horizon plans.

Abstract

Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to rely on language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.

Overview

Consider a long-horizon task: “Store the leftovers in the fridge.” To succeed, a robot must reason semantically about what to store, which containers to use, and which actions must happen first. It must also reason geometrically about how those containers fit inside the fridge and whether there is enough free space. These decisions are tightly coupled: a semantically valid plan can still fail if it ignores spatial constraints.

Key insight: long-horizon planning requires joint reasoning about semantic structure and geometric feasibility.

We propose APIVOT, a VLM-based planner that interleaves language reasoning with visual thoughts: imagined future scene states that help the model reason about spatial constraints. APIVOT uses language where symbolic and semantic reasoning are sufficient, and invokes visual thoughts when geometric precision is needed. By adaptively pivoting between these modalities, our model produces more feasible plans for long-horizon tasks.

Method

Given a task instruction, scene observation, and detected objects, APIVOT generates an interleaved reasoning trace of vision-language thoughts, followed by an executable plan. Language is used to decompose goals, track prerequisites, and choose actions, while visual thoughts are used when the subgoal depends on geometry, free space, or potential collisions.

APIVOT interleaves language thoughts (text steps) and visual thoughts (imagined future scenes) into a single reasoning trace, then outputs an executable action plan.

Training Curriculum

APIVOT is trained with supervised fine-tuning on reference traces that demonstrate interleaved reasoning and successful planning.

We train APIVOT with a three-stage curriculum that progressively teaches it to use visual thoughts, generate them, and adaptively select when they are needed. Across all stages, training uses a standard cross-entropy loss over reasoning traces and plans, along with an image-alignment loss that aligns generated visual thoughts with encoded future scene observations.

STAGE 1

Use Visual Thoughts

The model is given ground-truth visual thoughts and learns to use them for downstream planning.

STAGE 2

Generate Visual Thoughts

The model autoregressively generates its own visual thoughts, representing imagined subgoals.

STAGE 3

Adaptively Use Vision

The model learns to use visual thoughts selectively for geometry-sensitive subgoals.

Results

We evaluate APIVOT on three long-horizon KitchenWorlds task families: Containment and Sorting, which are in-distribution (ID), and Storing Leftovers, a held-out, out-of-distribution (OOD) task that composes the two ID ones.

APIVOT improves planning success over general-purpose VLMs (Gemini-ER-1.5, +8.1 points) and planning baselines (VLM-TAMP, +9.0 points). Its gains are consistent across all three task families, including the held-out composition, suggesting that interleaved language and visual reasoning supports generalization beyond the training distribution.

Overall Planning Performance

Model Average Containment (ID) Sorting (ID) Storing Leftovers (OOD)
VLM Baselines
Gemini-3.1-Pro0.2450.2810.2410.213
Gemini-ER-1.50.3380.3640.3390.311
Qwen3-VL-8B-Instruct0.1880.2180.1830.162
Qwen3-VL-8B-Thinking0.2320.2590.2340.204
Planning Baselines
FastDownward0.2720.3310.2470.238
Reflect-VLM0.2920.3480.2720.257
VLM-TAMP0.3290.3700.3110.307
APIVOT (Ours)0.4190.4720.4210.365

Task success rate (fraction of episodes that succeed) across task families. Containment and Sorting are in-distribution (ID); Storing Leftovers is a held-out, out-of-distribution (OOD) task.

Additional Results

Grouped bar chart of success rate by goal-area occupancy for five methods; APIVOT leads at every level, with the largest margin under the most geometric constraints.
  • We analyze performance as scenes become more geometrically constrained. We define an occupancy ratio that measures how crowded the available space is.
  • In low-occupancy (under-constrained) settings, all methods achieve high success.
  • As occupancy increases, baseline performance drops sharply, while APIVOT remains more robust. Its advantage over baselines grows from 0.07 to 0.17.
Line chart of success rate versus reasoning-token budget; APIVOT dominates all baselines across every token budget from 128 to 8192.
  • Across matched reasoning-token budgets, APIVOT consistently outperforms language-only VLM baselines.
  • Visual thoughts make planning more token-efficient.
  • Instead of using long text traces to describe spatial layouts, APIVOT represents geometry directly through imagined future states, enabling stronger planning with fewer tokens.
Success rate versus token usage; Adaptive APIVOT sits near Always-Image performance while using far fewer tokens, and well above No-Image and all baselines.
  • At inference time, APIVOT can operate in three modes: Always-Image, Adaptive, and No-Image.
  • Always-Image achieves the highest task success by using visual thoughts at every planning step, but incurs substantially higher token usage.
  • Adaptive reasoning offers a stronger efficiency trade-off: it retains 91% of Always-Image performance while using 39% fewer tokens, showing that APIVOT can effectively use visual thoughts selectively.

Examples

APIVOT successfully anticipates geometric constraints, producing feasible plans where baselines struggle.

BibTeX

@article{apivot2026,
  title   = {APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts},
  author  = {Jin, Emily and Hsu, Joy and Xu, Yiqing and Liu, Weiyu and Haber, Nick and Wu, Jiajun},
  year    = {2026}
}