Robotics capabilities depend on shared relationships between observations, actions, and goals. We introduce PsiBorg, a new paradigm for pretraining robotics foundation models that defines many capabilities as prompts to a unified model through any-to-any token prediction.
Our work builds on Probabilistic Structure Integration (PSI), which trains an autoregressive model on large-scale internet video to support flexible prompting through visual intermediates. We extend this approach to robotics by jointly training on a mixture of internet videos depicting people, scenes, and objects; robot videos spanning different embodiments; and paired observations and actions.
We focus on three inference pathways: policy inference, simulation, and progress tracking. Across these capabilities, we demonstrate strong generalization to new objects, scenes, and goals.
Policy and progress tracking
Given the current observation and one goal frame, PsiBorg first predicts Δt, the estimated temporal distance to the goal. It then uses Δt to predict the next action chunk.
We find that predicting temporal distance first and incorporating it into the goal image greatly improves action prediction compared with conditioning on the goal image alone. The examples show new ways to interact with known objects, flexible interaction with objects not seen during training, and diverse goals involving the same objects.
Loading videos…
Simulation
Given a current observation and a sequence of low-level robot commands, PsiBorg predicts the resulting visual observations. Here, the action inputs are joint positions and gripper commands. We show simulation results for held-out teleoperated episodes, including picking up and dropping off objects, as well as counterfactual scenarios. We also demonstrate generalization to environments and objects not seen during training.
Loading videos…
Progress tracking
Given a current observation and a single goal image, PsiBorg predicts a distribution over their temporal distance, Δt. We learn this prediction from the observed time separation between frames in training videos. As new observations arrive, the estimated temporal distance provides a way to track progress toward the fixed goal. We demonstrate this capability in human videos and across different robot embodiments.
Loading videos…
Technical Details
Model
Our base vision model, PSI0.5, uses a GPT-style autoregressive transformer architecture trained on next-token prediction across the following modalities: RGB, depth, optical flow, and camera pose. We extend this framework to robot learning by adding actions as another modality in the training mix. We refer to this variant of the model as PsiBorg.
Data and training
Our training data falls into three groups:
General visual data: Diverse internet videos and vision datasets used to train the base model, including human activity and egocentric video.
Robot videos: Open-source datasets spanning different robot embodiments and tasks. These broaden the model’s exposure to robot motion, object interactions, and task structure.
Paired observations and actions: YAM data from ABC, World Engine, Molmo, and diverse manipulation tasks collected on our own bimanual YAM setup. These datasets connect visual observations to actions in the YAM embodiment.
