JoyAI-RA 0.5 — Technical Report

Scaling Robot Manipulation Learning via Dual Action Alignment

Tech Report Datasets

We propose JoyAI-RA 0.5, a generalist Vision-Language-World-Action framework that scales robot manipulation learning from heterogeneous human, simulation, and robot data through dual action alignment. Implicit alignment learns transferable physical dynamics from action-free video, while explicit alignment maps reliable human and robot trajectories into a unified action space for executable control. An inner-outer-loop reinforcement learning stage combines rapid task adaptation with continued foundation-policy improvement. On the Real-World AgiBot Benchmark, JoyAI-RA 0.5 achieves strong performance on seen tasks and unseen variations. Performance improves consistently as human egocentric pretraining data scales, with no saturation observed at the largest tested scale, establishing human video as a primary scaling axis for real-world manipulation.

Data

Heterogeneous Training Data

The pre-training corpus combines 53K+ hours of human egocentric video, 11K+ hours of simulation, and 8K+ hours of real-robot demonstrations across bimanual and single-arm embodiments. Human video contributes broad visual and behavioral diversity, while simulation and robot trajectories provide structured and executable supervision across complementary dynamics, semantics, and physical control signals.

JoyAI-RA heterogeneous data composition across human egocentric videos, simulation trajectories, and real-robot demonstrations

Closed-Loop Data Pipeline

Raw human ego videos are annotated for visual, language, and hand-motion signals, then processed with robot trajectories under shared validity, quality, and diversity criteria. Representation-guided curation removes low-quality and redundant episodes while preserving rare strategies and scene configurations. Coverage analysis then identifies sparse capabilities and directs subsequent data collection.

Closed-loop JoyAI-RA data preprocessing and curation pipeline

Framework

Architecture

JoyAI-RA combines a VLM, a Latent-Action-Conditioned World Model, and a Flow-Matching Action Expert. The VLM encodes task goals and semantics, the LAC-WM supplies physical-dynamics priors from visual transitions, and late fusion exposes both to the action expert, which generates continuous action chunks in a canonical 130-dimensional space.

Inner–Outer Loop Reinforcement Learning

The fast inner loop runs on the edge server, learning a lightweight residual policy for task-specific adaptation while the foundation model remains frozen. The asynchronous outer loop aggregates successful interactions, improves the foundation VLWA on the central server, and periodically synchronizes updated parameters back to the edge, forming a closed self-improving system.

Experiments

Real-World AgiBot Benchmark

We evaluate the AgiBot G1 on six scenarios spanning easy pick-and-place, precise manipulation, and long-horizon tasks. Seen trials test familiar objects and layouts; unseen trials vary object categories, spatial arrangements, backgrounds, and illumination. Each task is scored from subtask completion across 20 seen and 10 unseen trials, measuring both execution quality and generalization.

Inner–Outer Loop RL Results

We compare the original policy, inner-loop-only adaptation, outer-loop-only adaptation, and the combined method under identical training budgets on Mouse and Headphone pick-and-place with unseen position shifts. Both loops improve over the foundation policy, while their combination performs best, reaching 70% and 50% success and confirming their complementary benefits.

Web-native success-rate comparison of the original VLWA policy, inner-loop RL, outer-loop RL, and inner-outer loop RL

Human Video Scaling

We run two controlled studies: scaling EgoLive before identical robot post-training, and scaling human video for LAC-WM pretraining with downstream policy training fixed.

With the world model frozen, larger EgoLive subsets improve both pretraining generalization and downstream policy initialization. The full dataset maintains the lowest validation loss and raises seen/unseen scores from 47.8/37.6 to 85.6/60.2 after identical robot post-training.

Validation-loss curves for policy pretraining with 10%, 25%, 50%, and 100% of EgoLive

Policy performance across increasing fractions of human-video pretraining data

Demonstrations

In this multi-stage desk-organizing task, AgiBot G1 places the headphones on their stand and then returns the mouse to its pad. The sequence combines the paper's precision-demanding headphone manipulation (PnP-Hard) with common-object mouse placement (PnP-Easy), shown from synchronized first- and third-person views.

First View Third View

More task rollouts cover storing pens, pencil cases, erasers, staplers, chargers, and correction fluid, shelving books, and discarding trash. Together, they span common-object organization, higher-precision placement, and longer multi-step execution across different target locations.

Store Pen

Store Pencil Case

Shelve Books

Discard Trash

Store Correction Fluid

Store Eraser

Store Stapler

Store Charger

Citation

@misc{joyairateam2026joyaira05,
  title={JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment},
  author={JoyAI-RA Team},
  year={2026},
  eprint={2608.05674},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2608.05674}
}

Overview

Abstract