Genie Sim: Towards the Next Generation of Embodied Intelligence

CHAPTER 01

Simulation, evaluation, and evolution仿真、评测与进化

1.1从物理对齐到模型改进

Genie Sim 将系统辨识、可组合的物理和渲染引擎，以及多维能力评测连接起来。遥操作与自动采集提供交互数据。RLinf 通过强化学习后训练优化策略，Genie Sim Dream 则利用世界模型生成机器人交互中的视觉观测。这些模块共同围绕模型的研发与迭代展开。

物理一致性的惯性辨识约束机器人参数估计，RigPI 则结合视觉语言先验、力与力矩观测及运动信息进行物体辨识。Genie Sim 将动力学、接触与控制校准连接到真实—仿真配对评测，以检验任务层面的虚实一致性。

1.2随模型共同演进的评测

Genie Sim 3.0 的指令遵循、空间推理、通用操作与抗扰动四个 track，是基于特定发展阶段的具身模型能力水平及算法研究重点设计的，旨在从指令条件下的交互、空间关系推理、物体操作与环境变化适应等方面，对模型的基础能力进行结构化评估。

随着模型能力提升，制约性能的主要瓶颈和算法优化的重点也会发生变化。评测体系需要与模型和算法的演进协同更新，才能持续区分模型能力、识别尚未解决的局限，并为后续迭代提供依据。因此，逐步扩展任务结构与能力覆盖，是保持评测诊断价值和研究指导作用的必要条件。

扩展后的 benchmark 在保留原有 track 的基础上，新增灵巧手操作、全身协同、精细操作与跨本体操作四类任务套件，分别考察手指功能性协同、底盘—躯干—机械臂联合控制、精细几何对齐，以及操作能力的跨本体迁移，从而为评估持续发展的具身模型提供更充分的实证基础。

1.1From physical alignment to model improvement

Genie Sim connects system identification, composable physics and rendering engines, and multidimensional evaluation. Teleoperation and automated collection provide interaction data. RLinf supports reinforcement-learning post-training for policy optimization; Genie Sim Dream uses a world model to generate visual observations of robot interactions. Together, these modules support embodied-model research, development, and iterative improvement.

Physically consistent inertial identification constrains robot parameter estimates, while RigPI combines vision-language priors with force–torque and motion observations for object identification. Genie Sim connects dynamics, contact, and control calibration with paired real–simulation evaluation to assess task-level consistency.

Genie Sim overview: physical simulation, data collection, benchmarking, post-training, and world-model simulation.

1.2Evaluation that evolves with models

The four tracks in Genie Sim 3.0—Instruction Following, Spatial Reasoning, General Manipulation, and Robustness—were designed around the capabilities of embodied models and the principal objectives of algorithm development at a certain stage. They provide a structured assessment of foundational competencies in instruction-conditioned interaction, spatial reasoning, object manipulation, and robustness to environmental variation.

As model capabilities advance, the dominant performance bottlenecks and the objectives of algorithm development also change. Evaluation must evolve in concert with these developments to maintain its ability to distinguish model capabilities, identify unresolved limitations, and provide evidence for subsequent iterations. Extending task structures and capability coverage is therefore necessary for evaluation to remain informative throughout model development.

The expanded benchmark complements the established tracks with Dexterous-Hand Manipulation, Whole-Body Coordination, Precision Manipulation, and Cross-Embodiment Manipulation. These suites assess functional finger coordination, integrated base–body–arm control, fine geometric alignment, and transfer of manipulation capabilities across robot embodiments, broadening the empirical basis for evaluating increasingly capable embodied models.

CHAPTER 02

System identification and Real2Sim系统辨识与 Real2Sim

拿金牌 · 真机与仿真对比

仿真的基础是尽可能真实地复现现实世界中的物理交互。仿真评测首先需要回答的核心问题是，被测模型在仿真中的表现能在多大程度上反映其在真实机器人上的表现。为此，我们开展机器人与物体的系统辨识及控制校准工作，以提升虚实一致性，并通过 Real2Sim 实验检验 Genie Sim 的可靠性。

切换阅读主题回放协议

2.1记录指令回放

回放的目标是复现给定指令下机器人与物体的响应。x 包含其构型与速度，u 表示记录的关节及夹爪目标；动力学与视觉观测分别建模：

状态演化

x_{k + 1} =

f_{ℰ} (x_{k}, u_{k}; θ_{r}, θ_{o}, ψ)

视觉观测

{\hat{y}}_{k} =

h_{η} (x_{k})

ℰ 指物理后端，θ_r、θ_o 分别为机器人与物体参数，ψ 为控制和积分设置，η 为观测标定。图像点与 RGB-D 特征估计不是对完整物理状态的直接测量，因此机器人动力学、物体交互和执行设置需要分开表述。

每项对比明确机器人本体、末端执行器、交互资产、初始状态和指令接口。G2 物体回放实验以约 100 Hz 保持记录的关节及夹爪位置目标，物理子步为 1 ms。物体运动由动力学和接触产生，不使用抓取绑定或逐帧替换物体位姿。

物体回放协议。冻结参数迁移与原生 PhysX 适配是独立对比；海绵重拟合单独说明。虚线框中的未来扩展尚未执行。

2.2机器人动力学与验证

G2 crsB 双臂采用固定的电流—力矩系数 K，将电机电流 I 与刚体动力学、摩擦、阻尼、等效电枢惯量及偏置联系起来：

力矩平衡

K I =

τ_{rigid} + F φ (\dot{q})

+ D \dot{q} + A \ddot{q} + b

刚体动力学

τ_{rigid} =

M (q; π) \ddot{q}

+ c (q, \dot{q}; π) + g (q; π)

平滑摩擦

φ (v) =

tanh (\frac{v}{ε})

ε = 0.001 rad/s

惯性参数 π 决定刚体动力学；对角矩阵 F、D、A 分别表示类库仑摩擦、黏性阻尼及等效电枢惯量。等效惯量可能吸收模型近似误差，不能视为经确认的物理转子惯量。匀速激励估计摩擦和阻尼，正弦激励在临时惯性参数下估计电枢惯量，随后固定 F、D、A，利用耦合 Fourier 激励拟合惯性与偏置。分阶段辨识减少参数间的补偿，但物理一致性约束不保证唯一恢复各连杆属性。X5 因不支持力控而采用简化模型。

力矩重建残差

r_{k}^{I} =

K I_{k}

- τ_{model} (q_{k}, {\dot{q}}_{k}, {\ddot{q}}_{k})

左臂 Fourier 拟合诊断。保留全部记录及受驱动关节；蓝色实线为测量值，橙色虚线为辨识模型。

无夹爪仿真在执行器限幅前采用位置 PD 控制；传感器修正仅作用于后处理，不改变物理受力或控制：

位置 PD 控制

τ_{drive} =

K_{p} (q_{cmd} - q_{sim})

- K_{d} {\dot{q}}_{sim}

传感器读数

{\hat{τ}}_{sensor} =

τ_{link,sim}

+ f_{sensor} ({\dot{q}}_{sim}, h)

K_p = 1000 I₇ N·m/rad，K_d = diag(1, 5, 1, 1, 1, 1, 1) N·m·s/rad 为采用的仿真设置，并非辨识得到的硬件增益；h 表示观测器历史。电流推算力矩、仿真连杆力矩和关节侧实测反馈是不同物理量。逆动力学重建、前向预测和闭环回放分别检验不同响应；强反馈可能在模型不完善时仍压低位置误差，因此位置与未去中心化的传感器力矩 RMSE 分别报告。机器人姿态、控制器和力矩零点的来源记录仍不完整。

在 15 次长时序运行、21 个有效关节/激励单元中，位置 RMSE 为 0.035°–1.002°。位置跟踪与力矩一致性分别评估。MuJoCo/MJWarp 的一致性及同一记录内的时间检查，尚不能证明独立 episode 的泛化。

同一段 60 s Fourier 记录中的位置与关节侧力矩诊断，保留可见的力矩残差。

Medal handling · Real vs Sim

Simulation aims to reproduce real-world physical interactions as faithfully as possible. A central question for simulation-based evaluation is therefore how well a model’s performance in simulation reflects its performance on a physical robot. To address this question, we conduct robot and object system identification and control calibration to improve sim–real consistency, and use Real2Sim experiments to evaluate the reliability of Genie Sim.

Explore this chapterReplay protocol

2.1Recorded-command replay

Replay aims to reproduce command-conditioned robot and object responses. Let x contain their configurations and velocities and u the recorded joint/gripper targets. Dynamics and visual observation are modeled separately:

State transition

x_{k + 1} =

f_{ℰ} (x_{k}, u_{k}; θ_{r}, θ_{o}, ψ)

Visual observation

{\hat{y}}_{k} =

h_{η} (x_{k})

ℰ selects the physics backend, θ_r and θ_o denote robot and object parameters, ψ contains control/integration settings, and η is observation calibration. Image points and RGB-D feature estimates do not directly measure the complete physical state. Robot dynamics, object interactions, and execution settings therefore retain distinct roles.

Each comparison fixes the embodiment, end effector, interacting assets, initial state, and command interface. In the G2 object-replay study, recorded joint and gripper position targets are held at approximately 100 Hz, with 1 ms physics substeps. Object motion comes from dynamics and contact, without grasp attachments or frame-wise pose replacement.

Object replay protocols. Frozen transfer and native PhysX adaptation are separate comparisons; the sponge refit is declared. The dashed future-extension strip has not been executed.

2.2Robot dynamics and validation

For the G2 crsB dual arms, a fixed current-to-effort scale K relates motor current I to rigid-body dynamics, friction, damping, effective armature, and an offset:

Torque balance

K I =

τ_{rigid} + F φ (\dot{q})

+ D \dot{q} + A \ddot{q} + b

Rigid-body model

τ_{rigid} =

M (q; π) \ddot{q}

+ c (q, \dot{q}; π) + g (q; π)

Smooth friction

φ (v) =

tanh (\frac{v}{ε})

ε = 0.001 rad/s

The inertial parameters π determine rigid-body dynamics. Diagonal F, D, and A represent Coulomb-like friction, viscous damping, and effective armature. The latter can absorb model approximations and is not a certified physical rotor inertia. Constant-speed sweeps estimate friction/damping, sine motion estimates armature with temporary inertia, and coupled Fourier excitation then fits inertia and offsets with F, D, and A fixed. This staging limits parameter compensation; physical-consistency constraints do not guarantee unique recovery of individual link properties. X5 uses a simplified model because force control is unavailable.

Effort residual

r_{k}^{I} =

K I_{k}

- τ_{model} (q_{k}, {\dot{q}}_{k}, {\ddot{q}}_{k})

Fourier fitting diagnostics for the left arm. All recordings and driven joints are retained; solid blue is measured effort and dashed orange is the identified model.

The gripper-free simulation uses position PD control before actuator limits. Its sensor correction is postprocessing only and changes neither plant forces nor control:

Position PD control

τ_{drive} =

K_{p} (q_{cmd} - q_{sim})

- K_{d} {\dot{q}}_{sim}

Sensor readout

{\hat{τ}}_{sensor} =

τ_{link,sim}

+ f_{sensor} ({\dot{q}}_{sim}, h)

K_p = 1000 I₇ N·m/rad and K_d = diag(1, 5, 1, 1, 1, 1, 1) N·m·s/rad are adopted simulation settings, rather than identified hardware gains; h denotes observer history. Current-derived effort, simulated link torque, and measured joint-side feedback are distinct quantities. Inverse-dynamics reconstruction, forward prediction, and closed-loop replay test different behaviors. Strong feedback can reduce position error even with an imperfect plant, so position and uncentred sensor-torque RMSE are reported separately. Robot posture, controller, and torque-zero provenance remain incomplete.

Across 15 long-horizon runs, position RMSE spans 0.035°–1.002° over 21 active joint/excitation cells. Position tracking and torque agreement are evaluated separately. MuJoCo/MJWarp consistency and same-recording temporal checks do not establish independent-episode generalization.

Position and joint-side torque diagnostics from the same 60 s Fourier recording. Residual torque errors remain visible.

CHAPTER 03

Composable physics and rendering engines可组合的物理和渲染引擎

切换阅读主题统一协议

3.1统一协议边界

GenieSim Protocol 将控制、物理状态演化与视觉观测生成分开，交换控制指令、关节状态、刚体位姿、仿真时间及形变几何更新。物理端维护状态，渲染端读取状态；场景元数据关联物理实体与视觉表示。

物理与渲染适配器共享 GenieSim Protocol。MuJoCo 还通过求解器集成及 IPC 使用的 CPU 机器人动力学进程参与执行。

3.2选择物理与渲染路径

后端职责

组件	职责
PhysX	Isaac Sim 物理执行路径
Newton	独立运行及 Isaac 集成执行
libuipc	刚柔耦合接触仿真
MuJoCo	求解器集成与协同 CPU 机器人动力学
Isaac Sim / OVRTX	通过共享状态接口提供可选渲染路径

适配器实现初始化、指令执行、步进、状态读取和重置。场景组装准备资产与资源清单，并执行后端专属转换。独立进程隔离依赖并支持不同更新频率；相机与外观设置可独立于物理模型选择。

Explore this chapterProtocol

3.1A common protocol boundary

The GenieSim Protocol separates control, physical state evolution, and visual observation generation. It exchanges control commands, joint states, rigid-body poses, simulation time, and geometry updates for deformation. Physics owns state; rendering consumes it, with scene metadata connecting physical entities to their visual representations.

Physics and rendering adapters share the GenieSim Protocol. MuJoCo also participates through solver integration and the CPU robot-dynamics worker used by IPC.

3.2Select the physics and rendering path

Backend roles

Component	Role
PhysX	Isaac Sim physics path
Newton	Standalone and Isaac-integrated execution
libuipc	Rigid–deformable contact simulation
MuJoCo	Solver integration and cooperating CPU robot dynamics
Isaac Sim / OVRTX	Alternative rendering paths over shared state interfaces

Adapters implement initialization, command application, stepping, state readback, and reset. Scene assembly prepares assets and a resource manifest, with backend-specific conversions. Separate processes isolate dependencies and allow distinct update rates. Camera and appearance settings can be selected independently of the physical model.

CHAPTER 04

Teleoperation and automated data collection遥操作与自动化数据采集

切换阅读主题遥操作

4.1遥操作

PICO VR 支持双臂、腰部与移动底盘控制。控制器位姿、按键与摇杆输入被转为运动目标，经逆运动学和轨迹处理后驱动仿真机器人；动作捕捉手套提供灵巧手输入。机器人状态与多视角图像同步记录。

Both teleoperation and automated collection pass through replay, multi-view re-rendering, and data quality control.

Explore this chapterTeleoperation

4.1Teleoperation

PICO VR supports dual-arm, waist, and mobile-base control. Controller poses, buttons, and joysticks become motion targets; inverse kinematics and trajectory processing drive the simulated robot. Motion-capture gloves add dexterous-hand input. Robot states and multi-view images are recorded synchronously.

CHAPTER 05

Benchmark task suitesBenchmark 任务套件

5.1基础评测 Track

指令遵循

空间推理

抗扰动

通用操作

已有四个 Track 评估具身模型的基础能力。指令遵循考察模型能否将语言指令转化为物体选择与操作；空间推理关注相对位置、朝向与空间关系；通用操作覆盖常见物体交互与多步骤操作；抗扰动评估外观及环境条件变化下的任务表现。

这四个 Track 为 VLA 策略、WAM 策略和模型驱动的 Agent 提供共同的比较基础。新增 Track 在此基础上进一步考察更复杂的操作能力及跨本体迁移。

5.2新增评测 Track

四类新增任务套件灵巧手

灵巧手操作

颠锅

任务交互演示0:21

颠锅

手指持续控制锅柄，机械臂协调完成颠锅动作，展示工具抓持与多指协同。

静音循环 · 点击或按空格暂停

抓剪刀

任务交互演示0:20

抓剪刀

手指接近剪刀并围绕手柄建立接触，随后从桌面提起工具，展示多指抓持与工具操作。

静音循环 · 点击或按空格暂停

叠碗

任务交互演示0:39

叠碗

手指进入碗内并提供支撑，双手协调提起与放置，展示对碗口几何和抓持方式的适配。

静音循环 · 点击或按空格暂停

取书

任务交互演示0:57

取书

机器人伸手进入书架，抓取并抽出书本，在受限空间中保持手指接触和书本朝向。

静音循环 · 点击或按空格暂停

该套件使用搭载 Wuji 灵巧手的 G2 平台，关注多指如何参与物体交互。取书、剪刀操作、双手搬运包裹、盖杯盖和叠碗共同覆盖手指构型选择、抓取后的持续操作及双手协作，将手部动作的功能性纳入评测。

剪刀操作要求同一只手的拇指和另一根手指分别进入两个手柄孔，并协调运动来驱动开合。普通平行夹爪无法复现这种入环与驱动方式。取书任务则要求在密集书架中先倾斜目标书本，再将其取出，考察接触选择与取物顺序。

包裹搬运强调双手控制同一物体；盖杯盖与叠碗连接抓取、接近、对齐和放置。任务比较有助于识别功能性抓取规划、持续操作和双手协调中的相对短板。

5.3Benchmark Results

我们使用此前技术报告提出的 Agent2Act 框架 [1]，接入 GPT-6 Astra、Claude Opus 5.5 和 Gemini 4 Argon 三个前沿 LLM，在灵巧手操作、全身协同、精细操作和跨本体操作四个新增 Track 上进行评测，得到如下 Score。

[1] Huang 等，Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum，技术报告，2026。arXiv:2610.27865。

三个 Agent 的四类新增任务评测

任务套件	GPT-6 Astra	Claude Opus 5.5	Gemini 4 Argon
精细操作	18.86	19.62	12.84
跨本体操作	16.94	14.61	12.29
全身协同	13.65	7.63	11.60
灵巧手操作	0.41	7.05	2.40
平均值	12.47	12.23	9.78

三个 Agent 在四个新增任务套件上的报告 Score。平均值沿用源表；粗体标出各行最高分。

报告中的平均 Score 为 Astra 12.47、Opus 12.23、Argon 9.78。Astra 与 Opus 的平均值仅相差 0.24，但优势分布不同：Astra 领先跨本体操作与全身协同，Opus 领先精细操作与灵巧手操作。

实验结果分析

精细操作

Opus 得分 19.62，Astra 为 18.86。报告中的逐任务分析进一步区分了两者的策略特点：Opus 在手机打包上表现更好，Astra 在盖笔帽和投硬币上领先。抓持方式、接近方向及接触调整需要适配具体物体与操作。

全身协同

Astra 以 13.65 领先，Argon 为 11.60，Opus 为 7.63。此类任务把底盘移动、身体姿态和机械臂操作连接为连续过程，要求机器人在位置和构型变化时持续控制物体。

灵巧手操作

这是三个 Agent 各自得分最低的套件：Opus 为 7.05，Argon 为 2.40，Astra 为 0.41。以剪刀操作为例，手指需要进入手柄环、建立稳定抓持，并协同完成开合。多指协同和功能性工具使用仍是主要困难。

跨本体迁移

Astra 在跨本体操作上以 16.94 领先，Opus 为 14.61，Argon 为 12.29。报告还指出，排名随机器人变化：Astra 在 G2 Omnipicker 上领先，Opus 则在 AgileX 上领先。

同一任务目标需要适配不同机器人的可达范围、末端几何与相机布局。在一个机器人上的表现无法完整预测另一个机器人上的表现；套件总体得分与具体本体上的行为共同为模型选择和本体适配提供依据。

5.4开放世界 Genie Sim Sandbox

开放世界仿真环境面向下一代具身智能的仿真与评测，将导航、环境交互与物体操作融入多样化场景和多阶段任务，用于考察智能体完成复杂任务的综合能力。

示例覆盖家居、便利店、餐厅、酒店与办公室五个典型场景。

场景浏览 · 每页五个任务家居

送早餐

家居