Cross‑Embodiment Transfer via Latent Observation Alignment
1Stanford University2University of Southern California3UC Berkeley
*Equal contribution†Equal advising‡Corresponding author
Conference on Robot Learning (CoRL) 2026
TL;DR A vision policy trained on one robot fails on another, because the arm it sees in the image has changed. UniLatent closes this gap in the policy’s feature space instead of in pixels. It trains the source robot’s image encoder to produce the features that a frozen encoder gives the target robot in the same pose, using paired renderings from simulation. Source demonstrations encoded this way look to the policy as if they came from the target robot, so the policy runs on the target directly, with no image translated or generated.
Abstract
Robot datasets are costly to collect and are usually tied to a specific embodiment, making it difficult to reuse demonstrations across robots with different geometry, appearance, and kinematics. We introduce UniLatent, a cross-embodiment transfer framework that reduces the embodiment-induced observation gap in a shared latent space. UniLatent renders paired source–target robot observations in simulation and aligns a source visual encoder to a frozen target encoder. A policy trained on encoded source demonstrations can then run directly on target observations, with no pixel-level image translation. Across three simulation tasks and two target robot arms, UniLatent reaches up to 93% zero-shot success, exceeding the strongest pixel-translation baseline by over 30 points on average. In real-world xArm-to-Franka transfer, fine-tuning with only 10 target demonstrations per task yields 27/30 successful rollouts.
Overview video
The problem
The two arms below share the same end-effector action interface, so the actions themselves transfer. What does not transfer is the image: in a third-person view the robot fills much of the frame, and swapping the arm changes its appearance, how it occludes itself and what the camera sees of the contact. On Stack Cup into Bowl, both policies reach 80–89% on the Franka they were trained on, and 0% on Sawyer.
Prior work closes the gap in pixel space: it edits the source frames by masking, inpainting or diffusion-based generation until the source robot looks like the target. The policy then learns from edited images but acts on real ones, so segmentation errors, distorted grippers and misplaced contact regions become a mismatch between training and deployment.
The idea
UniLatent removes the embodiment gap inside the policy’s visual representation. A frozen, pretrained encoder ET defines the features the policy will receive on the target robot. A second encoder ES is trained so that an image of the source robot produces the features ET assigns to the target robot in the same pose. Demonstrations recorded on the source robot and encoded by ES then look to the policy as if they had been recorded on the target, so the same policy runs on the target robot through ET.
Only ES is trained. ET is frozen, so the features the alignment aims for are exactly the features the deployed policy reads.
The policy trains and acts in one representation. Pixel translation, by contrast, trains on synthesized images and acts on real ones.
Paired images of both robots at the same tool-center-point pose are rendered in simulation from robot models, inverse kinematics and randomized scenes.
Alignment needs no demonstrations on either robot, so an encoder aligned on generic pairs can be reused across tasks.
ET is off-the-shelf DINOv2, pretrained on real images. Simulation only teaches ES to map into that fixed real-image representation.
On the real xArm and Franka, one encoder aligned on about 100k generic rendered pairs serves all three tasks.
Align observations. Each pair shows two robots at the same tool-center-point pose, from the same camera in the same randomized scene. ES is trained with an MSE loss on CLS and patch tokens to output, for the source image, the features the frozen ET gives the target image. After training, both images of a pair map to the same representation.
Alignment over training. Left: t-SNE of Franka (source) and Sawyer (target) embeddings of paired renderings during alignment training. The two embodiments start as separate clusters, merge, and end with each Franka point on its paired Sawyer point. Right: decoding aligned Franka features back to pixels shows an increasingly Sawyer-like robot, a visual check that ES now encodes the target embodiment. The policy never uses these images.
Train the policy. Source demonstrations are encoded with the aligned ES, now frozen. A flow-matching policy learns to predict chunks of 8 end-effector actions (position, orientation, gripper) from the last 3 latent observations. It trains on target-aligned features, never on pixels.
Deploy on the target. The target robot’s camera images go through the frozen ET, the encoder that defined the alignment, and the same policy acts on them in closed loop. No image is translated or generated at test time.
Results · simulation
Each policy is trained on 300 Franka demonstrations and run on xArm7 and Sawyer with no target demonstrations. The three arms share a Cartesian end-effector action interface but differ in link geometry, joint limits, gripper mechanism and appearance, so the robot in the image is the only thing that changes. The tasks stress articulated contact, grasp-and-place precision and nonprehensile pushing.
Articulated contact
Grasp-and-place precision
Nonprehensile pushing
Each clip is one UniLatent rollout. For display, all clips use the same rendered background; the demonstrations and evaluation rollouts themselves are rendered with randomized backgrounds, lighting and materials. Percentages are UniLatent’s success over 100 rollouts on that robot (Table 1).
The source Franka in five more of the rendered backgrounds.
Zero-shot success over 100 rollouts, trained on Franka.
On Stack Cup with Sawyer, every baseline is at or below 5%. UniLatent reaches 51%.
| Method | Close Box Lid | Stack Cup into Bowl | Kick Football | Target mean | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| F | X | S | F | X | S | F | X | S | ||
| Direct transfer | 97 | 2 | 5 | 80 | 0 | 0 | 90 | 1 | 1 | 1.5 |
| Frozen DINOv2, no alignment | 97 | 40 | 16 | 89 | 3 | 0 | 93 | 11 | 1 | 11.8 |
| RoVi-Aug* (pixel translation) | 97 | 93 | 54 | 89 | 15 | 5 | 93 | 60 | 40 | 44.5 |
| UniLatent (ours) | 97 | 93 | 81 | 89 | 65 | 51 | 93 | 89 | 77 | 76.0 |
Success rate (%) over 100 rollouts. F = Franka (source, in-domain), X = xArm7, S = Sawyer (zero-shot). Target mean averages the six xArm7 and Sawyer cells. RoVi-Aug* uses the same paired renderings as UniLatent but executes in pixel space, so the gap isolates latent versus pixel transfer.
Results · real robots
Demonstrations come from a real xArm, and the target is a real Franka seen from the same camera. One task-agnostic encoder serves all three tasks. It was aligned only on about 100k generic simulated pairs, with off-the-shelf DINOv2 as the frozen target encoder. The physical robots also differ in controllers and calibration, so the latent policy is pretrained on the xArm data and fine-tuned with 10 Franka demonstrations per task.
Left: one of the xArm teleoperated demonstrations used for pretraining. Right: UniLatent on the real Franka after fine-tuning on 10 Franka demonstrations. The score is UniLatent’s success over 10 rollouts.
UniLatent’s real-robot failures, out of 10 rollouts per task. Close Drawer had none. In both, the arm follows the task motion and the failure happens at contact.
| Setting | Close Drawer | Close Box Lid | Wipe Table |
|---|---|---|---|
| Source-only (xArm, in-domain reference) | 10/10 | 8/10 | 10/10 |
| Target-only, few-shot (Franka) | 7/10 | 2/10 | 2/10 |
| RoVi-Aug pretrain + few-shot | 9/10 | 9/10 | 5/10 |
| UniLatent pretrain + few-shot (ours) | 10/10 | 9/10 | 8/10 |
Source-only trains and evaluates on xArm, an in-domain reference rather than a transfer result. The other rows run on Franka with the same 10 Franka demonstrations and differ only in how the xArm data is used: not at all (target-only, 11/30), translated into Franka images by the diffusion-based RoVi-Aug pipeline (23/30), or encoded into the aligned latent space (UniLatent, 27/30).
Why latent transfer helps
Pixel translation and UniLatent both map the source robot onto the target, but in different places. A translated frame can keep the global layout while getting the wrist, the gripper opening or the contact region subtly wrong, and the policy trained on those frames then meets real target images at test time. UniLatent’s policy trains and acts on features from one frozen encoder, so there is no such mismatch.
What we compare. Five xArm frames (top row) are translated into the Franka domain by three strategies and set against the true Franka rendering at the same end-effector pose (bottom row). A perfect translation would match the bottom row.
Diffusion cross-painting (SDXL + ControlNet) segments out the source robot and paints the target robot in its place.
Often plausible, but links and joint structure become unstable under hard poses and occlusions. About 10 s per frame, roughly 17 minutes per 10 s demonstration.
Segmentation masking removes the source robot from the image.
It avoids hallucinating a robot, but also erases the gripper and the contact evidence near the object.
Our controlled baseline. It decodes the aligned ES features into images with a pretrained decoder, using the same paired renderings, encoder and demonstrations as UniLatent.
The most faithful translation, so the only remaining difference is whether the policy learns from translated pixels or from aligned features: 44.5% against 76% mean target success.
RoVi-Aug*, the strongest baseline in simulation, uses the same paired renderings to translate source demonstrations into target-robot images, then trains a pixel-space policy on them. UniLatent never edits an image: it trains and acts in the frozen target encoder’s feature space. Rollouts below are zero-shot from Franka demonstrations, with success over 100 rollouts (Table 1).
xArm7 success
Sawyer success
xArm7 success
Sawyer success
xArm7 success
Sawyer success
Further analyses
| Source robot(s) | Demos | Success |
|---|---|---|
| Sawyer | 100 | 64 |
| Franka | 100 | 63 |
| Sawyer | 200 | 84 |
| Franka | 200 | 92 |
| Sawyer + Franka | 200 | 91 |
Franka and Sawyer each get a source encoder aligned to the same frozen xArm encoder, and one policy trains on both sets of demonstrations. 100 from each source (91%) match 200 Franka demonstrations (92%). Close Box Lid, Table 3a.
| Alignment | Deploy v1 | Deploy v2 |
|---|---|---|
| Matched view (v1↔v1) | 97 | 62 |
| Cross-view (v1↔v2) | 96 | 89 |
The pairs can also be two camera views of the same robot. Aligning view v1 to v2 lifts success on v2 from 62% to 89% while keeping 96% on v1. Close Box Lid, Table 3b.
When the Franka demonstrations are rendered with randomized backgrounds, the transferred policies keep most of their success in five held-out scenes: visual diversity added on the source side carries over to the target robot through the aligned latent space.
| Close Box Lid | Franka (source) | xArm7 | Sawyer |
|---|---|---|---|
| Randomized source backgrounds, 5 held-out scenes | 88.6 | 85.6 | 80.0 |
| Matched training and evaluation backgrounds | 97 | 93 | 81 |
Simulated background generalization (Table 3c), reported for Close Box Lid only. Success rate (%).
Limitations
UniLatent currently targets 7-DoF single-arm robots with parallel-jaw grippers and a shared end-effector action interface. Substantially different morphologies, such as dexterous hands, humanoids or mobile manipulators, and tasks that require tool-specific contact would need additional retargeting and contact modeling. Our tasks were designed so that both robots perform the same task in the same scene; applying UniLatent to external source datasets such as Open X-Embodiment remains to be demonstrated. The method also assumes the target robot can complete source tasks in a comparable workspace. Finally, the background-shift evaluation covers one simulated task and the real-world experiments use a single physical setup, so robustness to real-world clutter, lighting, distractors and broader object variation is not yet established.
Citation
Coming soon. The BibTeX entry will be posted here once the paper is on arXiv.