RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning
Lawrence Yunliang Chen Chenfeng Xu Karthik Dharmarajan Zubair Irshad Richard Cheng Kurt Keutzer Masayoshi Tomizuka Quan Vuong Ken Goldberg

^*Equal Contribution
^1UC Berkeley
^2Toyota Research Institute
^3Physical Intelligence

Tight minimum degree condition to guarantee C_2k+1-free graphs to be r-partite
     Zilong Yan [School of Mathematics, Hunan University,
 Changsha 410082, P. R. China. E-mail: zilongyan@hnu.edu.cn.]      Yuejian Peng [Corresponding author. School of Mathematics, Hunan University, Changsha 410082, P. R. China. E-mail: ypeng1@hnu.edu.cn. Supported in part by National Natural Science Foundation of China (No. 11931002)]        Xiaoli Yuan [School of Mathematics, Hunan University,
 Changsha 410082, P. R. China. E-mail: xiaoliyuan@hnu.edu.cn.] 


    
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
    Scaling up robot learning requires large and diverse datasets, and how to efficiently reuse collected data and transfer policies to new embodiments remains an open question.
    Emerging research such as the Open-X Embodiment (OXE) project has shown promise in leveraging skills by combining datasets including different robots. However, imbalances in the distribution of robot types and camera angles in many datasets make policies prone to overfit. 
    To mitigate this issue, we propose , which leverages state-of-the-art image-to-image generative models to augment robot data by synthesizing demonstrations with different robots and camera views. Through extensive physical experiments, we show that, by training on robot- and viewpoint-augmented data, can zero-shot deploy on an unseen robot with significantly different camera angles. Compared to test-time adaptation algorithms such as Mirage, requires no extra processing at test time, does not assume known camera angles, and allows policy fine-tuning. Moreover, by co-training on both the original and augmented robot datasets, can learn multi-robot and multi-task policies, enabling more efficient transfer between robots and skills and improving success rates by up to 30%. 



 







§ INTRODUCTION



Emerging research in robot learning suggests that scaling up data can help learned policies be more generalizable and robust <cit.>. However, compared to state-of-the-art foundation models <cit.> in computer vision (CV) <cit.> and natural language processing (NLP), the size of robotic data is still several orders of magnitude smaller than those used to train large language and multi-modal models <cit.>. Collecting real robot data is time-consuming <cit.> and labor intensive <cit.>, and ensuring data diversity for generalizable policies requires careful balance <cit.>. 
Can we more effectively leverage currently available real robot data?

In an unprecedented community effort, the Open-X Embodiment (OXE) project <cit.> combines 60 robot datasets and finds that co-training can exhibit positive transfer and improves the capabilities of multiple robots by leveraging experience from each other. However, the OXE dataset is highly unbalanced, dominated by a few robot types such as Franka and xArm. Additionally, most datasets have a limited diversity of camera poses. Policies trained on such data tend to overfit to those robot types and viewpoints and need fine-tuning when deploying on other robots or at even slightly different camera angles. To mitigate this issue, a test-time adaptation algorithm, Mirage <cit.>, uses “cross-painting” to transform an unseen target robot into the source robot seen during training, to create an illusion as if the source robot is performing the task at test time. While Mirage can achieve zero-shot transfer on unseen target robots, it has a few limitations: (1) It requires precise robot models and camera matrices; (2) It does not allow policy finetuning; (3) It is limited to small camera pose changes due to depth reprojection error.



In this work, we seek to bridge these limitations. Rather than naively co-training on combined data from multiple robots, we aim to more explicitly encourage the model to learn the cross-product of the robots and skills contained in each dataset. We aim to improve the robustness and generalizability of the policy to different robot visuals and camera poses during training instead of relying on an accurate test-time cross-painting pipeline. We propose , a robot and viewpoint augmentation pipeline that synthetically generates images with different robot types and camera poses using diffusion models. Through extensive real-world experiments, we show that, by training on robot- and viewpoint-augmented data, can zero-shot control different robots with significantly different camera poses compared to the poses seen during training. In contrast to Mirage, does not assume known camera matrices and allows policy fine-tuning to increase performance on challenging tasks. Furthermore, by co-training on original and augmented robot datasets, can learn multi-robot and multi-task policies and improve finetuning sample efficiency.


This paper makes 3 contributions:


    
  * , a novel approach to robot data augmentation that uses diffusion models to generate trajectories with novel robots and viewpoints;
    
  * Physical experiments with Franka and UR5 suggesting that robot augmentation enables zero-shot deployment on target robots and viewpoint augmentation improves the robustness of policies to camera pose changes. When combined, they yield policies that work for target robots at camera poses significantly different from those in the initial demonstration data;
    
  * Experiments suggesting that can learn multi-robot multi-task policies and improve the finetuning sample efficiency of a generalist policy on novel robot-task combinations.












§ RELATED WORK





 §.§ Cross-Embodiment Robot Learning



Recognizing the high cost of collecting real robot data, many prior works have studied using other data sources, such as simulation <cit.>, other robot data <cit.>, and human or animal videos <cit.>, to increase sample efficiency and accelerate learning <cit.>. In a transfer learning setting, one can first pretrain a visual encoder <cit.>, dynamics model <cit.>, or policy <cit.> and then perform online finetuning using reinforcement learning. In a cross-domain imitation paradigm, methods often involve learning correspondences between the source and target domains <cit.>, and then constructing auxiliary rewards <cit.> or applying adversarial training <cit.>.  <cit.> use meta-learning to enable a new robot to quickly learn from few-shot trajectories at test time.

Cross-embodiment learning could also be used to learn more robust and generalizable policies through joint training in a multi-robot multi-task fashion. For example, by training on a family of robots with varying kinematics and dynamics in simulation, robot-conditioned policies <cit.> are robust to novel morphologies within the range of training distribution, and modular policies <cit.> can be more transferrable to different robots and tasks. More recently, many works have also explored training on large and diverse real robot data <cit.> to learn visual representations <cit.> and predictive world models <cit.> and showed that policies trained are more generalizable to new objects, scenes, tasks, and embodiments <cit.>. 
In this work, we build on these insights and propose to more explicitly encourage positive transfer between robots and skills by performing data augmentation. 


Our method is inspired by Mirage <cit.>, a recent test-time adaptation algorithm that uses “cross-painting” to achieve cross-embodiment policy transfer by replacing the target robot in the image with a source robot seen during training. While Mirage avoids modifying the source robot policy and enables zero-shot transfer, it has several limitations, such as requiring a fast renderer, precise robot models, and accurate camera calibration. We address these issues by using training time data augmentation with diffusion models trained on randomized robot poses and camera angles, eliminating the need for camera matrix knowledge. Our approach additionally allows zero-shot deployment as well as finetuning or cotraining on additional data to improve the performance and learn multi-robot multi-skill policies that are robust to significant camera angle changes.






 §.§ Generative Models and Data Augmentation in Robotics



With the significant progress in generative models including large language and multi-modal models <cit.> and diffusion models <cit.> trained on Internet-scale data, there is a growing interest in leveraging these models for robotics. For example, prior work has explored using language models for planning <cit.>, control <cit.>, reward specification <cit.>, and data relabeling <cit.>. Image and video generation models have been used for generative simulation <cit.>, data augmentation <cit.> and visual goal planning <cit.>. Our method falls into the data augmentation category. However, unlike prior work that generates distractor objects, backgrounds, and new tasks <cit.>, we use diffusion models to generate alternative robots and camera viewpoints. As such, enables trained policies to generalize to different robots with different camera setups.








 §.§ Viewpoint Adaptation and Viewpoint Robust Policy




Visuomotor control policies that take in images as inputs tend to overfit to the camera angle in the training data, and even small changes between training and testing could severely hurt performance <cit.>. While using 3D representations <cit.> alleviates the problem, it requires a calibrated depth camera or multiple views <cit.>, and is more computationally expensive. For mobile robots, <cit.> extract a 3D point cloud from the training data and performs re-rendering, and Ex-DoF <cit.> applies virtual rotation of the robot's 360^∘ camera to augment training data. To improve viewpoint robustness of image-based policies, <cit.> use a recurrent neural network to understand how actions affect arm movement through history. <cit.> use many simulated viewpoints to learn a visual representation, whose downstream policy exhibits viewpoint robustness. Instead of pretraining in simulation with diverse rendering, we synthesize novel views of real scenes. SPARTN <cit.> and DMD <cit.> use neural radiance fields (NeRFs) and diffusion models, respectively, to generate perturbed viewpoints for wrist cameras, whereas our viewpoint augmentation applies to fixed third-person views.


	






§ PROBLEM STATEMENT







We assume a demonstration dataset 𝒟^𝒮 = {τ_1^𝒮, τ_2^𝒮, ..., τ_n^𝒮} consisting of n successful trajectories of a source robot 𝒮 performing some task. Each trajectory τ_i^𝒮  = ({o_1..H_i^𝒮}, {p_1..H_i^𝒮}, {a_1..H_i^𝒮}), where {o_1^𝒮, ..., o_H_i^𝒮} is a sequence of RGB camera observations, {p_1^𝒮, ..., p_H_i^𝒮} is the sequence of corresponding gripper poses, and {a_1^𝒮, ..., a_H_i^𝒮} is the sequence of corresponding robot actions. This dataset can be used to train models with behavior cloning for robot 𝒮. 
Our goal is to augment 𝒟^𝒮 into 𝒟^Aug such that we can learn a policy that can be successfully deployed on a different robot 𝒯, known as the target robot, with a potentially different camera viewpoint. In this work, we focus on robot arms mounted on a stationary base and assume the grippers are similar in shape and function. 


Similar to prior work <cit.>, we use Cartesian control and assume knowledge of the two robots' end effector coordinate frames with respect to their bases (e.g., moving forward corresponds to an increase in the x-axis) such that we can use a rigid transformation T_𝒯^𝒮 to preprocess the data and align the robots' end effector poses p^𝒮 = T_𝒯^𝒮 p^𝒯 and actions a^𝒮 = T_𝒯^𝒮 a^𝒯 into the same vector space. Thus, for notational convenience, we omit the superscript differentiating gripper poses and actions between the robots. However, the image observations o^𝒮 and p^𝒯 cannot be easily aligned since the robots may look very different. We do not assume knowledge of the camera matrices in either setup.

After augmentation, we learn a policy π(a_t | o_t^𝒯, p_t) on 𝒟^Aug using imitation learning. At test time, it takes as inputs the observations from the target robot and outputs actions that can be deployed on the target robot. Additionally, by co-training on the original data 𝒟^𝒮 as well as 𝒟^Aug, we can also obtain a multi-robot policy.




	







§ 










In this section, we describe , an automated pipeline for augmenting and scaling up robot data. Our key insight is that the robot's actions should be invariant to its visual appearances and camera viewpoints. 
Our robot augmentation pipeline leverages state-of-the-art diffusion models <cit.> to synthesize alternative robots and novel viewpoints. Fig. <ref> illustrates pipeline.





 §.§ Robot Augmentation (Ro-Aug)



Given a sequence of robot image observations D_i^𝒮 = {o_1^𝒮, ..., o_H_i^𝒮}, we seek to transform the robot 𝒮 in the images into a different robot 𝒯 at the same gripper pose, a process known as cross-painting. While Mirage <cit.> proposes to perform cross-painting using a renderer to compute source robot masks and target robot visuals, it requires precise camera calibration which is unavailable for most open-source datasets. To relax this assumption, we approach cross-painting as an image-to-image translation problem.  begins by predicting semantic mask on the robot 𝒮, which are then extracted and transformed into robot 𝒯 using a robot-to-robot (R2R) diffusion model. Meanwhile, the masked regions in the original images are inpainted using a video inpainting network to ensure visual continuity and integrity. Finally, the generated robot 𝒯 is pasted back into the background image (see Fig. <ref>). 


Robot Segmentation.
In order to replace robot 𝒮 with robot 𝒯 in the image, we first need to detect the robot using semantic segmentation <cit.>. We find that off-the-shelf segmentation models <cit.> often fail to accurately segment out the robot, potentially due to the fact that robot images are under-represented in their training data. 
As such, we finetune a pretrained Segment Anything Model (SAM) <cit.> using Low-Rank Adaptation (LoRA) <cit.>. We use simulation to synthetically generate a large dataset of different robot images with corresponding masks, where we randomly sample a wide range of camera and robot poses. We apply brightness augmentation and resizing to simulate different lighting and fields of view. To create diverse backgrounds, we paste the generated robot parts into various background images <cit.>.
By training the LoRA layer on this synthetic dataset, we obtain a mask model capable of handling different robot and camera poses.






Robot-to-Robot (R2R) Generation
Next, we aim to transform the segmented robot 𝒮 into robot 𝒯. We use an image-to-image diffusion model.
Similar to semantic segmentation, training a diffusion model capable of handling various camera and robot poses requires a large dataset of paired images. As collecting paired real robot data is challenging due to the need for precise adjustments of camera and robot poses, we again use simulation to generate pairs of robots at the same randomly sampled robot poses and camera poses, with brightness and resizing augmentations. Inspired by <cit.>, we use a ControlNet <cit.> to finetune a pretrained Stable Diffusion <cit.>. Even though we train the model on simulation images, we find that it still performs well on real segmented robot images.


Robot Inpainting
Inspired by <cit.>, after segmenting out robot 𝒮 from the image, we inpaint the missing region using a video inpainting model E^2FGVI <cit.>. The final step involves pasting the generated robot 𝒯 back to the image. As the R2R diffusion model is trained on simulated robot images, there is a visual gap from the real robot, particularly with the illumination. To prevent the trained policy on the augmented data from overfitting to the synthetic robot visuals, we perform random brightness augmentation to the generated robot before pasting it. We find in our experiments that this randomization significantly helps the performance of the trained policy (Section  <ref>).


At the end of the Robot Augmentation pipeline, we obtain a sequence of cross-painted observations with synthesized target robot: D_i^𝒮→𝒯 = {o_1^𝒮→𝒯, ..., o_H_i^𝒮→𝒯}.






 §.§ Viewpoint Augmentation (Vi-Aug)





To increase robustness of the trained policy to camera pose changes, we propose to augment the viewpoints of the images. This is orthogonal to robot augmentation and can be applied to both D_i^𝒮 and D_i^𝒮→𝒯. 


We use ZeroNVS <cit.>, a state-of-the-art 3D-aware diffusion model that can zero-shot synthesize 360^∘ view of a scene from a single image. Compared to prior methods <cit.> that are limited to segmented object with no background, ZeroNVS works with multi-object scenes with complex backgrounds.
For each image o_t ∈ D_i, we uniformly sample perturbations (R̃_t, T̃_t) ∈ SE(3) from a box range, where each component in T̃_t is bounded by an interval. We parametrize R̃_t with Euler angles and each of those three angles is uniformly sampled within an interval described in Section <ref>. This process produces a resulting image as if the camera were perturbed by the sampled transformation:
o^R̃, T̃_t = f ( o_t; R̃, T̃), where we use f to denote the camera transformation. We denote the resulting augmented data as D_i^Vi-Aug = {o_1^R_1, T_1, ..., o_H_i^R_H_i, T_H_i}.
We experiment with two strategies for sampling the perturbations: independently sampling random (R̃_t, T̃_t) for each image, or applying a consistent random transformation (R̃, T̃) across the entire trajectory in D_i.







 §.§ Policy Training



After applying robot and viewpoint augmentation, we can train a policy π based on the Diffusion Policy architecture <cit.> on the augmented dataset 𝒟^𝒮→𝒯^Vi-Aug and zero-shot deploy the policy on the target robot 𝒯. For challenging tasks or when there is a large difference in the dynamics between the robots, we can also collect a small demonstration dataset 𝒟^𝒯 on the target robot directly and few-shot finetune π on 𝒟^𝒯 to further improve policy performance. Alternatively, we can co-train π on 𝒟^𝒮^Vi-Aug⋃𝒟^𝒮→𝒯^Vi-Aug to obtain a multi-robot policy. 
Additionally, if we have multiple datasets with different tasks, can mix-and-match the datasets and train a multi-robot multi-task policy. For example, given data 𝒟_1^𝒮 and 𝒟_2^𝒯 with robot 𝒮 performing task 1 and robot 𝒯 performing task 2, we can train on the cross-product 𝒟_1^𝒮⋃𝒟_2^𝒯→𝒮⋃𝒟_1^𝒮→𝒯⋃𝒟_2^𝒯 and their viewpoint-augmented versions to obtain a policy that can perform both tasks on both robots. In this way, we efficiently reuse the datasets and explicitly encourage transfer between robots and skills.









§ EXPERIMENTS









 §.§ Implementation Details

To train our robot segmentation and Robot-to-Robot generation models, we use the Robosuite simulator <cit.> to generate a large dataset of paired robot images with corresponding masks with randomly sampled robot poses and camera poses (see supplementary material for details). We use 4 robots: Franka, UR5, Sawyer, and Jaco, with  800k images each. We finetune a LoRA layer while keeping SAM frozen with a learning rate of 1e-4 for just one epoch to avoid the overfitting. We train a ControlNet for each robot pair based on Stable Diffusion v1.5 <cit.> with a learning rate of 1e-4 for 20k steps. During robot inpainting, we randomly sample perturbations of the value channel in the HSV space between -30 and 30.

For view augmentation sampling, T̃_x, T̃_z  ∈ (-0.25 m, 0.25 m), T̃_y ∈ (-0.1 m, 0.1 m). The y (vertical) direction has a lower translation range, as we have noticed that when moving excessively along the vertical direction, ZeroNVS outputs larger, more distracting artifacts. For rotation, we sample each Euler angle between ± 0.1 radians.







 §.§ Experiment Setup

We design experiments to answer the following research questions:
(1) Can robot augmentation (Ro-Aug) effectively bridge the visual gap between the robots?
(2) Can viewpoint augmentation (Vi-Aug) improve policy robustness to camera pose changes?
(3) Can policies trained with be successfully deployed zero-shot on a different robot with camera changes?
(4) Does enable multi-robot multi-task training and better facilitate transfer between robots and skills?










To answer the first three questions, we study policy transfer between a Franka and a UR5 robot on 5 tasks (Fig. <ref>): (1) Open a drawer, (2) Pick up a toy tiger from the table and put it into a bowl (Place Tiger), (3) Stack cups, (4) Sweep cloth from right to left, and (5) Transport a toy tiger between two bowls. See the Appendix for more details. For the first three tasks, we collect demonstrations on the Franka, and for the latter two, we collect demonstrations on the UR5. All demonstrations are collected via teleoperation at 15 Hz <cit.>, with 150 trajectories each. A typical trajectory consists of 75-120 timesteps (5-8 s). We use a ZED 2 camera positioned from the side for each robot. We augment the demonstration data with robot augmentation (Ro-Aug) using the other robot, viewpoint augmentation (Vi-Aug), as well as both (), train a diffusion policy, and evaluate on the other robot. All experiments are evaluated with 10 trials each.


To answer the last question, we combine demonstration data from Franka and UR5 for different tasks, perform robot augmentations, and train a multi-robot multi-task diffusion policy. We also select the Berkeley UR5 dataset <cit.> from the OXE data <cit.>, apply to generate synthetic Franka images and finetune a generalist policy, Octo <cit.>, on the augmented datasets. We additionally collect 50 demonstrations on the target robot (Franka) and further finetune Octo-Base in a language goal-conditioned format. We compare whether training Octo on the augmented data improves the finetuning sample efficiency on the downstream tasks.




    









 §.§ Results


Table <ref> shows the effect of robot augmentation when the camera poses are the same. The policy is deployed zero-shot.  We compare Ro-Aug with 2 baselines, no augmentation and Mirage, and an ablation that does not apply random brightness augmentation during the Ro-Aug pipeline. Without robot augmentation, the policy trained on the source robot only barely achieves success on the target robot. On the other hand, Ro-Aug achieves comparable zero-shot performance as Mirage. 
Additionally, we see that brightness randomization helps performance, suggesting that it effectively prevents the policy from overfitting to the lighting in simulation that the R2R model is trained on.





Table <ref> shows the policies trained on Ro-Aug data can be finetuned with 5-10 demonstrations on the target robot to further improve performance. Compared to few-shot policies trained without Ro-Aug, we see that Ro-Aug improves finetuning sample efficiency and exceeds the performance of all policies in Table <ref>. In contrast, Mirage does not allow finetuning and cannot improve performance on challenging tasks such as cup stacking.
 
Table <ref> evaluates the effect of viewpoint augmentation. We choose the Tiger Place task on the Franka robot and study how different strategies of camera perturbation sampling affect policy robustness. We sample translations T̃_x and T̃_z between ± 0.1 m,  ± 0.25 m, and ± 0.4 m, and compare consistent perturbation across trajectories or independently on each image. From Table <ref>, we see that larger variation during augmentation improves policy robustness under severe camera pose changes. However, the performance decreases under the original camera angle, potentially due to lower density of each camera pose as the sampling range increases. Additionally, inconsistent augmentation seems to slightly outperform consistent augmentation., suggesting potential benefit from more augmentation. We note that the diffusion policy takes in only 2 steps of history, so viewpoint inconsistency may not matter much. Future work can study whether inconsistent augmentation would harm policies that use a longer history. Based on the results, we choose to apply inconsistent augmentation with 25 cm perturbation range for other experiments.









[b]0.48
    max width=
    
        Franka           UR5      
 
        Place Tiger      80%               70%      

        Transport Tiger       60%         80%          

        
        
      tableRobot-Skill Cross Product. We train a multi-robot multi-task diffusion policy trained on pooling the Franka Tiger Place data and UR5 Tiger Transport data as well as their versions together.
      
      
    
    [b]0.48
        max width=
        
        
        2*[width=8em,trim=l]PoliciesTasks      2cOXE UR5 → Franka  

            (lr)2-3 
                  Sweep Cloth     Transport Tiger      

            Octo-Base           30%       20% 

            Octo-Base +     60%      40%  

            
            
      tableOcto finetuning from the OXE datasets with 50 in-domain demonstrations for each task.  improves finetuning sample efficiency.
      
    





Table <ref> evaluates on different robots with different viewpoints. We can see that viewpoint augmentation is crucial and Mirage struggles with larger camera pose changes. In contrast, can still achieve success  when the target robot viewpoint is significantly different from source robot.


To evaluate robot-skill cross-product, we combine the Tiger Place demonstration data from the Franka and Tiger Transport demonstration data from the UR5, as well as their robot-augmented UR5 and Franka versions, and train a multi-robot multi-task diffusion policy. From Table <ref>, we can see that the policy can successfully execute the two tasks on both robots. Additionally, we evaluate whether improves finetuning sample efficiency. From Table <ref>, we can see that after training Octo on the augmented OXE data, the policy has seen the synthetic target robots performing the tasks, accelerating downstream finetuning of similar tasks.










§ LIMITATIONS AND FUTURE WORK





We present , a pipeline for robot and viewpoint augmentation that bridges different robot datasets and better facilitates transfers between robots and skills. There are several limitations, which open up possibilities for future work: (1) Our robot augmentation pipeline relies on a sequence of different models so artifacts can cascade. For example, inaccuracies in the robot segmentation stage (e.g., mistakenly segmenting the object out) could lead to bad robot-to-robot generations in the second stage. See the Appendix for more details on artifacts. Additionally, instead of training an R2R diffusion model for each robot pair, future work could explore a unified model that handles multiple pairs. 
(2) For viewpoint augmentation, 
future work could improve the quality of novel view synthesis by finetuning the model on robotics data or using video-based models <cit.>. (3) While we mitigate viewpoint changes in this work, there are also often background changes in practice during cross-embodiment transfer. Future work could combine with prior orthogonal approaches such as object, background, and task augmentation <cit.> to further obtain more generalizable policies. 
(4) We only demonstrate transfer between stationary robot arms 
and do not consider very different grippers such as multi-fingered hands. We leave these extensions to future work.





This research was performed at the AUTOLab at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, and the CITRIS “People and Robots” (CPAR) Initiative, and in collaboration with Google DeepMind. The authors are supported in part by donations from Google, Toyota Research Institute, and equipment grants from NVIDIA. L.Y. Chen is supported by the National Science Foundation (NSF) Graduate Research Fellowship Program under Grant No. 2146752. We thank reviewers for valuable feedback.




  




§ APPENDIX

In this section, we provide additional implementation details of and our physical experiments.





 §.§ Algorithm Pseudocode

In this section, we provide the pseudocode for Ro-Aug and Vi-Aug.












 §.§ Robot Augmentation



  §.§.§ Training Data Generation


To train our robot segmentation and Robot-to-Robot generation models, we use the Robosuite simulator <cit.> to generate a large dataset of paired robot images with corresponding masks with randomly sampled robot poses and camera poses. The sampling procedure is as follows:
The robot pose is specified by the end-effector pose. The translation component is sampled uniformly with (x, y, z) ∈ [-0.25, 0.25] × [-0.25, 0.25] × [0.6, 1.3] (unit in meters). For the rotation component, we parameterize it as [inward, rightward, z_axis]. To bias the unit vector z_axis towards pointing downward, we parameterize it using spherical coordinate θ, ϕ where θ (zenith angle) is sampled from a normal distribution 𝒩(π, π / 3.5) and ϕ (azimuthal angle) is uniformly sampled between 0 and 2π.

After sampling the robot pose, we randomly sample the camera pose with the following procedure:
The position is sampled from a half hemisphere with radius r ∈ 𝒩(0.85, 0.2) and zenith angle θ∈𝒩(π/4, π / 2.2), and azimuthal angle ϕ∈Unif[-π·3.7/4, π· 3.7/4]. The viewing direction is towards the center of the hemisphere, which we offset as the gripper position. We also sample camera field of view between 40 and 70. Finally, we randomly perturb the camera pose with noises. 

We randomly sample robot poses, and for each robot pose, we randomly sample 5 different camera poses. In addition to pure random sampling, we also add some camera poses and robot poses similar to those in the RT-X datasets and add perturbations. We obtain paired images between different robots and their segmentation mask from Robosuite, and we add random brightness augmentation with range [-40, 40] to the source robot images to increase the robustness of the segmentation model and R2R model to real-world lighting. In this way, we obtain about 800k images for each of the 4 robot types: Franka, UR5, Sawyer, and Jaco. See Fig. <ref> for some example images.






To create the dataset for training the segmentation model, we paste the generated robot image onto backgrounds from ImageNet <cit.>. See Fig. <ref> for some example images.









  §.§.§ Model Training Details

Regarding the robot segmentation model, we fine-tune SAM with LoRA with 4 A6000 GPU for 1.5 hours. In particular, we leverage mixed-precision (8-bit and 16-bit) and the torch.compile feature to accelerate training. The model is trained with a mini-batch size of 64, a learning rate of 1e-5, and a LoRA rank of 4.


Regarding the Robot-to-Robot generation model, we finetune Stable Diffusion with ControlNet on 1 A100 GPU for 36 hours on 800K paired images. We use a learning rate of 1e-4 and a batch size of 512. During inference, we leverage the Stream Batch proposed by  <cit.> to batchify the generation phase, making the generation phase achieve around 3.2 FPS.

We use ZeroNVS and the video inpainting model E2FGVI off-the-shelf without finetuning.










  §.§.§ Computation Time for Data Augmentation

The advantage of over Mirage is that the primary of the compution is performed offline, not during execution time. Moreover, each model in RoVi-Aug’s pipeline can be parallelized to process batchified video frames efficiently. We measured the throughputs of each module: Robot segmentation model achieves 4.1FPS, Robot-to-Robot achieves 3.2FPS, and the video inpainting model achieves 4.6FPS. On a single A100 GPU, it takes about 4-5 hours to perform Ro-Aug on a dataset of 200 trajectories. Similarly, the throughput for ZeroNVS inference is 1.3FPS, translating to 4.2 hours of viewpoint augmentation time on a dataset of 200 trajectories. 













  §.§.§ Example Augmented Images


In Fig. <ref>, we show some example results of applied to the training images of the 5 tasks. The left column is the original images; the middle column is the cross-painted images using the robot augmentation pipeline; the right column shows the view augmented images applied on top of the robot augmented images. The black regions in the generated robot are due to incomplete segmentation mask (missing some regions in the generated robot) when pasting the generated robot to the original image. We can see that in general, generates diverse view angles of the target robot performing the task of interest.





  §.§.§ Generation Artifacts

We observe a few different types of artifacts: 1) illumination difference, 2) inaccurate object segmentation, 3) temporal inconsistency, and 4) inaccurate robot-to-robot generation.

For 1), since there are almost always differences in the lighting conditions between the simulated images that are used to train the R2R diffusion model and that of the test robots which are unknown a priori, we perform random brightness augmentation to the generated robot scenes in the augmentation pipeline. As shown in Table <ref>, we find this mitigation strategy is generally effective. 

For 2), the robot segmentation model may sometimes under-segment or over-segment, particularly when the source robot is occluded or interacting with objects. As the R2R diffusion model is not trained on source robot images with objects in the gripper or with a partially segmented robot, the generated target robot can have large artifacts including distortion or hallucination due to out-of-distribution inputs. 

For 3), due to the stochastic nature of diffusion models and possible multiple inverse kinematics solutions for putting the end effector of the target robot at the position of the source robot with different joint angles, the generated images may not be consistent across time. We did not observe this as a big problem potentially due to two reasons: (1) The Diffusion Policy does not use a long history so temporally inconsistent artifacts may not have a large effect; (2) The stochasticity of the generated images has an effect of randomization, which may help the policy be more robust to visual artifacts. 
Future work could also use a video diffusion model <cit.> to perform robot generation based on the entire robot trajectory to improve robot pose consistency.

For 4), even though our robot-to-robot diffusion model is trained on a large number of paired robot data, the generated images may still contain visible artifacts. For example, due to the ambiguity of inferring the field of view parameter from an image, the generated robot arm may be too thin or too thick. The generated gripper may also have artifacts or its position or orientation may not completely align with the source robot. 

Due to these artifacts, we observe in Table <ref> that Mirage achieves better performance than Ro-Aug on tasks that require more precision, such as cup stacking. This is because Mirage has the benefit of using a URDF with precise camera calibration to put the gripper at the exact location desired. On the other hand, artifacts in the R2R Generation model mean that the gripper of the target robot may not have the exact same pose as the original robot. However, as we show in Table <ref>, the ability of RoVi-Aug to perform finetuning can bring the performance higher than Mirage. 



 §.§ Physical Experiment Details

We provide more details on the physical experiment setups described in Section <ref>. 

For the Franka-UR5 transfer experiments, we study 5 tasks: (1) Open a drawer, (2) Pick up a toy tiger from the table and put it into a bowl (Place Tiger), (3) Stack cups, (4) Sweep cloth from right to left, and (5) Transport a toy tiger between two bowls. For each task, the initial position of the robot gripper is randomized. For (1), the position and orientation of the drawer on the table is randomized, and the goal for the robot gripper is to go into the handle, pull it out, and leave the drawer. For (2), the positions of the tiger and the drawer are randomized. For (3), the positions of both cups are randomized. For (4), the initial position of the cloth is randomized in the right region of the table, and the robot needs to push it to the left region of the table, a distance of about 0.5 m. For (5), there are 2 bowls (red and grey) whose positions are randomized, and the toy tiger is always in the red bowl initially. The robot needs to grasp it and drop it into the grey bowl. Among them, stacking cup requires high precision and is most difficult, and sweeping cloth is the easiest.


For the OXE dataset experiments, the 2 tasks from the Berkeley UR5 datasets (Transport Tiger, Sweep Cloth) are the same as (4) and (5) above. For the 2 tasks from the Jaco Play datasets, the “Pick Cup” task requires the robot to pick up a cup that is randomly initialized on the table, and the “Bowl in Oven” task requires the robot to pick up a bowl and put it into a toaster oven.




  §.§.§ Policy Learning Details

We use the codebase from DROID <cit.> as our Diffusion Policy implementation, which is an open-source version integrated with Robomimic <cit.>. Similar to them, We use downsample camera observations at a resolution of 128 × 128 and the robot proprioception as input, and produce absolute robot end-effector translation, rotation, and gripper actions. And as with DROID and the original Diffusion Policy implementation, we train the diffusion policy to generate 16-step action sequences, and during rollouts, step 8 actions open loop before re-running policy inference. Compared to DROID, we use a ResNet-18 visual encoder instead of a pre-trained ResNet-50 for faster training, and we do not condition the policy with language input since we train a separate policy for each task (or 2 tasks for Table <ref>). 


For few-shot finetuning experiments, we did not freeze any part of the diffusion policy and simply continued training on the target robot dataset (5/10 demonstrations) for only 100 epochs (about 20 minutes) to prevent the policy from overfitting to the target data too much.



  §.§.§ Failure Modes

We describe the common failure modes of and baselines here. 
For the 3 pick and place tasks (“Place Tiger,” “Stack Cup,” and “Transport Tiger”), failure cases are usually missed grasp or inaccurate placing. For “Open Drawer,” failure cases are typically gripper missing the drawer handle. For “Sweep Cloth," failure cases include inaccurate reaching and gripper being too high or leaving the table too early during the trajectory. For baselines, failure modes also include the robot getting confused and simply hovering over the objects without performing the task.




 §.§ Model and Computation Details

For Ro-Aug, our segmentation model is a 636M-parameter SAM model with 35.6M-parameter LoRA layers; the video inpainting model E2FGVI is a 41.8M parameter model that we use off-the-shelf; the Robot-to-Robot (R2R) Generation model is a 1B-parameter Stable Diffusion model with around 350M-parameter ControlNet. For Vi-Aug, ZeroNVS is a 1B model that we use off-the-shelf. For policy learning, we use Diffusion Policy with a ResNet18 encoder and 1D-UNet with 80M parameters in total.





 §.§ Example of Cross-Painting with a Mobile Robot





Generalization from arms mounted on a stationary base to mobile robots is much more challenging.
In this section, we try an experiment using images of the Franka arm and apply robot augmentation to replace the Franka with a Boston Dynamics Spot to illustrate some examples with cross-painting to a mobile robot. While we do not have the hardware to perform physical experiments, the cross-painted images look somewhat realistic (see Figure <ref>), so it may be possible that the cross-painted Franka dataset could jumpstart the training for Spot. There are additional challenges associated with mobile manipulation, such as coordination between base and arm movements and less accurate arm control, which we will leave as future work. 






