Adapose

Install

Install dependencies using pip install -r requirements.txt. The instalation is tested with setuptools==65.6.3. Some later versions may not be compatible.

Potential Dependency Inssues: Users have reported Module not Found error with mplib. This project is developed when the newest version of mplib is mplib==0.0.8. Newer versions if mplib may not be compatible with this project. We recommend using mplib<=0.1.0

Download Dataset and Checkpoints

We provided scripts to fully automate dataset and checkpoint downloading from google drive. Just run sh install.sh to do the stuff and wait. The total file size to download is around 1GB.

You can also manually download them following the steps:

First, the dataset can be downloaded from google drive. Download it and decompress it into downloads/dataset

Second, download checkpoints for Pose Estimator from google drive, and move all four checkpoints into downloads/pose_estimator/<name>.pth.

Third, download checkpoints for Global Scheduling Policy from google drive, and save all checkpoints into downloads/global_sheduling_policy/<name>_0.pt. Notice the _0 is necessary since our code examines this number. Use controller.load=CheckpointPath option to load checkpoints.

Run Experiments

To run experiments, use command line options: python train.py dataset=?? task=?? pose_estimator=?? manipulation=?? controller=?? train=??

Possible options include:

• dataset:
  • cabinet_train
  • cabinet_test
  • drawer_train
  • drawer_test
  • mug_train
  • mug_test
  • pot_train
  • pot_test
  • real_world
• task:
  • open_cabinet
  • open_cabinet_45
  • open_cabinet_no_dr
  • open_drawer
  • open_drawer_30
  • open_drawer_no_dr
  • open_pot
  • pick_mig
  • real_world
• pose_extimator:
  • adapose_cabinet
  • adapose_drawer
  • adapose_mug
  • adapose_pot
  • ground_truth
• manipulation:
  • open_cabinet
  • open_cabinet_open_loop
  • open_drawer
  • open_drawer_open_looop
  • open_pot
  • open_pot_open_loop
  • pick_mug
  • pick_mug_open_loop
• controller
  • gt_pose
  • heuristic_pose
  • rl
• train
  • controller
  • test
  • collect: Used to collect offline data for some baselines.
  • test_baselines: Test with offline actions generated by some baselines.

Some other possible arguments are:

• task.num_envs=X: Number of parallel environments.
• headless=True/False: Whether to show graphics.
• viewerless=True/False: Whether to disable on-hand camera to accelerate some non-vision tasks.
• exp_name=XXX: The name of the experiment (used for saving models and logging).
• controller.load=Checkpoint: Load the saved controller checkpoint.

All other configurations can be modified using the hydra style command line options.

Miscs

The scripts for drawing the plots included in our paper is also provided in miscs/plot.

Citations

To cite us at

@article{an2023rgbmanip,
  title={RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation},
  author={An, Boshi and Geng, Yiran and Chen, Kai and Li, Xiaoqi and Dou, Qi and Dong, Hao},
  journal={arXiv preprint arXiv:2310.03478},
  year={2023}
}