Table of Contents
All authors are with Learning and Adaptive Robotics Laboratory, Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, TX 76013, USA.
Reconstructing three-dimensional (3D) scenes with semantic understanding is vital in many robotic applications. Robots need to identify which objects, along with their positions and shapes, to manipulate them precisely with given tasks. Mobile robots, especially, usually use lightweight networks to segment objects on RGB images and then localize them via depth maps; however, they often encounter out-of-distribution scenarios where masks over-cover the objects. In this paper, we address the problem of panoptic segmentation quality in 3D scene reconstruction by refining segmentation errors using non-parametric statistical methods. To enhance mask precision, we map the predicted masks into a depth frame to estimate their distribution via kernel densities. The outliers in depth perception are then rejected without the need for additional parameters in an adaptive manner to out-of-distribution scenarios, followed by 3D reconstruction using projective signed distance functions (SDFs). We validate our method on a synthetic dataset, which shows improvements in both quantitative and qualitative results for panoptic mapping. Through real-world testing, the results furthermore show our method's capability to be deployed on a real-robot system. The demo is available at YouTube.
- KDEpy (version 1.1.5)
- panoptic-mapping-ros (ROS package)
- pyrealsense2 (version 2.51.1.4348)
- Open3D (version 0.16.0)
A bit of modification for KDEpy library can be found at KDEpy_mod. Please replace the bw_selection.py file with the original one.
Indoor mobile robots operating in an environment with multiple objects refines RGB-based segmentation masks using kernel density estimation via depth perception and rebuilds panoptic map with object instances using projective signed distance functions.
Refined Panoptic Mapping when Robot Moving in an Indoor Envinronment.
Depth maps of object instances containing depth outliers (top row) due to the imperfection of segmentation models and their density estimations along depth perception (middle row), and refined depth maps (bottom row). The shaded depth values on the density lines in between vertical red cutoff lines are considered inliers; otherwise, rejects as they appear to be outliers.
Segmentation Mask Refinement Process.
To capture a scene with segmentation (point clouds of instances), run semantic_scene_perception/capture_segment_scene.py file:
python semantic_scene_perception/capture_segment_scene.py
The resultant point cloud (and their respective colored/depth images) is saved in semantic_scene_perception/cap_data/exp[NO]. A notification line is notified in the terminal.
To produce the removal process, run refined_panoptic_mapping/remove_depth_outliers.py file:
python refined_panoptic_mapping/remove_depth_outliers.py
The plots illustrating cutoffs is displayed followed by side-by-side view of noised and cleaner point clouds.
Outlier Removal on Point Clouds of Instances.
Qualitative results on the flat dataset of (a) the original panoptic mapping approach, (b) the original approach coupled with mask refinement, (c) our approach without mask refinement, and (d) our approach with mask refinement. The room texture and its panoptic segmentation ground truth are retrieved based on RGB images and annotation masks provided by the original approach's framework. Note that the robot frame indicating its pose is simplified and represented as the RGB mesh frame in each reconstructed map.
Qualitative Results on Flat Dataset.
The flat dataset can be downloaded from the original ASL Datasets.
Refer to the panoptic_mapping/flat_dataset/run1/download.txt file to download the dataset. Remember to change the path to your desired destination.
export FLAT_DATA_DIR="your/path/to/flat/dataset"
Once downloaded, copy run1 and rename to run1_mod_2, run1_mod_4, run1_mod_5, replace the predicted .png images with .png images from the respective folders, and proceed with building the maps.
To build (a) original panoptic map, use contents of the following files:
mapper/flat_detectron_org.yamllaunch/run_org.launch
To build (b) original panoptic with refinement map, use contents of the following files:
mapper/flat_detectron_org_with_refinement.yamllaunch/run_org_with_refinement.launch
To build (c) our approach without refinement map, use contents of the following files:
mapper/flat_detectron_without_refinement.yamllaunch/run_without_refinement.launch
To build (d) our approach with refinement map, use contents of the following files:
mapper/flat_detectron_with_refinement.yamllaunch/run_with_refinement.launch
To build (e) room texture map, use contents of the following files:
mapper/flat_detectron_room_texture.yamllaunch/run_room_texture.launch
To build (f) panoptic ground truth map, use contents of the following files:
mapper/flat_detectron_panoptic_ground_truth.yamllaunch/run_panoptic_ground_truth.launch- also change
flat_data_player.py (line 105)topred_file = file_id + "_segmentation.pngto allow ROS reads and publishes the ground truth segmentation.
We mount the Intel RealSense D435i RGB-D camera on the display of the Baxter robot. The Baxter robot moves in an indoor environment to observe the objects in the scene while performing volumetric rendering and perception outlier rejection.
Deployment on Baxter Robot and Reconstructed Panoptic Mapping Result.
To run the demo with an Intel RealSense D435i RGB-D camera, run refined_panoptic_mapping/demo.py file:
python refined_panoptic_mapping/demo.py
NOTE: If you find this repositpory useful, please also give the GitHub repository of panoptic-mapping kudos when you use this code for your work!