Align 3D model to point cloud (Point Cloud Registration)

[anhtuduong]

Point Cloud Registration is a fundamental problem in 3D computer vision and photogrammetry. Given several sets of points in different coordinate systems, the aim of registration is to find the transformation that best aligns all of them into a common coordinate system.

Point Cloud Registration plays a significant role in this localization phase. Having point cloud rendered from the 3D model mesh (source point cloud) and the point cloud of an object taken from ZED camera (target point cloud), we need to perform some calculation to align the source point cloud to the target point cloud. The transformation matrix calculated after alignment is the coordinate of the object in the scene.

The transformation matrix is in format of 6DoF (6 degrees of freedom). It refers to the movement of a rigid body in 3-dimensional space. 6DoF considers the user’s head position, head movement, and overall orientation.

6DoF comprises two movement parameters – Translation and Rotation.

1. Translation involves moving along the X, Y and Z axis

• X (sway): Moving left and right on the x-axis
• Y (heave): Moving up and downward on the y-axis
• Z (surge): Move forward and backward on the z-axis

2. Rotation involves turning to face a different axis

• Pitch: Moving between the X and Y-axis.
• Yaw: Moving between the X and Z-axis.
• Roll: Moving between Z and Y-axis.

[anhtuduong]

ICP stands for Iterative Closest Point, which is a widely used algorithm for registering two 3D point clouds by minimizing the distance between corresponding points.

The general idea of ICP is to iteratively estimate the transformation that aligns the source point cloud to the target point cloud. The steps of ICP can be summarized as follows:

Initialization: Define an initial transformation matrix that represents the rough alignment between the two point clouds. This initial transformation can be computed using some other method, or simply set to the identity matrix if no prior information is available.

Correspondence: Find the corresponding points between the source point cloud and the target point cloud. One common method is to find the nearest neighbors in the target point cloud for each point in the source point cloud.

Compute transformation: Compute the transformation that best aligns the corresponding points. One common method is to use Singular Value Decomposition (SVD) to compute the rotation and translation that minimize the sum of squared distances between corresponding points.

Update transformation: Update the transformation matrix with the computed rotation and translation.

Check convergence: Check if the convergence criterion is met. This can be done by comparing the change in the transformation matrix between iterations with a predefined threshold. If the convergence criterion is not met, go back to step 2 and repeat until convergence is reached.

Output: The final transformation matrix gives the registration result. Apply this transformation to the source point cloud to align it with the target point cloud.

In summary, ICP iteratively estimates the transformation matrix that aligns two point clouds by minimizing the distance between corresponding points. The algorithm is simple to implement and can be used for a wide range of applications such as robot localization, object recognition, and surface reconstruction.

[anhtuduong]

Point Pair Features (PPF) is a method for matching 3D models to 3D point clouds. The idea behind PPF is to define a feature descriptor that captures the relative position and orientation of a pair of points in 3D space. By computing and matching these descriptors, PPF can be used to estimate the pose of a 3D model relative to a 3D point cloud.

The PPF descriptor consists of four components: the normal direction of the first point, the relative position vector between the two points, the angle between the normal directions of the two points, and the angle between the relative position vector and the normal direction of the first point. By computing this descriptor for each pair of points in the model and comparing it to the descriptors of the points in the point cloud, PPF can identify potential matches between the model and the point cloud.

Once potential matches are identified, PPF can be used to estimate the pose of the model relative to the point cloud. This involves finding a set of matching pairs of points, and then computing the translation and rotation that aligns the pairs of points in the model with the pairs of points in the point cloud. This translation and rotation can then be used to transform the model into the coordinate system of the point cloud.

PPF has been shown to be an effective method for matching 3D models to 3D point clouds, with high accuracy and efficiency. It is often used in robotics and computer vision applications for tasks such as object recognition, pose estimation, and tracking.

[anhtuduong]

FPFH (Fast Point Feature Histograms) is a feature descriptor commonly used in point cloud analysis and 3D computer vision tasks. It is an extension of the Point Feature Histograms (PFH) descriptor, designed to capture the geometric properties of 3D points.

The FPFH descriptor calculates the local surface characteristics of a point in a point cloud by considering its neighbors. Here's a high-level overview of how FPFH features are computed:

1. Select a point of interest (the query point) in the point cloud.
2. Identify the k-nearest neighbors of the query point.
3. For each neighbor, calculate the relative position with respect to the query point, forming a pairwise point-pair feature.
4. Compute the difference in normal directions between the query point and each neighbor.
5. Calculate a weighted histogram of the pairwise point-pair features, where the weights are based on the differences in normal directions.
6. Concatenate the histograms from all the neighbors to obtain the final FPFH feature for the query point.

The FPFH descriptor takes into account not only the geometric positions of the neighboring points but also the differences in their surface normals. This combination allows for a more robust representation of local surface properties and provides a measure of distinctiveness between different points.

FPFH features are widely used in point cloud registration, object recognition, and point cloud segmentation tasks. They facilitate the matching and comparison of points in 3D space, enabling algorithms to reason about the geometric properties of the underlying objects or scenes.

[anhtuduong]

PPF (Point Pair Features) and FPFH (Fast Point Feature Histograms) are both feature descriptors used in point cloud analysis. Here are the main differences between the two:

1. Computation Method:

• PPF: PPF calculates features based on the pairwise relationships between points in a point cloud. It considers the positions and surface normals of two points, forming a feature vector that represents their relative geometry.
• FPFH: FPFH extends the idea of PPF by considering a set of neighboring points around a query point. It computes a weighted histogram of pairwise point-pair features using the relative positions and surface normals of the query point and its neighbors.

2. Descriptor Type:

• PPF: PPF descriptors are point-to-point descriptors, meaning they capture the geometric relationship between two points in the point cloud.
• FPFH: FPFH descriptors are point-to-point descriptors as well, but they aggregate information from multiple neighbors around a query point to generate a more comprehensive representation of its local surface characteristics.

3. Robustness and Efficiency:

• PPF: PPF descriptors are known for their robustness in handling partial occlusions and noise in point clouds. However, they can be computationally expensive, especially when dealing with large point clouds.
• FPFH: FPFH descriptors are designed to be computationally efficient. They achieve a good balance between computation time and descriptiveness, making them suitable for real-time applications. However, they may be less robust than PPF descriptors in handling occlusions and noise.

4. Applications:

• PPF: PPF features are commonly used in point cloud registration tasks, where the goal is to align two or more point clouds into a common coordinate system.
• FPFH: FPFH features are often employed in tasks such as point cloud segmentation, object recognition, and scene understanding. They provide a more holistic representation of local surface properties, enabling better discrimination between different points in a point cloud.

In summary, while both PPF and FPFH are feature descriptors for point clouds, they differ in their computation methods, descriptor types, robustness, efficiency, and applications. PPF focuses on pairwise relationships between points, while FPFH aggregates information from neighboring points to capture local surface characteristics. The choice between the two depends on the specific requirements and constraints of the application at hand.

[anhtuduong]

1st version using FPFH and ICP are made in align_pointclouds.py.

The alignment result is not yet satisfactory. To be improved:

1. Refine the parameters: Adjust the parameters used in the feature matching and registration steps. For example, try changing the radius_feature, distance_threshold, ransac_n, or other parameters to see if it improves the alignment.

2. Increase the number of iterations: In the RANSAC-based registration, increase the number of iterations by modifying the criteria argument in registration_ransac_based_on_feature_matching. Increasing the number of iterations allows for more robust outlier rejection and may lead to a better alignment.

3. Try different feature descriptors: Instead of using FPFH, explore other feature descriptors like SHOT or 3DShapeNet to compute feature descriptors for the point clouds. Different feature descriptors may capture different aspects of the point clouds and yield better alignment results.

4. Apply preprocessing techniques: Preprocessing steps such as downsampling, outlier removal, or surface smoothing can help improve the quality of the point clouds and facilitate better alignment. Experiment with different preprocessing techniques to see if they have a positive impact on the alignment.

5. Consider global registration methods: If the initial alignment using feature matching and local registration methods is not sufficient, explore global registration methods such as the Go-ICP algorithm or the ICP variants like Point-to-Plane ICP. These methods can provide more global context and handle larger misalignments.

6. Iterative refinement: Perform an iterative refinement process by alternating between feature matching and local registration steps. After each registration step, update the transformation matrix and repeat the process to gradually improve the alignment.

[anhtuduong]

2nd version updating FPFH parameters and iterative refinment ICP shows good result in PointCloudRegistration.py

Except for this block, which seems to misaligned: