Efficient Semantic 3D Mapping for Indoor Environments

This project, conducted as part of my bachelor's thesis and later published at ICRA, involved implementing state-of-the-art semantic 3D maps using Normal Distribution Transform (NDT). We projected class labels, predicted by a semantic segmentation model, onto individual 3D points to create class histograms. Alongside the semantic 3D NDT maps, we also implemented 3D semantic voxel maps for comparison. We evaluated the accuracy of these semantic mapping approaches by projecting the 3D map back to a 2D image and comparing it with the ground truth segmentation.

A key proficiency an autonomous mobile robot must have to perform high-level tasks is a strong understanding of its environment. This involves information about what types of objects are present, where they are, what their spatial extend is, and how they can be reached, i.e., information about free space is also crucial. Semantic maps are a powerful instrument providing such information. However, applying semantic segmentation and building 3D maps with high spatial resolution is challenging given limited resources on mobile robots. In this paper, we incorporate semantic information into efficient occupancy normal distribution transform (NDT) maps to enable real-time semantic mapping on mobile robots. On the publicly available dataset Hypersim, we show that, due to their sub-voxel accuracy, semantic NDT maps are superior to other approaches. We compare them to the recent state-of-the-art approach based on voxels and semantic Bayesian spatial kernel inference (S-BKI) and to an optimized version of it derived in this paper. The proposed semantic NDT maps can represent semantics to the same level of detail, while mapping is 2.7 to 17.5 times faster. For the same grid resolution, they perform significantly better, while mapping is up to more than 5 times faster. Finally, we prove the real-world applicability of semantic NDT maps with qualitative results in a domestic application.