Home

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds

Created by Francis Engelmann, Theodora Kontogianni, Alexander Hermans, Jonas Schult and Bastian Leibe from RWTH Aachen University.

Introduction

This work is based on our Paper Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds, which is appeared at the IEEE International Conference on Computer Vision (ICCV) 2017, 3DRMS Workshop.

You can also check our project page for further details.

Deep learning approaches have made tremendous progress in the field of semantic segmentation over the past few years. However, most current approaches operate in the 2D image space. Direct semantic segmentation of unstructured 3D point clouds is still an open research problem. The recently proposed PointNet architecture presents an interesting step ahead in that it can operate on unstructured point clouds, achieving decent segmentation results. However, it subdivides the input points into a grid of blocks and processes each such block individually. In this paper, we investigate the question how such an architecture can be extended to incorporate larger-scale spatial context. We build upon PointNet and propose two extensions that enlarge the receptive field over the 3D scene. We evaluate the proposed strategies on challenging indoor and outdoor datasets and show improved results in both scenarios.

In this repository, we release code for training and testing various pointcloud semantic segmentation networks on arbitrary datasets.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{3dsemseg_ICCVW17,
  author    = {Francis Engelmann and
               Theodora Kontogianni and
               Alexander Hermans and
               Bastian Leibe},
  title     = {Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds},
  booktitle = {{IEEE} International Conference on Computer Vision, 3DRMS Workshop, {ICCV}},
  year      = {2017}
}

Installation

Install TensorFlow. The code has been tested with Python 3.6 and TensorFlow 1.8.

Usage

Our framework is mainly influenced by Martin Thoma's blog entry about structuring machine learning projects.

In order to get more representative blocks, it is encouraged to uniformly downsample the original point clouds. This is done via the following script:

python tools/downsample.py --data_dir path/to/dataset --cell_size 0.01

This statement will produce pointclouds whereeach point will be representative for its 10cm x 10cm x 10cm neighborhood

To train/test a model for semantic segmentation on pointclouds, you need to run:

python run.py --config path/to/config/file.yaml

Detailed instruction of the structure for the yaml config file can be found in the wiki. Additionally, some example configuration file are given in the folder experiments.

Note that the final evaluation is done on the full sized point clouds using k nearest neighbor interpolation.