Skip to content

Efficient Learning on Point Clouds with Basis Point Sets

License

Notifications You must be signed in to change notification settings

sergeyprokudin/bps

Repository files navigation

Efficient Learning on Point Clouds with Basis Point Sets

Update: pure PyTorch implementation of the BPS encoding is now available, thanks to Omid Taheri.

Basis Point Set (BPS) is a simple and efficient method for encoding 3D point clouds into fixed-length representations.

It is based on a simple idea: select k fixed points in space and compute vectors from these basis points to the nearest points in a point cloud; use these vectors (or simply their norms) as features:

Teaser Image

The basis points are kept fixed for all the point clouds in the dataset, providing a fixed representation of every point cloud as a vector. This representation can then be used as input to arbitrary machine learning methods, in particular it can be used as input to off-the-shelf neural networks.

Below is the example of a simple model using BPS features as input for the task of mesh registration over a noisy scan:

Teaser Image

FAQ: what are the key differences between standard occupancy voxels, TSDF and the proposed BPS representation?

  • continuous global vectors instead of simple binary flags or local distances in the cells;
  • smaller number of cells required to represent shape accurately;
  • BPS cell arrangement could be different from a standard rectangular grid, allowing different types of convolutions;
  • significant improvement in performance: simply substituting occupancy voxels with BPS directional vectors results in a +9% accuracy improvement of a VoxNet-like 3D-convolutional network on a ModelNet40 classification challenge.

Check our ICCV 2019 paper for more details.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{prokudin2019efficient,
  title={Efficient Learning on Point Clouds With Basis Point Sets},
  author={Prokudin, Sergey and Lassner, Christoph and Romero, Javier},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={4332--4341},
  year={2019}
}

Usage

Requirements

  • Python >= 3.7;
  • scikit-learn >= 0.21;
  • tqdm >= 4.3;
  • PyTorch >= 1.3 (for running provided demos)
  • trimesh, pillow, pyntcloud (for running human mesh registration code)
  • psbody-mesh (to compute FAUST registrations correspondences)

Installation

pip3 install git+https://github.com/sergeyprokudin/bps

Code snippet

Converting point clouds to BPS representation takes few lines of code:

import numpy as np
from bps import bps

# batch of 100 point clouds to convert
x = np.random.normal(size=[100, 2048, 3])

# optional point cloud normalization to fit a unit sphere
x_norm = bps.normalize(x)

# option 1: encode with 1024 random basis and distances as features
x_bps_random = bps.encode(x_norm, bps_arrangement='random', n_bps_points=1024, bps_cell_type='dists')

# option 2: encode with 32^3 grid basis and full vectors to nearest points as features
x_bps_grid = bps.encode(x_norm, bps_arrangement='grid', n_bps_points=32**3, bps_cell_type='deltas')
# the following tensor can be provided as input to any Conv3D network:
x_bps_grid = x_bps_grid.reshape([-1, 32, 32, 32, 3])

Demos

Clone the repository and install the dependencies:

git clone https://github.com/sergeyprokudin/bps
cd bps
python setup.py install
pip3 install torch h5py

Check one of the provided examples:

  • ModelNet40 3D shape classification with BPS-MLP (~89% accuracy, ~30 minutes of training on a non-GPU MacBook Pro, ~3 minutes of training on Nvidia V100 16gb):
python bps_demos/train_modelnet_mlp.py
  • ModelNet40 3D shape classification with BPS-Conv3D (~92% accuracy, ~120 minutes of training on Nvidia V100 16gb):
python bps_demos/train_modelnet_conv3d.py
  • FAUST body mesh registration: check tutorial notebook to run the evaluation of the pre-trained model.

You can directly download the results (predicted alignments, computed correspondences, demo video) here.

Results are also visualised in this video.

Teaser Image

  • Running pre-trained model on your own 3D body scans: download the model checkpoint (mirror):
mkdir data
cd data
wget --output-document=mesh_regressor.h5 https://www.dropbox.com/s/u3d1uighrtcprh2/mesh_regressor.h5?dl=0

Run the model, providing the paths to your own *.ply file and output directory. You can test that everything works by running the following synthetic example:

cd bps_demos
python run_alignment.py demo_scan.ply ../logs/demo_output

If a directory is provided as a first parameter, the alignment model will be ran on all *.ply files found.

Here is an example of a prediction on some noisy real scan:

Teaser

License

This library is licensed under the MIT-0 License. See the LICENSE file.

Note: this is the official fork of the Amazon repository written by the same author during the time of internship. Latest features and bug fixes are likely to appear here first.

About

Efficient Learning on Point Clouds with Basis Point Sets

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •