Bimanual Predictor

Conda Env

conda create -n bimanual-pred python=3.7
conda install pytorch==1.6.0 cudatoolkit=10.1 -c pytorch
conda install matplotlib
conda install conda-forge::opencv
conda install open3d-admin::open3d
pip install --upgrade jupyter_client
conda install anaconda::scikit-learn
pip install tqdm
pip install pyyaml
pip install addict
pip install pandas
pip install plyfile

Dataset

Create data/bimanual and save point clouds within their object directories. An example for tissue object is provided at data/bimanual/tissue/0_axis.csv and data/bimanual/tissue/0.csv

• 0.csv - Nx7 - (x, y, z, nx, ny, nz, seg)
• 0_axis.csv - 2x3 - s_hat, q

Training

Contact prediction:

python train_bimanual_contact.py --obj tissue --model pointnet_part_seg --normal --log_dir pointnet_part_seg --gpu 0 --epoch 1001

Axis prediction:

python train_bimanual_axis.py --obj tissue --model pointnet_reg --normal --log_dir pointnet_reg --gpu 0 --epoch 1001

Set --use_q flag to additionally predict q values

Evaluation

Validation set:

Rotations and translations of 0.csv Metrics and visualizations are saved in eval/metrics.txt and eval/viz

python test_bimanual_contact.py --obj tissue --log_dir pointnet_part_seg --normal --split val
python test_bimanual_axis.py --obj tissue --log_dir pointnet_reg --normal --split val

Test point cloud:

Validates single sample test.csv with no GT. PCL is of the form (x, y, z, nx, ny, nz) Visualizations are saved in eval/test.png

python test_bimanual_contact.py --obj tissue --log_dir pointnet_part_seg --normal --split test
python test_bimanual_axis.py --obj tissue --log_dir pointnet_reg --normal --split test

Pytorch Implementation of PointNet and PointNet++

This repo is implementation for PointNet and PointNet++ in pytorch.

Update

2021/03/27:

(1) Release pre-trained models for semantic segmentation, where PointNet++ can achieve 53.5% mIoU.

(2) Release pre-trained models for classification and part segmentation in log/.

2021/03/20: Update codes for classification, including:

(1) Add codes for training ModelNet10 dataset. Using setting of --num_category 10.

(2) Add codes for running on CPU only. Using setting of --use_cpu.

(3) Add codes for offline data preprocessing to accelerate training. Using setting of --process_data.

(4) Add codes for training with uniform sampling. Using setting of --use_uniform_sample.

2019/11/26:

(1) Fixed some errors in previous codes and added data augmentation tricks. Now classification by only 1024 points can achieve 92.8%!

(2) Added testing codes, including classification and segmentation, and semantic segmentation with visualization.

(3) Organized all models into ./models files for easy using.

Install

The latest codes are tested on Ubuntu 16.04, CUDA10.1, PyTorch 1.6 and Python 3.7:

conda install pytorch==1.6.0 cudatoolkit=10.1 -c pytorch

Classification (ModelNet10/40)

Data Preparation

Download alignment ModelNet here and save in data/modelnet40_normal_resampled/.

Run

You can run different modes with following codes.

• If you want to use offline processing of data, you can use --process_data in the first run. You can download pre-processd data here and save it in data/modelnet40_normal_resampled/.
• If you want to train on ModelNet10, you can use --num_category 10.

# ModelNet40
## Select different models in ./models 

## e.g., pointnet2_ssg without normal features
python train_classification.py --model pointnet2_cls_ssg --log_dir pointnet2_cls_ssg
python test_classification.py --log_dir pointnet2_cls_ssg

## e.g., pointnet2_ssg with normal features
python train_classification.py --model pointnet2_cls_ssg --use_normals --log_dir pointnet2_cls_ssg_normal
python test_classification.py --use_normals --log_dir pointnet2_cls_ssg_normal

## e.g., pointnet2_ssg with uniform sampling
python train_classification.py --model pointnet2_cls_ssg --use_uniform_sample --log_dir pointnet2_cls_ssg_fps
python test_classification.py --use_uniform_sample --log_dir pointnet2_cls_ssg_fps

# ModelNet10
## Similar setting like ModelNet40, just using --num_category 10

## e.g., pointnet2_ssg without normal features
python train_classification.py --model pointnet2_cls_ssg --log_dir pointnet2_cls_ssg --num_category 10
python test_classification.py --log_dir pointnet2_cls_ssg --num_category 10

Performance

Model	Accuracy
PointNet (Official)	89.2
PointNet2 (Official)	91.9
PointNet (Pytorch without normal)	90.6
PointNet (Pytorch with normal)	91.4
PointNet2_SSG (Pytorch without normal)	92.2
PointNet2_SSG (Pytorch with normal)	92.4
PointNet2_MSG (Pytorch with normal)	92.8

Part Segmentation (ShapeNet)

Data Preparation

Download alignment ShapeNet here and save in data/shapenetcore_partanno_segmentation_benchmark_v0_normal/.

Run

## Check model in ./models 
## e.g., pointnet2_msg
python train_partseg.py --model pointnet2_part_seg_msg --normal --log_dir pointnet2_part_seg_msg
python test_partseg.py --normal --log_dir pointnet2_part_seg_msg

Performance

Model	Inctance avg IoU	Class avg IoU
PointNet (Official)	83.7	80.4
PointNet2 (Official)	85.1	81.9
PointNet (Pytorch)	84.3	81.1
PointNet2_SSG (Pytorch)	84.9	81.8
PointNet2_MSG (Pytorch)	85.4	82.5

Semantic Segmentation (S3DIS)

Data Preparation

Download 3D indoor parsing dataset (S3DIS) here and save in data/s3dis/Stanford3dDataset_v1.2_Aligned_Version/.

cd data_utils
python collect_indoor3d_data.py

Processed data will save in data/stanford_indoor3d/.

Run

## Check model in ./models 
## e.g., pointnet2_ssg
python train_semseg.py --model pointnet2_sem_seg --test_area 5 --log_dir pointnet2_sem_seg
python test_semseg.py --log_dir pointnet2_sem_seg --test_area 5 --visual

Visualization results will save in log/sem_seg/pointnet2_sem_seg/visual/ and you can visualize these .obj file by MeshLab.

Performance

Model	Overall Acc	Class avg IoU	Checkpoint
PointNet (Pytorch)	78.9	43.7	40.7MB
PointNet2_ssg (Pytorch)	83.0	53.5	11.2MB

Visualization

Using show3d_balls.py

## build C++ code for visualization
cd visualizer
bash build.sh 
## run one example 
python show3d_balls.py

Using MeshLab

Reference By

halimacc/pointnet3
fxia22/pointnet.pytorch
charlesq34/PointNet
charlesq34/PointNet++

Citation

If you find this repo useful in your research, please consider citing it and our other works:

@article{Pytorch_Pointnet_Pointnet2,
      Author = {Xu Yan},
      Title = {Pointnet/Pointnet++ Pytorch},
      Journal = {https://github.com/yanx27/Pointnet_Pointnet2_pytorch},
      Year = {2019}
}

@InProceedings{yan2020pointasnl,
  title={PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling},
  author={Yan, Xu and Zheng, Chaoda and Li, Zhen and Wang, Sheng and Cui, Shuguang},
  journal={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2020}
}

@InProceedings{yan2021sparse,
  title={Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion},
  author={Yan, Xu and Gao, Jiantao and Li, Jie and Zhang, Ruimao, and Li, Zhen and Huang, Rui and Cui, Shuguang},
  journal={AAAI Conference on Artificial Intelligence ({AAAI})},
  year={2021}
}

@InProceedings{yan20222dpass,
      title={2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds}, 
      author={Xu Yan and Jiantao Gao and Chaoda Zheng and Chao Zheng and Ruimao Zhang and Shuguang Cui and Zhen Li},
      year={2022},
      journal={ECCV}
}

Selected Projects using This Codebase

• PointConv: Deep Convolutional Networks on 3D Point Clouds, CVPR'19
• On Isometry Robustness of Deep 3D Point Cloud Models under Adversarial Attacks, CVPR'20
• Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions, ECCV'20
• PCT: Point Cloud Transformer
• PSNet: Fast Data Structuring for Hierarchical Deep Learning on Point Cloud
• Stratified Transformer for 3D Point Cloud Segmentation, CVPR'22