NOTE: This section is a not part of the official document. If your looking for orignal document, go down to find the official contents or go to the orignal repository.
This repository is NVlabs/contact_graspnet with ROS implementation for testing purpose. The code has been tested with following system.
Ubuntu 18.04
Python 3.6.9
ROS-melodic
CUDA 10.2
tensorflow 2.3.1
If you don't use the conda, follow this instuction. Or you can follow original intstuction guide bellow(not tested yet).
1.1.1 We highly recomand you to use virtual environment for python.
-
Genterate virtual environment on package root dir.
virtualenv -p python3 --system-site-packages venv
-
Activate
virtualenv
source venv/bin/activate
1.1.2 Install required python packages on your virtualenv
.
-
Tested
tensorflow==2.3.1 mayavi==4.7.3 PyQt5==5.15.4 trimesh==3.9.29 pyrender==0.1.45
-
Not tested
scipy==1.4.1
- Recompile pointnet2 tf_ops:
sh compile_pointnet_tfops.sh
Find instuction on the official documents bellow.
3.1 grasp_planner (ContactGraspNetPlanner)
- color_image (sensor_msgs/Image)
- Color image. It is not used for generating grasps.
- depth_image (sensor_msgs/Image)
- Detph image
- camera_info (sensor_msgs/Camerainfo)
- Depth camera intrinsic for deprojecting depth image to point clouds.
- segmask (sensor_msgs/Image)
- Object instance segmentation for grasp filltering.
- grasps (ContactGrasp[])
- List of generated grasps.
- ckpt_dir (str)
- Contact-GraspNet checkpoint directory.
- Default: checkpoints/scene_test_2048_bs3_hor_sigma_001
- z_min (double)
- Z min value threshold to crop the input point cloud.
- Default: 0.2
- z_max (double)
- Z max value threshold to crop the input point cloud.
- Default: 1.8
- local_regions (bool)
- Crop 3D local regions around given segments.
- Default: False
- filter_grasps (bool)
- Filter grasp contacts according to segmap.
- Default: False
- skip_border_objects (bool)
- When extracting local_regions, ignore segments at depth map boundary.
- Default: False
- forward_passes (int)
- Run multiple parallel forward passes to mesh_utils more potential contact points.
- Default: 1
- segmap_id (int)
- Only return grasps of the given object id
- Default: 0
roslaunch contact_graspnet_planner grasp_planner.launch
python contact_graspnet/ros_client_exmaple.py --ckpt_dir checkpoints/scene_test_2048_bs3_hor_sigma_001 --np_path test_data/7.npy
ORIGINAL README DOCUMENT
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox
ICRA 2021
This code has been tested with python 3.7, tensorflow 2.2, CUDA 11.1
Create the conda env
conda env create -f contact_graspnet_env.yml
- Recompile pointnet2 tf_ops:
sh compile_pointnet_tfops.sh
Training: 1x Nvidia GPU >= 24GB VRAM, >=64GB RAM
Inference: 1x Nvidia GPU >= 8GB VRAM (might work with less)
Download trained models from here and copy them into the checkpoints/
folder.
Download the test data from here and copy them them into the test_data/
folder.
Contact-GraspNet can directly predict a 6-DoF grasp distribution from a raw scene point cloud. However, to obtain object-wise grasps, remove background grasps and to achieve denser proposals it is highly recommended to use (unknown) object segmentation [e.g. 1, 2] as preprocessing and then use the resulting segmentation map to crop local regions and filter grasp contacts.
Given a .npy/.npz file with a depth map (in meters), camera matrix K and (optionally) a 2D segmentation map, execute:
python contact_graspnet/inference.py \
--np_path=test_data/*.npy \
--local_regions --filter_grasps
Given a .npy/.npz file with just a 3D point cloud (in meters), execute for example:
python contact_graspnet/inference.py --np_path=/path/to/your/pc.npy \
--forward_passes=5 \
--z_range=[0.2,1.1]
--np_path
: input .npz/.npy file(s) with 'depth', 'K' and optionally 'segmap', 'rgb' keys. For processing a Nx3 point cloud instead use 'xzy' and optionally 'xyz_color' as keys.
--ckpt_dir
: relative path to checkpooint directory. By default checkpoint/scene_test_2048_bs3_hor_sigma_001
is used. For very clean / noisy depth data consider scene_2048_bs3_rad2_32
/ scene_test_2048_bs3_hor_sigma_0025
trained with no / strong noise.
--local_regions
: Crop 3D local regions around object segments for inference. (only works with segmap)
--filter_grasps
: Filter grasp contacts such that they only lie on the surface of object segments. (only works with segmap)
--skip_border_objects
Ignore segments touching the depth map boundary.
--forward_passes
number of (batched) forward passes. Increase to sample more potential grasp contacts.
--z_range
[min, max] z values in meter used to crop the input point cloud, e.g. to avoid grasps in the foreground/background(as above).
--arg_configs TEST.second_thres:0.19 TEST.first_thres:0.23
Overwrite config confidence thresholds for successful grasp contacts to get more/less grasp proposals
Download the Acronym dataset, ShapeNet meshes and make them watertight, following these steps.
Download the training data consisting of 10000 table top training scenes with contact grasp information from here and extract it to the same folder:
acronym
├── grasps
├── meshes
├── scene_contacts
└── splits
When training on a headless server set the environment variable
export PYOPENGL_PLATFORM='egl'
Start training with config contact_graspnet/config.yaml
python contact_graspnet/train.py --ckpt_dir checkpoints/your_model_name \
--data_path /path/to/acronym/data
The scene_contacts
downloaded above are generated from the Acronym dataset. To generate/visualize table-top scenes yourself, also pip install the acronym_tools package in your conda environment as described in the acronym repository.
In the first step, object-wise 6-DoF grasps are mapped to their contact points saved in mesh_contacts
python tools/create_contact_infos.py /path/to/acronym
From the generated mesh_contacts
you can create table-top scenes which are saved in scene_contacts
with
python tools/create_table_top_scenes.py /path/to/acronym
Takes ~3 days in a single thread. Run the command several times to process on multiple cores in parallel.
You can also visualize existing table-top scenes and grasps
python tools/create_table_top_scenes.py /path/to/acronym \
--load_existing scene_contacts/000000.npz -vis
@article{sundermeyer2021contact,
title={Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes},
author={Sundermeyer, Martin and Mousavian, Arsalan and Triebel, Rudolph and Fox, Dieter},
booktitle={2021 IEEE International Conference on Robotics and Automation (ICRA)},
year={2021}
}