README.md

Prepare Datasets for MaskFormer

A dataset can be used by accessing DatasetCatalog for its data, or MetadataCatalog for its metadata (class names, etc). This document explains how to setup the builtin datasets so they can be used by the above APIs. Use Custom Datasets gives a deeper dive on how to use DatasetCatalog and MetadataCatalog, and how to add new datasets to them.

MaskFormer has builtin support for a few datasets. The datasets are assumed to exist in a directory specified by the environment variable DETECTRON2_DATASETS. Under this directory, detectron2 will look for datasets in the structure described below, if needed.

$DETECTRON2_DATASETS/
  ADEChallengeData2016/
  ADE20K_2021_17_01/
  coco/
  cityscapes/
  mapillary_vistas/

You can set the location for builtin datasets by export DETECTRON2_DATASETS=/path/to/datasets. If left unset, the default is ./datasets relative to your current working directory.

The model zoo contains configs and models that use these builtin datasets.

Expected dataset structure for ADE20k Scene Parsing:

ADEChallengeData2016/
  annotations/
  annotations_detectron2/
  images/
  objectInfo150.txt

The directory annotations_detectron2 is generated by running python datasets/prepare_ade20k_sem_seg.py.

Expected dataset structure for ADE20K panoptic segmentation:

ADEChallengeData2016/
  images/
  annotations/
  objectInfo150.txt
  # download instance annotation
  annotations_instance/
  # generated by prepare_ade20k_sem_seg.py
  annotations_detectron2/
  # below are generated by prepare_ade20k_panoptic_annotations.py
  ade20k_panoptic_train.json
  ade20k_panoptic_train/
  ade20k_panoptic_val.json
  ade20k_panoptic_val/

Install panopticapi by:

pip install git+https://github.com/cocodataset/panopticapi.git

Download the instance annotation from http://sceneparsing.csail.mit.edu/:

wget http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar

Then, run python datasets/prepare_ade20k_pan_seg.py, to combine semantic and instance annotations for panoptic annotations.

Expected dataset structure for ADE20k-Full:

ADE20K_2021_17_01/
  images/
  images_detectron2/
  annotations_detectron2/
  index_ade20k.pkl
  objects.txt

The directories images_detectron2 and annotations_detectron2 are generated by running python datasets/prepare_ade20k_full_sem_seg.py.

Expected dataset structure for cityscapes:

cityscapes/
  gtFine/
    train/
      aachen/
        color.png, instanceIds.png, labelIds.png, polygons.json,
        labelTrainIds.png
      ...
    val/
    test/
    # below are generated Cityscapes panoptic annotation
    cityscapes_panoptic_train.json
    cityscapes_panoptic_train/
    cityscapes_panoptic_val.json
    cityscapes_panoptic_val/
    cityscapes_panoptic_test.json
    cityscapes_panoptic_test/
  leftImg8bit/
    train/
    val/
    test/

Install cityscapes scripts by:

pip install git+https://github.com/mcordts/cityscapesScripts.git

Note: to create labelTrainIds.png, first prepare the above structure, then run cityscapesescript with:

CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createTrainIdLabelImgs.py

These files are not needed for instance segmentation.

Note: to generate Cityscapes panoptic dataset, run cityscapesescript with:

CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createPanopticImgs.py

These files are not needed for semantic and instance segmentation.

Expected dataset structure for COCO-Stuff-10K:

coco/
  coco_stuff_10k/
    annotations/
      COCO_train2014_000000000077.mat
      ...
    imageLists/
      all.txt
      test.txt
      train.txt
    images/
      COCO_train2014_000000000077.jpg
      ...
    # below are generated by prepare_coco_stuff_10k_v1.0_sem_seg.py
    annotations_detectron2/
      train/
      test/
    images_detectron2/
      train/
      test/

Get the COCO-Stuff-10k v1.0 annotation from https://github.com/nightrome/cocostuff10k.

wget http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.0.zip

Unzip cocostuff-10k-v1.0.zip and put annotations, imageLists and images to the correct location listed above.

Generate COCO-Stuff-10k annotation by python datasets/prepare_coco_stuff_10k_v1.0_sem_seg.py

Expected dataset structure for Mapillary Vistas:

mapillary_vistas/
  training/
    images/
    instances/
    labels/
    panoptic/
  validation/
    images/
    instances/
    labels/
    panoptic/

No preprocessing is needed for Mapillary Vistas.