README.md 7.1 KB
Newer Older
Radim Tylecek's avatar
Radim Tylecek committed
1 2
# 3D Reconstruction meets Semantics 

Torsten Sattler's avatar
Torsten Sattler committed
3 4
Part of the ICCV 2017 workshop [3D Reconstruction meets Semantics](http://trimbot2020.webhosting.rug.nl/events/3drms/) is a challenge on combining 3D and semantic information in complex scenes. 
To this end, a challenging outdoor dataset, captured by a robot driving through a semantically-rich garden that contains fine geometric details, is released. 
Radim Tylecek's avatar
Radim Tylecek committed
5 6
A multi-camera rig is mounted on top of the robot, enabling the use of both stereo and motion stereo information. 
Precise ground truth for the 3D structure of the garden has been obtained with a laser scanner and accurate pose estimates for the robot are available as well. 
Torsten Sattler's avatar
Torsten Sattler committed
7 8 9 10 11 12 13 14 15 16 17 18 19
Ground truth semantic labels and ground truth depth from a laser scan will be used for benchmarking the quality of the 3D reconstructions.

## Reconstruction Challenge
Given a set of images and their known camera poses, the goal of the challenge is to create a semantically annotated 3D model of the scene. 
To this end, it will be necessary to compute depth maps for the images and then fuse them together (potentially while incorporating information from the semantics) into a single 3D model.

We provide the following data for the challenge:
* A set of training sequences consisting of
  * calibrated images with their camera poses,
  * ground truth semantic annotations for a subset of these images,
  * a semantically annotated 3D point cloud depicting the area of the training sequence.
* A testing sequence consisting of calibrated images with their camera poses.

Radim Tylecek's avatar
Radim Tylecek committed
20 21
Please notice that only a subset of all cameras (cameras 0 and 2) provide colour images while the other cameras capture the scene only in greyscale. 
Labels are only provided for the colour cameras, while the greyscale cameras are intended to be used for obtained better depth maps (cameras 0 and 1 and cameras 2 and 3 form a stereo pair).
Torsten Sattler's avatar
Torsten Sattler committed
22

Radim Tylecek's avatar
Radim Tylecek committed
23 24
## Data

Radim Tylecek's avatar
Radim Tylecek committed
25 26
_IMPORTANT_: Please install [git lfs](https://git-lfs.github.com/) before cloning this repository to retrieve PLY files.

Radim Tylecek's avatar
Radim Tylecek committed
27 28
_NOTE_: Due to bug in Gitlab server valid PLY files are not downloaded with ZIP web link. You can still download them via web individually.

Radim Tylecek's avatar
Radim Tylecek committed
29 30 31
* File [`labels.yaml`](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/calibration/labels.yaml) - semantic label definition list
* File [`colors.yaml`](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/calibration/colors.yaml) - label color definition (for display)
* File [`calibration/camchain-DDDD.yaml`](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/calibration/camchain-2017-05-16-09-53-50.yaml) - camera rig calibration
Radim Tylecek's avatar
Radim Tylecek committed
32

Radim Tylecek's avatar
Radim Tylecek committed
33

Radim Tylecek's avatar
Radim Tylecek committed
34
### Training
Radim Tylecek's avatar
Radim Tylecek committed
35

Torsten Sattler's avatar
Torsten Sattler committed
36
| Sequence | cameras | range | frames | annotated frames |
Radim Tylecek's avatar
Radim Tylecek committed
37
| -------- | ------- | ----- | ------ | ---------   |
Torsten Sattler's avatar
Torsten Sattler committed
38 39 40 41
| train_around_hedge  | cam_0, cam_1*, cam_2, cam_3*   | 100:10:260 | 68  | 34 |
| train_boxwood_row  | cam_0, cam_1*, cam_2, cam_3*   | 90:10:650 | 228  | 114 |
| train_boxwood_slope  | cam_0, cam_1*, cam_2, cam_3*   | 120:10:340 | 92 | 46 |
| train_around_garden_roses  | cam_0, cam_1*, cam_2, cam_3*   | 590:10:690 | 44 | 22 | 
Radim Tylecek's avatar
Radim Tylecek committed
42

Radim Tylecek's avatar
Radim Tylecek committed
43
* File `model_SSSS.ply` - point cloud with semantic labels (field `scalar_s` or color) for sequence `SSS`
Radim Tylecek's avatar
Radim Tylecek committed
44
  * Subfolders `uvc_camera_cam_X`
Torsten Sattler's avatar
Torsten Sattler committed
45
    * Files `uvc_camera_cam_X_fXXXXX_gtr.png` - GT annotation with label set IDs (indexed bitmap)
Radim Tylecek's avatar
Radim Tylecek committed
46 47
    * Files `uvc_camera_cam_X_fXXXXX_undist.png` - undistorted color image (RGB)
    * Files `uvc_camera_cam_X_fXXXXX_over.png` - overlay of annotation over greyscale image (for display)
Radim Tylecek's avatar
Radim Tylecek committed
48 49
    * Files `uvc_camera_cam_X_fXXXXX_cam.txt` - camera parameters (f,c,q,t)

Radim Tylecek's avatar
Radim Tylecek committed
50
* *For `cam_1` and `cam_3` there is no annotation provided, ie. _gtr and _over are missing, and _undist is greyscale only
Radim Tylecek's avatar
Radim Tylecek committed
51

Radim Tylecek's avatar
Radim Tylecek committed
52
#### Camera parameters
Radim Tylecek's avatar
Radim Tylecek committed
53
The pose format is `fx fy cx cy qw qx qy qz tx ty tz`, where `f` is focal length in pixels `c` centre point in pixels, `q` is the quaternion denoting the camera orientation and `t` is the camera translation. 
Radim Tylecek's avatar
Radim Tylecek committed
54
The transformation from world to camera coordinates is given as `[R(q)|t]`, where `R(q)` is the rotation matrix corresponding to quaternion `q`.
Radim Tylecek's avatar
Radim Tylecek committed
55 56 57 58

### Testing

| Sequence | cameras | range | frames |
Radim Tylecek's avatar
Radim Tylecek committed
59
| -------- | ------- | ------ | ------- | 
Radim Tylecek's avatar
Radim Tylecek committed
60
| test_around_garden  | cam_0, cam_1, cam_2, cam_3   | 140:10:580, 700:10:1480 | 257 | 
Radim Tylecek's avatar
Radim Tylecek committed
61

Radim Tylecek's avatar
Radim Tylecek committed
62 63
  * Subfolders `uvc_camera_cam_X`
    * Files `uvc_camera_cam_X_fXXXXX_undist.png` - undistorted color image (RGB)
Radim Tylecek's avatar
Radim Tylecek committed
64
    * Files `uvc_camera_cam_X_fXXXXX_cam.txt` - camera parameters (f,c,q,t)
Radim Tylecek's avatar
Radim Tylecek committed
65

Radim Tylecek's avatar
Radim Tylecek committed
66
## Evaluation
Radim Tylecek's avatar
Radim Tylecek committed
67

Radim Tylecek's avatar
Radim Tylecek committed
68
We evaluate the following measures:
Torsten Sattler's avatar
Torsten Sattler committed
69 70 71
* Reconstruction accuracy in % for a set of distance thresholds (similar to [1,2])
* Reconstruction completeness in % for a set of distance thresholds (similar to [1,2])
* Semantic quality in % of the triangles that are correctly labeled.
Torsten Sattler's avatar
Torsten Sattler committed
72

Radim Tylecek's avatar
Radim Tylecek committed
73
We use distance thresholds from 1 to 30 cm. 
Torsten Sattler's avatar
Torsten Sattler committed
74

Radim Tylecek's avatar
Radim Tylecek committed
75 76
The evaluation code and GT can be found [here](evaluation).
The methodology and performance is given in a [report](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/evaluation/report/rms_challenge.pdf).
Radim Tylecek's avatar
Radim Tylecek committed
77

Radim Tylecek's avatar
Radim Tylecek committed
78 79 80 81
#### References

* [1] Seitz et al., A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR 2006
* [2] Schöps et al., A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos, CVPR 2017
Radim Tylecek's avatar
Radim Tylecek committed
82

Radim Tylecek's avatar
Radim Tylecek committed
83 84
## Submission

Torsten Sattler's avatar
Torsten Sattler committed
85
In order to submit to the challenge, please create a semantically annotated 3D triangle mesh from the test sequence. 
Radim Tylecek's avatar
Radim Tylecek committed
86 87 88 89
* The mesh should be stored in the [PLY text format](http://paulbourke.net/dataformats/ply/). 
* The file should store for each triangle a color corresponding to the triangle’s semantic class (see the [`calibrations/colors.yaml`](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/calibration/colors.yaml) file for the mapping between semantic classes and colors). 
  * Semantic labels 'Unknown' and 'Background' are only for 2D images, and should not be present in the submitted 3D mesh, ie. only values 1-8 are valid.

Torsten Sattler's avatar
Torsten Sattler committed
90
Once you have created the mesh, please submit it using [this link](https://www.dropbox.com/request/23XzljBTn93zYl3ETjXn). 
Radim Tylecek's avatar
Radim Tylecek committed
91
In addition, please send an email to `torsten.sattler@inf.ethz.ch` that includes the filename of the file you submitted as well as contact information.
Torsten Sattler's avatar
Torsten Sattler committed
92 93 94

## Questions

Radim Tylecek's avatar
Radim Tylecek committed
95
For questions, please contact `torsten.sattler@inf.ethz.ch`.
Radim Tylecek's avatar
Radim Tylecek committed
96 97 98 99 100 101 102

## Credits

Dataset composed by @tsattler and @rtylecek.

Data recorded and processed by @dhonegger and @mblaich.

Radim Tylecek's avatar
Radim Tylecek committed
103
### Annotators:
Radim Tylecek's avatar
Radim Tylecek committed
104
* Christos Maniatis
Radim Tylecek's avatar
Radim Tylecek committed
105 106 107 108
* Alex Chan
* Nanbo Li
* Stefanie Speichert
* Xinda Xu
Radim Tylecek's avatar
Radim Tylecek committed
109
* Omar Abarca Arriaga
Radim Tylecek's avatar
Radim Tylecek committed
110

Torsten Sattler's avatar
Torsten Sattler committed
111
Please report any errors via [issue tracker](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/issues/new) or via email to torsten.sattler@inf.ethz.ch.
Radim Tylecek's avatar
Radim Tylecek committed
112

Radim Tylecek's avatar
Radim Tylecek committed
113
### Acknowledgements
Radim Tylecek's avatar
Radim Tylecek committed
114

Radim Tylecek's avatar
Radim Tylecek committed
115 116
Production of this dataset was supported by EU project TrimBot2020.

Radim Tylecek's avatar
Radim Tylecek committed
117 118 119 120 121 122 123 124 125 126 127
Please cite the following [report](https://gitlab.inf.ed.ac.uk/3DRMS/Challenge2017/blob/master/evaluation/report/rms_challenge.pdf) when using the dataset: 


    @techreport{sattler2017rms,
      author={Torsten Sattler and Thomas Brox and Marc Pollefeys and Robert B. Fisher and Radim Tylecek},
      title={3D Reconstruction meets Semantics – Reconstruction Challenge},
      institution={ICCV Workshops}, 
      month={October},
      year={2017},
      URL={http://trimbot2020.webhosting.rug.nl/events/3drms/challenge/}
    }