3D all the way!


This code implements the approach detailed in the paper 3D All The Way: Semantic Segmentation of Urban Scenes From Start to End in 3D, check the project page:

Required libraries

Optional libraries

Running the application

First, open the code/setup.m file and redirect the symlinks in the code/external/ subfolder to the installed libraries on your system. Second, compile the dependencies in the code/3rdparty/ subfolder.

In the next step you need to set up your dataset. A toy dataset with 4 images and a low-resolution point cloud is provided in the dataToy/ subfolder. Use this as a template for your new dataset. The data folder should have the following structure:

  • images/ folder with jpg images
  • labels/ folder with ground truth labels in the form of png images (colormap maps pixel colors into classes)
  • cameras.txt camera file, VisualSFM output
  • listtrain.txt training set (list of filenames without extensions)
  • listeval.txt evaluation (test) set (list of filenames without extensions)
  • listall.txt all filenames in images/
  • pcl.ply the point cloud (e.g. from SfM, PMVS, CMP MVS)
  • pcl_depth.mat depth of each point from the estimated facade plane
  • pcl_gt_train.ply ground truth point cloud of the training set (colormap maps point colors to classes)
  • pcl_gt_test.ply ground truth point cloud of the test set (colormap maps point colors to classes)
  • pcl_split.mat output of facade splitting: list of integers, each 3D point assigned to one facade ID

The parameters are set-up in code/DatasetConfig.m. Modify this file to reflect the particularities of your dataset.

Finally, run the main script code/facade_run.m.

TODO: Facade splitting in 3D has not yet been integrated. A precalculated split file is provided in dataToy/pcl_split.mat.

Creating 3D facade models

After running all three layers in 3D, the result will be a set of labeled facades, each facade represented as a set of elements, such as windows, balconies, doors... All of these elements are represented as bounding boxes. To create a realistic-looking model from the labeling, we provide additional code that exports the result in 3dsMax and renders the result.

In short, elements such as windows are modeled as intrusions in the wall. This boolean operation for ,,digging'' window holes is performed directly in 3ds Max. The facade texture is then orthographically projected on the final mesh. The environment also allows us to set lights, cast shadows, use shaders or other kind of post-processing techniques. Now, we will show how to run this code.

Rendering estimated Facades in 3ds max

First, Matlab exports our result to output/export/3D_3L_facades_3ds/* as several files for each facade,

  • facadeID.obj contains sky, roof, shop and door
  • facadeID_balc.obj containes balconies
  • facadeID_wallWin.boxesbin contains wall and windows
  • facadeID.png containes texture

Second, run 3ds max by simple opening scene file code/3ds_max/textured_facades.max.

Third, open the script using MaxScript->RunScript as shown at the following figure,


choose code/3ds_max/ You should get the following simple gui,


The input path should correspond to yours C:\...\output\export\3D_3L_facades_3ds\


Once you click on RUN, it reads each exported facade and render it from the window is now selected (be sure it Camera01 which is normalized for the facade).


Possible problems

Code runs without problems on 3ds max 2009.

In some versions of 3dsmax, read binary file may return error because ReadFloat shoud be changed to ReadLong.