3D all the way!
This code implements the approach detailed in the paper
3D All The Way: Semantic Segmentation of Urban Scenes From Start to End in 3D, check the project page: http://homes.esat.kuleuven.be/~amartino/2015_cvpr_www/index.html.
OpenCV(recommended version: 2.4), http://opencv.org/downloads.html for region-based features from STAIR vision library (integrated) OR
MatConvNetfor CNN-based region features, http://www.vlfeat.org/matconvnet/
libSVMlibrary for Support Vector Machines, http://www.csie.ntu.edu.tw/~cjlin/libsvm/ OR
liblinearlibrary for large linear classification (recommended if using CNN features), http://www.csie.ntu.edu.tw/~cjlin/liblinear/
CVXMATLAB software for disciplined convex programming, http://cvxr.com/cvx/ (needed only if running the third layer)
Doppiaobject detector, https://bitbucket.org/rodrigob/doppia
Running the application
First, open the
code/setup.m file and redirect the symlinks in the
code/external/ subfolder to the installed libraries on your system. Second, compile the dependencies in the
In the next step you need to set up your dataset. A toy dataset with 4 images and a low-resolution point cloud is provided in the
dataToy/ subfolder. Use this as a template for your new dataset. The data folder should have the following structure:
images/folder with jpg images
labels/folder with ground truth labels in the form of png images (colormap maps pixel colors into classes)
cameras.txtcamera file, VisualSFM output
listtrain.txttraining set (list of filenames without extensions)
listeval.txtevaluation (test) set (list of filenames without extensions)
listall.txtall filenames in images/
pcl.plythe point cloud (e.g. from SfM, PMVS, CMP MVS)
pcl_depth.matdepth of each point from the estimated facade plane
pcl_gt_train.plyground truth point cloud of the training set (colormap maps point colors to classes)
pcl_gt_test.plyground truth point cloud of the test set (colormap maps point colors to classes)
pcl_split.matoutput of facade splitting: list of integers, each 3D point assigned to one facade ID
The parameters are set-up in
code/DatasetConfig.m. Modify this file to reflect the particularities of your dataset.
Finally, run the main script
TODO: Facade splitting in 3D has not yet been integrated. A precalculated split file is provided in
Creating 3D facade models
After running all three layers in 3D, the result will be a set of labeled facades, each facade represented as a set of elements, such as windows, balconies, doors... All of these elements are represented as bounding boxes. To create a realistic-looking model from the labeling, we provide additional code that exports the result in 3dsMax and renders the result.
In short, elements such as windows are modeled as intrusions in the wall. This boolean operation for ,,digging'' window holes is performed directly in 3ds Max. The facade texture is then orthographically projected on the final mesh. The environment also allows us to set lights, cast shadows, use shaders or other kind of post-processing techniques. Now, we will show how to run this code.
Rendering estimated Facades in 3ds max
First, Matlab exports our result to
output/export/3D_3L_facades_3ds/* as several files for each facade,
facadeID.objcontains sky, roof, shop and door
facadeID_wallWin.boxesbincontains wall and windows
Second, run 3ds max by simple opening scene file
Third, open the script using MaxScript->RunScript as shown at the following figure,
code/3ds_max/show_facades.ms. You should get the following simple gui,
The input path should correspond to yours
Once you click on RUN, it reads each exported facade and render it from the window is now selected (be sure it Camera01 which is normalized for the facade).
Code runs without problems on 3ds max 2009.
In some versions of 3dsmax, read binary file may return error because ReadFloat shoud be changed to ReadLong.