Playing for Data
This source code release accompanies the paper
Playing for Data: Ground Truth from Computer Games
Stephan Richter, Vibhav Vineet, Stefan Roth and Vladlen Koltun. In ECCV 2016.
git clone --recursive https://bitbucket.org/visinf/projects-2016-playing-for-data ./playing-for-data.
This code is released under the MIT license, see LICENSE.md for full text as well as 3rd party library acknowledgements.
- CMake (for making OpenEXR and zlib)
- Visual Studio (for compiling renderdoc) - tested on VS 2013
- Python - tested with Anaconda
- MATLAB (for annotating frames) - tested on ML 2013a
Let <PFD_DIR> be the Playing for Data directory with the following structure:
- renderdoc | Graphic Debugger - dump frames to disk
- scripts | Python scripts for extracting data from games using renderdoc
- README.md | this file.
- Use CMake to build zlib <PFD_DIR>/renderdoc/renderdoc/3rdparty/zlib
- Use CMake to build IlmBase (located at <PFD_DIR>/renderdoc/renderdoc/3rdparty/openexr/IlmBase).
CMAKE_INSTALL_PREFIXto your preferred directory.
- Compile IlmBase by building the
- Use CMake to build OpenEXR (located at <PFD_DIR>/renderdoc/renderdoc/3rdparty/openexr/OpenEXR).
Make sure that
ILMBASE_PACKAGE_PREFIXin CMake is set to the install directory of IlmBase
(the one you specified as
CMAKE_INSTALL_PREFIXin the 2nd step).
If you built zlib in a separate build directory, copy the zconf.h from that directory to the
- Compile OpenEXR. Make sure that the ilmbase libraries (Half, IlmThread) are on your library search path
(for setting the path under windows see Preparation 2. below),
otherwise compiling the IlmImf project may fail with some cryptic error (Visual Studio returns
"cmd.exe" exited with code -1073741515).
Disclaimer (Before building renderdoc)
The renderdoc version used here is our modification of an now outdated version of renderdoc.
renderdoc evolves, but we do not integrate these changes into our version.
- We added functions that are not available in plain renderdoc. Running our scripts against a version from the regular renderdoc repository will not work.
- Bugs you find in our version should not be reported to the official renderdoc repository as they may have been fixed already. (except they also occur in the regular renderdoc version independent of using our code).
- Do not ask for help in capturing commercial games on the renderdoc forums.
- If you need features from more recent renderdoc releases, you will need to modify renderdoc on your own.
- Compile the complete renderdoc solution.
- Make sure the OpenEXR dlls are in a directory pointed to by your path variable.
(Setting the path under windows is done at
Control Panel/System and Security/System/Advanced System Settings/Environment Variables)
Setup for capturing:
- Run renderdocui.exe with admin privileges.
- Specify the path where dumped frames are stored: Tools/Options/Directory for temporary capture files
- Enable global process hooking in Tools/Options/Allow global process hooking (May not be needed by all games)
- Open the Capture Executable Tab.
- Point the Executable Path to your game
- If you are using the global hook, the other settings don't matter as they are set in a different process.
If you need to modify them, check <PFD_DIR>/renderdoc/renderdoccmd/renderdoccmd_win32.cpp, starting at line 610.
- Enable Global Hook
- Start the game
- If you see an overlay on the top left of the screen, it works. Otherwise check if your path points to the OpenEXR libraries.
F12 will enable the capture mode and dump a frame every 40 frames. Stop capturing by pressing F12 again.
Setup for processing frames (single frame):
- Start renderdocui.exe
- Load a capture file via Open Log
- Open the python shell
- Run the script we provided via Run scripts or use the interactive shell.
You may need to configure the python script we provided to your environment.
Setup for processing frames (automated):
renderdoc saves UI settings in
If it does not exist, run renderdoc once and close it again.
Open the file and edit the following values:
LoadScriptFileto the script you want to run for each capture file.
ExecuteScriptOnLoadto true. This will load the script you specified above
and execute it right after a capture file has been loaded.
- You can start renderdoc from the command line:
renderdocui.exe <path to capture file>
- To close renderdoc after processing is done, add
renderdoc.AppWindow.Close()to your script
Annotating the data
initLabels.mwill create a label.mat containing
the class names, ids, and colors for annotation.
For annotating of a single frame, we need the following artifacts to be
extracted using renderdoc:
- <frame>_final.png | rendered image, rgb 3 channels, 8-bit each
- <frame>_tex.txt | resource ID to hash list
- <frame>_mesh.txt | resource ID to hash list
- <frame>_shader.txt | resource ID to hash list
<frame>_buffer.png | where buffer is one of meshID, texID, shaderID. images created from the ID buffers. The ID buffers are 24-bit floats encoded into 3 uint8 channels. You can decode them via
img = double(imread(<frame>__<buffer>.png)); id = img(:,:,1) + 256 * (img(:,:,2) + 256 * img(:,:,3)));
for each buffer. The Matlab script requires a <frame>__id.mat, which simply holds these ID buffers.
The resource ID to hash lists have the format
in each line, where the first argument is a resource ID and the last two
arguments are 2 64-bit integers forming the 128-bit hash for this resource.
After extracting artifacts for each frame, label it by running
labelMTS(dir, frame), where dir is the directory containing all extracted resources and frame
is the prexix identifying resources of a specific frame.
Make sure to press the STOP button in the top left corner after you
are done labeling each frame. It saves your annotations.
autolabelStencil.mis an example of how to label MTS automatically
using a class mask derived from e.g. a stencil buffer.
After labeling a certain amount of mesh, shader, texture tuples (MTS),
you can start creating some labeling rules, which will further speed up labeling.
If e.g. a certain shader is only used for cars, we can assign
the car label to all MTS tuples that contain this shader.
Create these kind of rules by running
minimum occurence thresholds of your choice.
labelMTSSingleFrame(dir, frame)if you want to make annotations
for only this frame. Using
labelMTSwill propagate annotations to all frames.
Finally, segment images by calling
Some technical background on the annotation process
During extraction we save ID buffers using resource IDs,
which will differ between game sessions. We convert them
to persistent hashes using the lists mentioned above.
These associations will be cached in
During annotation (labelMTS.m) we create the following files:
<frame>__res2hash.mat | caches resource IDs to hashes for a frame
hash2cid.mat | global MTS to class annotations
shaderHash2class.mat | shortcuts from association rule mining
meshHash2class.mat | shortcuts from association rule mining
* texHash2class.mat | shortcuts from association rule mining