1. Julian Kooij
  2. sensecap

Overview

HTTPS SSH

SenseCap data collection tool

(c) Julian Kooij, Delft University of Technology

This code is released under the 3-clause BSD license (see LICENSE.txt).

SenseCap is a recording tool to capture synchronized video and skeletal data streams from cheap sensors such as the Microsoft Kinect2, and LeapMotion. Instead of providing a virtual playback devices, such as KinectStudio, SenseCap is suitable for multi-media data collection and off-line processing. This is done by encoding image streams in common video codecs, and skeletal data in flat text tables. This approach enables long duration recordings (e.g. over 30 minutes), and supports post-hoc mapping of the Kinect2 depth video to the color space if needed. It is also possible to record the Kinect2 and LeapMotion simultaneously. The project is released under a 3-clause BSD license, and consists of an extensible C++11 framework, with support for the official Microsoft Kinect 2 and LeapMotion APIs to record, a command-line interface, and a Matlab GUI to initiate, inspect, and load Kinect2 recordings.

Screenshots

Capturing the Kinect2 from the command-line

Simultaneously recording Kinect2 and LeapMotion

Post-processing depth-to-color mapping

Matlab GUI screenshot

Installation instructions

You can read more about library dependencies and code compilations in the installation instructions.

Developing information

Some information on how to use the library, including a short tutorial can by found here

API documentation

The API reference can be found at docs\html\index.html To regenerate the documentation and API reference, run doxygen in the root of the project directory where the Doxyfile is located.

Matlab GUI

Please read the Matlab page for more info on the Matlab GUI.

The sensecap utility

The sensecap utility can execute data collection with the Kinect 2 and LeapMotion, playback the recorded data, and for the Kinect 2 map recorded depth data to the color image space. By default, it stores the captured streams in the current working directory, but the --prefix command-line option can be used to store the files in another directory.

Running sensecap --help

=--------------------------------------------=
 (c) Julian Kooij  (J.F.P.Kooij@tudelft.nl)
      Delft University of Technology
 Technology in Motion (https://tim.lumcs.nl)
=--------------------------------------------=

options:
  --help                produce help message
  --stdio               enable command mode via STDIN/STDOUT
  --option arg          select program to execute
  --prefix arg          filename prefix (including subdirs)

For the interactive menu, simply run sensecap. This should show something like:

=--------------------------------------------=
 (c) Julian Kooij  (J.F.P.Kooij@tudelft.nl)
      Delft University of Technology
 Technology in Motion (https://tim.lumcs.nl)
=-------------------------------------------=

  Use standard I/O: 0
              Mode: ?
       Path prefix: ./

*****************************
Select mode:
*****************************
 'A': capture kinect & leapmotion
 'H': capture leapmotion hand
 'K': capture kinect
 'V': capture video
 'a': replay kinect & leapmotion
 'c': convert kinect
 'h': replay leapmotion hand
 'k': replay kinect
 'l': view kinect
 'v': replay video
 'q': QUIT
*****************************
>>

Execute a mode by entering the letter and pressing ENTER. Note: if you started a recording, press q to end it, when either the terminal or one of the windows is focused.

If you have sensecap compiled without the Kinect2 or LeapMotion SDK, you can download an example recording file instead. Ensure that the .avi, .txt files from the zip are in the working directory, or in a path indicated with the --prefix argument.

Recorded data formats

For the Kinect 2, the following files are recorded: kinect.avi kinect.avi.packetinfo.txt kinect_depth.avi kinect_depth.avi.packetinfo.txt kinect_label.avi kinect_label.avi.packetinfo.txt kinect_body.txt kinect_colorbody.txt * kinect_depthbody.txt

For the LeapMotion, the following files are recorded: leap_left.avi leap_left.avi.packetinfo.txt leap_right.avi leap_right.avi.packetinfo.txt * leap_hands.txt

Each video file X.avi has an accompanying X.avi.packetinfo.txt file, which has at each line a timestamp for each frame in video. Timestamps are long integers representing milliseconds.

Note that you can open the video files in a standard video player, such as VideoLan, but that the playback framerate is just some hardcoded value. The real framerate depends on lighting conditions and your system, and can be computed from the timestamps in the packetinfo.txt file.

Kinect data

There are three video streams: kinect.avi stores the HD 1920 x 1080 RGB color video stream kinect_depth.avi stores the depth 512 x 424 depth camera stream. The intensity channel gives a 8-bit value (0-256) mapped to the 0 - 4 meter depth range. * kinect_label.avi stores the binary foreground/background masks in the depth image space

The kinect_body.txt, kinect_colorbody.txt, kinect_depthbody.txt store Kinect body skeletons in respectively world coordinates, color camera coordinates, and depth camera coordinates. World coordinates are the regular 3D coordinates, while color and depth image coordinates express the coordinates in terms of pixel locations with the color and depth image (+ z-distance). Each file has per line 202 values, namely 1: timestamp 2: skeleton id 3-102: for all 25 joints (x,y,z) position + tracking state 103-202: for all 25 joints 4D quaternion (see here)

The skeletal joints are numbered in the same way as the official Kinect 2 API, see here.

LeapMotion data

leap_left.avi and leap_right.avi store the left and right infrared images respectively, each as a 620 x 240 gray-scale video.

The leap_hands.txt file stores articulated hand data on each line using 125 numbers: 1: timestamp 2: is_valid flag, indicates if the LeapMotion marks the hand as a valid items 3: is_left, boolean to indicate left or otherwise right hand 4: confidence, indicates the tracker confidence * 5-125: for each of the 5 fingers, 4 bones, per bone an (x,y,z) 3D 'from' vector, and an (x,y,z) 3D 'to' vector

The 5 fingers are ordered "Thumb", "Index", "Middle", "Ring", "Pinky", and the 4 bones are ordered "Metacarpal", "Proximal", "Middle", "Distal", just as in the official LeapMotion API.