Online Multi-Target Tracking with Recurrent Neural Networks

This code accompanies the following paper: (pdf)

Online Multi-target Tracking Using Recurrent Neural Networks
A. Milan, S. H. Rezatofighi, A. Dick, I. Reid, K. Schindler. In: AAAI 2017


    title = {Online Multi-Target Tracking using Recurrent Neural Networks},
    booktitle = {AAAI},
    author = {Milan, A. and Rezatofighi, S. H. and Dick, A. and Reid, I. and Schindler, K.},
    month = {February},
    year = {2017}



* Lua
* Torch
* luarocks: nn, nngraph, lfs


* cunn, cutorch (optional for GPU usage)
* Matlab (visualization and metric computation only)



# Follow these instructions to install Torch:

# in a terminal, run the commands
curl -s | bash
git clone ~/torch --recursive
cd ~/torch; ./

Additional Dependencies


sudo apt-get install luarocks
luarocks install nn
luarocks install nngraph
luarocks install luafilesystem
luarocks install
luarocks install cutorch
luarocks install cunn


The current version uses the training set of the MOTChallenge benchmark to generate synthetic data for training.

You should download the dataset from here and set the local path in src/util/paths.lua



The code comes with a pre-trained model located in bin/rnnTracker.t7. It was trained on a subset of the MOTChallenge 2015 training set

Run th rnnTracker.lua to get a sense of the result on synthetic data or th rnnTracker.lua -model_name rnnTracker -seq_name TUD-Campus

to produce results on the TUD-Campus sequence. The bounding boxes are saved in ./out/rnnTrack/TUD-Campus.txt. Type th rnnTracker.lua -h to get a full list of options

This example uses Hungarian data association.


To see the visual results you can run th visBoxes.lua -file ../out/rnnTracker_r300_l1_n1_m1_d4/TUD-Campus.txt


th trainBF.lua -config ../config/configBF.txt

will start training a model on the TUD-Campus sequence. Type th trainBF.lua -h

to see the full set of options. You may define the training parameters in a separate text file, similar to config/configBF.txt and pass it as the -config option to the training script.


Training expects annotated image sequences. The annotation format is a CSV text file following the syntax of the MOTChallenge benchmark. For testing, the image sequence and the corresponding set of detection in the same format is required.

Internal representation

Internally, all data (tracks and detections) is stored in N x F x D tensors, where

  • N = max. number of targets / detections
  • F = number of frames in a batch
  • D = dimensionality (e.g. 2 for (x,y) or 4 for (x,y,w,h)

The labels (data association) is represented by an NxF tensor.

Furthermore, training, validation and real-data sets are kept in a lua table. I.e. each entry in a table is then a MB*N x F x D tensor, where MB is the mini-batch size. There are four tables for each set.

  1. Tracks
  2. Detections
  3. Labels
  4. Sequence names (used for generating that one specific datum)


The code is documented following the luadoc convention. To generate html docs, install luadoc luarocks install luadoc and run ./

Known issues

Data Association

The code for training data association is not included yet. We are working on releasing it soon.

Training data

Training data is generated synthetically by learning simple generative trajectory models from annotated data. Training with real data is not supported.

Acknowledgements and remarks

We thank Andrej Karpathy for releasing his code on character-level RNNs that served as basis for this project.

Pull requests welcome.


BSD License