Simulating Thermal Cameras

Create issue
Issue #1192 new
Stefan Kohlbrecher created an issue

Every now and then, requests for simulating different spectra than the default visible light pop up, notably thermal imaging. I implemented a quick hack for simulating thermal cameras as described here on ROS Answers and available in hector_gazebo_thermal_camera, but that clearly only works for some scenarios. This ticket is for discussion and tracking any effort towards simulating thermal cameras. I do not have the time to work on this, but maybe someone else steps in.

Issue #550 about arbitrary model properties is related to this one as implementing those could be a proper path towards thermal imaging simulation.

Comments (15)

  1. Nate Koenig

    This would be a nice feature. I think the best approach would involve creating a new Camera type in gazebo that uses a different glsl shader.

  2. Isura

    Our research group will be working in this direction. Arbitrary model properties may be a good idea or a standardization of a thermal property. @Ahsan_Habib

  3. David Rajaratnam

    Just wondering if anyone has made progress on this?

    I am particularly interested in the idea of arbitrary model properties (more so than a simulated thermal camera) and for my purposes want a virtual (depth) camera that assigns a unique color to each model.

    If no-one has made progress on this then I'll try to hack something myself, but I'm not at all confident about how far I'll get. I have limited time, not much experience with the gazebo source code, and most importantly zero experience with 3D rendering, Ogre, glsl shaders, etc,... so any hints would be appreciated.

  4. David Rajaratnam

    @arprice. I can't recall exactly what I did (it's been a while and I've moved away from robotics for the time being) but I did manage to get something working but only through some pretty horrible hacks.

    I never managed to understand Ogre and shaders enough that I could implement things properly so I ended up maintaining a global table that assigned a unique colour to individual models when they are inserted in gazebo. I think I actually replaced the material of the object with that solid colour (but only in the gazebo server and not in the gui).

    Ultimately, it worked for what I needed but I don't think it is anything that would be useful to anyone else.

  5. Andrew Price

    That's funny, those are the two approaches I'm considering. I think having custom shaders is the right way to go, but I'm struggling with how to pass a variable down to that level.

  6. Andrew Price

    Thanks @iche033, those look like exactly what I need! I think I get the broad strokes of how to add a label/segmentation camera, but there's a lot of layers in the shader->material->compositor->sensor->plugin relay race to understand.

  7. Andrew Price

    After a bit of investigation, I'm still wandering in the wilderness of all the various layers of Gazebo/Ogre. As alluded to earlier, my interest is specifically in publishing a "segmentation image" along the lines of the depth image functionality, where each pixel value is the id of the model at that pixel. I've successfully created and loaded a plugin with a custom material that takes a parameter named id and writes that as the color, but there's a number of other steps that I'm unclear on. For instance:

    • In the DepthCamera set of classes, there's both a gazebo::rendering::DepthCamera and a gazebo::sensor::DepthCameraSensor. What is the semantic relationship between these classes, and is an analog of these classes needed to implement a SegmentationCamera?
    • How can one dynamically set the material of a given object or link? I can't find any property that suggests material in either gazebo::physics::Model or gazebo::physics::Link.
    • Can this functionality be loaded purely through the plugin interface, or are modifications to Gazebo itself necessary?

    @iche033 or @nkoenig, can either of y'all advise on a path forward? I'm happy to write the feature as far as I am able, but my knowledge of gazebo's inner workings is currently pretty weak. If this would be straightforward to implement for someone with more framework familiarity but is currently a low priority, maybe there's a way to compensate them for the time investment required? From a logistics perspective, I'm happy to create a different issue/pull request/plugin repo as necessary.

  8. David Rajaratnam

    Hi @arprice. I am not an expert with Gazebo/Ogre and also spent a lot of time in the wilderness of the various layers. So I don't know if this will help but my modified gazebo (https://bitbucket.org/daveraja/gazebo-old) might give you some ideas.

    Note: It's out of date, it's pretty messy, and I probably can't really explain exactly why I did certain things (it was a while ago)! Also I wouldn't try to compile it as I made lots of hacks to the cmake files to be able to build and run it side by side with the standard gazebo versions (my requirements at the time).

    However, the main commit that might be useful is https://bitbucket.org/daveraja/gazebo-old/commits/54005ab2743535d1044d5991b9ea5176df92f8f6

    In particular, I created a new sensor ObjectCamera by basically duplicating DepthCamera; so, adding a gazebo::rendering::ObjectCamera and gazebo::sensor::ObjectCameraSensor etc.

    The code that replaces the model's material with a custom colour is in gazebo::rendering::ObjectUtil.[hh|cc]. I had to modify quite a few other files to make all this work, in particular to call my wrapper create_entity function where ever the original code called the OgreSceneManager::createEntity() function.

    Hope this helps in some way.

  9. Log in to comment