Add FiducialCameraPlugin for Camera Sensors

#2350 Merged at 224dc59
  1. Ian Chen

A camera sensor plugin that publishes the image coordinates of all fiducials detected in the image

a fiducial model is detected only when ~a) it's bounding box is within the camera frustum, and b)~ its center point is not occluded by other models in the camera view.

added an example fiducial world to demo the sensor plugin


homebrew - ok

Comments (13)

  1. Carlos Agüero

    What's the difference between the logicalCameraSensor and the FiducialCameraPlugin? The logicalCamera doesn't require to render the scene but on the other hand it doesn't take into account occlusions? It seems also that the user needs to specify the fiducials that are going to pass the filter. Am I right?

  2. Louise Poubel

    I made one of the fiducial models long and moved it to be partially outside the frustum, but the messages are still showing it (the coordinates for the yellow model come up as [639, 129], and the camera image size is [640, 480]).


    1. Ian Chen author

      looks like I had the wrong understanding of the Camera::IsVisible functionality. Objects that are partially visible are also returned as visible. I've added some comments in the plugin's doxygen to describe the expected behavior. 3b72055

  3. Carlos Agüero

    Optional request: It would be nice to allow another boolean parameter in the plugin filter_all or similar that reports all the objects within the FOV and not occluded. Maybe there are use cases where you don't know in advance the objects that you want to filter.

    1. Ian Chen author

      added ability to track all models. I made it somewhat consistent with logical camera sensor - if no fiducials are specified in the sdf, then all models will be tracked.


  4. Nathan Koenig

    Alternative implementation thought:

    1. Create a new material that renders an object white if it is a fiducial, and black otherwise.
    2. Apply the material to fiducials.
    3. Create a new IR camera sensor that renders the scene using the fiducial material.
    4. Crete a plugin the publishes the location of each visible fiducial.

    This approach is a major departure from this PR. However, it would probably be more efficient.

    Time being what it is; we can move this idea to a feature request. Just wanted to collect thoughts on the approach.

    1. Ian Chen author

      Ya I agree that making a new IR camera sensor will be more efficient than using the camera sensor and doing more processing in the plugin.

      A question about the proposed approach - how does the sensor identify the fiducials? Will it iterate over the image to find the location of white pixels? Each fiducial needs to be unique so the objects need to have different colors. So It's kind of like what selection buffer is doing?

      1. Nathan Koenig

        A plugin would report the location of fiducials, either by directly analyzing the pixels or using opencv. This approach would also let other people use opencv (or some other image processing library) to identify fiducials. The material properties for each fiducial could have a different color.

        I'll make a feature request for this.