Simulator fails to start properly (~20% of the time)

[Steven Gray]

Since adding a bunch of sensors in my custom config, I'm now seeing gear fail to start properly about once every five runs. Possible race condition?

No handlers could be found for logger "rosout"
[ INFO] [1553440576.585037505]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1553440576.585816886]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
[ INFO] [1553440576.665597179]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1553440576.666733524]: waitForService: Service [/gazebo_gui/set_physics_properties] has not been advertised, waiting...
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
[INFO] [1553440576.930366, 0.000000]: Loading model XML from ros parameter sensors_description
[INFO] [1553440576.934118, 0.000000]: Waiting for service /gazebo/spawn_urdf_model
[INFO] [1553440576.965256, 0.000000]: Loading model XML from ros parameter robot_description
[INFO] [1553440576.970430, 0.000000]: Waiting for service /gazebo/spawn_urdf_model
[INFO] [1553440577.018551, 0.000000]: Loading model XML from ros parameter robot_description
[INFO] [1553440577.023564, 0.000000]: Waiting for service /gazebo/spawn_urdf_model
[INFO] [1553440579.944332, 0.000000]: Calling service /gazebo/spawn_urdf_model
[        LOTS OF SENSOR PRINTOUTS HERE, REDACTED :)      ]
[INFO] [1553440579.981526, 0.000000]: Calling service /gazebo/spawn_urdf_model
[INFO] [1553440580.033269, 0.000000]: Calling service /gazebo/spawn_urdf_model
Service call failed: transport error completing service call: unable to receive data from sender, check sender's logs for details
[ERROR] [1553440580.101463, 0.000000]: Spawn service failed. Exiting.
Service call failed: unable to connect to service: [Errno 104] Connection reset by peer
Service call failed: unable to connect to service: [Errno 104] Connection reset by peer
[ERROR] [1553440580.113691, 0.000000]: Spawn service failed. Exiting.
[ERROR] [1553440580.113743, 0.000000]: Spawn service failed. Exiting.

Full output attached (though with sensor printouts redacted).

[Steven Gray]

@sloretz I just emailed you my competition config in case that helps debug

[Shane Loretz]

@stgray thanks for the reproducible example. I see the crash too on some starts on my machine.

[Steven Gray]

@sloretz Whew, glad it's not just me.

[Steven Gray]

Got into a situation where I could not get the sim to start properly with all my sensors on 6 consecutive attempts.

[Steven Gray]

@sloretz Any updates on this? At the least, will there be a check in the evaluation that the simulation came up okay, so I won't be penalized if the sim never starts?

[Shane Loretz]

@stgray You won't be penalized if the simulator fails to start.

[Steven Gray]

@sloretz Great, thanks. It seems to be almost 50% of the time now.

If I were to debug, where would you suggest I start?

[Shane Loretz]

If I were to debug, where would you suggest I start?

It would be helpful to know if the type of sensors in the config file made the crash more or less likely.

I'll be looking at this with a debug build of the ariac gazebo_ros_pkgs shortly.

[Steven Gray]

Laser profilers seem to be the culprit.

[Shane Loretz]

@stgray thanks for narrowing it down to laser profilers. I haven't seen the crash crash after building gazebo from source with this pull request: https://bitbucket.org/osrf/gazebo/pull-requests/3096/lock-subscribermutex-anywhere/diff

It may take a while for a new gazebo 9 release with that change, so I'll look at mitigating it here too.

[Shane Loretz]

pull request #133

[Shane Loretz]

@stgray if you're still building from source, would you mind checking if the crash is resolved now that pull request #133 is merged?

[Steven Gray]

@sloretz I couldn't reproduce the crash in a dozen tries, seems likely that it's fixed. Thanks!

[Shane Loretz]

Resolved in 3.0.3