ROS Timestamps out of sync between "real" time and and IHMCAtlasAPI topics.

[Stephen Hart]

We noticed when trying to get the multisense running in ROS that the joint_states from the head (reported from the multisense_ros node) vs. the joint_states from the IHMCATlasAPI node were out of sync. The multisense node seems to be reporting things correctly (according to the actual time), but the control node was reporting thigns from ~3 years ago.

Of course these need to be in sync in order for \tf to register these things together accurately.

[btshrewsbury NA]

@swhart115 What version of the multisense driver / firmware do you use? We are running the 3.3.1 driver, and subscribing to the stamped_pps topic to manage timing. We will check on our end and make sure we didn't do anything crazy.

[Stephen Hart]

It looks like we are running 2.3.1 (not sure why its an old version). But like I said, the ros timestamp on hte multisense ros driver is fine. its the timestamp on the robot side thats old. the result is that we can't construct a full /tf with robot and head, cause the /joint_states messages are thus out of sync.

[DouglasS]

@swhart115 Hey Steve. Atlas and the Multisense head have different clocks since they are different computers. We subscribe to the Multisense topic in the API node and then synchronize these two clocks, rectifying the time stamps in the controller output, using the stamped_pps signal. So without an up to date Multisense driver, you will continue to see this skew.

We just verified this on our robot by brining up the controller and API with the head off, observing the skew in the ROS topic headers, and then by turning on the head (thus exposing the pps signal to our software) the controller time stamps immediately came in line with the Multisense time stamps.

[Stephen Hart]

hmm, i would expect some skew, but not 3 years worth.... I'll look into updating our ms driver. Thanks.

[DouglasS]

@swhart115 That's actually pretty reasonable, the clocks are very very different. We use CLOCK_MONOTONIC and the real-time timestamp from the on-board Atlas computer, which are absolute elapsed wall-time clocks based on an uptime from a fixed point whereas Ros::Time uses CLOCK_REALTIME which is a weak representation of the current wall time. The skew between them can be much greater than 3 years given a lot of factors, especially because CLOCK_REALTIME can be impacted by NTP/leap-seconds/other adjustments and CLOCK_MONOTONIC can't.

[Stephen Hart]

updating the multisense driver, synchronized the clocks.

[Stephen Hart]

we needed to update to the 3.3.1 driver for the multisense