Issue #659 resolved

Log file doesn't play back

Justin Manzo
created an issue

I can't get the logfile to play back. This is most likely user error, but on my machine when I follow the tutorial steps, I can't run Gazebo -p and play back the log file. One is definitely generated in the -r step, but when I hit -p it says "...Exception [LogPlay.cc:86] Unable to parse log file..."

Is this user error or a bug?

Comments (39)

  1. Justin Manzo reporter

    Should probably also mention that I'm running ROS Groovy on this particular machine. This is not the machine I'm running for the DRC Simulator - just Gazebo 1.7.1 plus ROS, 64-bit Linux build.

    An update: I can get this to work just fine if I just load basic Gazebo without running the pr2 world, but if I try to load that it seems to crash. Is this a ROS compatibility issue?

  2. Ian Chen

    The end tag is probably not written to the log file. If that's the case then a quick workaround is to manually edit the state.log file to add </gazebo_log> to the end of the file.

  3. catskul

    We've noticed the same problem and work around in attempting to record a run for the DRC qualification.

    The work around however might not be acceptable for the DRC since it is technically violating the rules about editing the log file. I'd like to request that we reopen this bug and set priority to major.

  4. Ian Chen

    mkqual.bash in drcsim adds the end tag if it's not there.

    As for the missing chunks, only state changes are logged. If nothing is moving in the world, the states do not get recorded. This should not affect log playback.

  5. andy_somerville

    Switched accounts.

    The missing chunk was several seconds worth of movement. In our case it was after crossing the pillars in the qual world, the next few seconds where we pass through the last gate are gone.

  6. andy_somerville

    That is the way we tested originally. At first the error message was that it couldn't be parsed, and then I manually went in and added </gazebo_log> after which it played but stopped near the end.

  7. Ian Chen

    A few of us just went through and verified logging and playback in the qual worlds. We're not able to reproduce the problem.

    Just making sure, did you have enough disk space when doing the recordings?

  8. David Rusbarsky

    I am the colleague Andy refers to. It appears as if another team has run into this problem (they posted on theroboticschallenge.org site, I directed them to this thread). I've ran into the problem twice, although I have not been able to reproduce now that I'm trying to reproduce it. I have plenty of disk space. The only thing I can think of is that the screen recorder software we were using started processing its video at the same time as the sim logger and maybe something funky happened?

  9. gerkey

    David Rusbarsky , catskul, and anyone else who's seen this problem, could you please make the broken logs available to us?

    They're big files, so the best thing would be to put the files in some web-accessible space and send a URL to drcsim@osrfoundation.org.

  10. Steven Peters

    A quick test to see if a log file is valid is:

    gzlog info state.log
    

    which should output something like:

    Log Version:    1.0
    Gazebo Version: 1.7.1
    Random Seed:    10457
    Size:           8.0512 MB
    Encoding:       txt
    

    If it fails, try

    tail -1 state.log
    

    which should be </gazebo_log>. If it's not, then append that tag to your log file with echo '</gazebo_log>' >> state.log and try gzlog info again.

  11. andy_somerville

    From our experience that's not necessarily enough. While adding </gazebo_log> made it possible to parse the log, and play it, the last chunk of data seemed to be missing. This would be consistent with a failure to write to disk on SIGINT. David Rusbarsky suggested that it's possible that he hit Ctrl+c twice when shutting down. Which may have interrupted the SIGINT handler.

  12. David Rusbarsky

    I double checked the video and that doesn't seem to be the case (unless I'm missing something)... I'm uploading the score, state, and the video files and will send a link to drcsim@osrfoundation.org shortly. The score file appears to have written correctly based on the events that occurred in that run.

  13. michele hallak

    We have the same experience than the one described by "andy_somerville": indeed, if we add the "</gazebo_log>" at the end of the state.log, we can parse the log but meaningful part of the run is missing. We sent few logs to drcsim@osrfoundation.org. We are opened to any suggestion.

  14. gerkey

    We've verified that it's possible to quit Gazebo without writing all the accumulated state changes to state.log. The log can end up missing several seconds of data.

    We're working on a fix for this problem and will have it in place before Practice. We don't expect to backport the fix to the version of Gazebo used for Qualification.

  15. michele hallak

    Unfortunately, I updated (gazebo and drcsim) and I am executing "gzlog stop". It doesn't help. The closing tag </gazebo_log> is still missing and I have to add it manually: "echo \</gazebo_log> >> /tmp/qual_task_1/state.log" I tried several times without success. Can I upload qualification log files after echoing </gazebo_log> to the end of the file?

  16. michele hallak

    After I added the closing tag to the log:

    michele@darpaws5:/tmp/qual_task_1$ gzlog info state.log
    Log Version:    1.0
    Gazebo Version: 1.7.2
    Random Seed:    12933
    Size:           81.5436 MB
    Encoding:       bz2
    
  17. gerkey

    michele hallak : Sorry to hear that you're still having this problem. So far we've been unable to reproduce it. Can you please do two things:

    1. Verify that you're running Gazebo 1.7.2 by running:

      dpkg -l gazebo
      

      The last line should say:

      ii  gazebo                                      1.7.2-1~precise                           3D dynamics simulator for robotics
      
    2. Send your broken state.log to drcsim-support@osrfoundation.org. You've sent us broken logs before, which was very helpful; now we'd like to get one produced using Gazebo 1.7.2, including the use of gzlog stop before shutdown.

  18. gerkey

    OK, we've reproduced the problem at OSRF with the current released binary packages. We're debugging now, will provide more information ASAP.

  19. michele hallak

    Great that you solved it! So you don't need broken logs anymore, do you? BTW:

    michele@darpaws5:~/git/robil$ dpkg -l gazebo
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name                                                Version                                             Description
    +++-===================================================-===================================================-======================================================================================================================
    ii  gazebo                                              1.7.2-1~precise                                     3D dynamics simulator for robotics
    
  20. gerkey

    michele hallak : indeed, we don't need your broken logs, but thanks for the offer.

    What we do need is for you to try the latest code and let us know how it works for you. You can build Gazebo 1.7.3 from source now or wait about 30 minutes for the 64-bit Precise binary package to be ready.

  21. michele hallak

    Thanks for efforts to solve this issue. I run several times qual1 and now, I have a new feature:

    1. The state.log is getting the closing tag </gazebo_log>)some time after gzlog stop. It depends of the length of the log. Can take quite long time.
    2. The score.log is getting enhanced with the closing tag </gazebo_log> as well!!!!!
    3. Because of 2., mkqual doesn't include the score.log file in the zip!
    4. I am really sorry for all those problems....
    5. Do you want a sample of the score.log with the closing tag?
    
  22. michele hallak

    It seems that this problem occurs because the order of the files in the input command parameters:

    mkqual 1 score.log state.log instead of mkqual 1 state.log score.log
    
  23. Log in to comment