Filter rules error when using rsync 2.6.9

Create issue
Issue #461 closed
Ian Hinder created an issue

When I use rsync 2.6.9 (the default for Mac OS 10.6) on my local machine, I get the error

invalid modifier sequence at 'p' in filter rule: -p _darcs

when running sim sync. Probably this is an rsync 3 feature.

Keyword:

Comments (19)

  1. Frank Löffler
    • removed comment

    Could you try updating rsync? rsync 3.0.0 was released more than three years ago and 2.6.9 in 2006...

    Having said that - should we add a check for the version of rsync?

  2. Barry Wardell
    • removed comment

    Replying to [comment:1 knarf]:

    Could you try updating rsync? rsync 3.0.0 was released more than three years ago and 2.6.9 in 2006...

    The 'p' rule was added in 3.0.0, so updating to that version will fix the problem. Unfortunately 2.6.9 is included by default with the latest version of OSX so many people will likely encounter this.

    Having said that - should we add a check for the version of rsync?

    I agree that adding a check for the rsync version is a good idea.

  3. Erik Schnetter
    • removed comment

    Updating is not a general solution -- we have to work with whatever is available on the given systems, and many don't have rsync 3 yet. We will either have to disable this altogether, or test the rsync version and use a sensible fallback if rsync is too old.

    Instead of explicitly testing rsync's version, we can just call rsync, and if it fails, call it again with a reduced set of options.

  4. Frank Löffler
    • removed comment

    I agree that we could fall back in the case of the -p option. However I am also a bit hesitant when it comes to supporting software which is that old. Is this only affecting Macs, and is the -p option the only one affected? -p preserves the permissions - is the long option available in 2.6.9: --perms? If not we could also just not add this option for an old version of rsync, but warn about it.

  5. Erik Schnetter
    • removed comment

    Many HPC systems run older versions of rsync. Many also run Python 2.3. We need to support these. rsync 2.6.9 is a very common version. Mind that omitting the "-p" option etc. works fine -- the only drawback is that we see ten or twenty warning messages when we sync. That is a small price to pay for supporting these systems.

    Remember also IBM AIX systems, where the patch command is from 1988... we're dealing with these systems as well.

  6. Barry Wardell
    • removed comment

    I've attached a patch which only uses the p filter rule if we are using at least rsync 3.0.0. It prints a warning for older versions of rsync.

  7. Erik Schnetter
    • removed comment

    The patch seems fine, please apply. (I assume it's been tested.)

    I'm slightly worried that checking the version number may not always be correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.

    Does this only depend on the local rsync version, or also on the remote version? We have no control over the remote version.

  8. Barry Wardell
    • removed comment

    Replying to [comment:7 eschnett]:

    The patch seems fine, please apply. (I assume it's been tested.)

    Yes, I tested with both 3.0.7 and 2.6.9 on my Mac, syncing to Damiana.

    I'm slightly worried that checking the version number may not always be correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.

    The particular feature we're worried about was added in 3.0.0 according to the rsync changelog. Patch committed.

    Does this only depend on the local rsync version, or also on the remote version? We have no control over the remote version.

    Unfortunately, yes. Both the remote and local versions of rsync need to be up-to-date. If the version on the remote machine is too old it fails with the error:

    "filter rules are too modern for remote rsync."

    We seem to mostly provide our own rsync on remote machines, though, don't we?

  9. Erik Schnetter
    • removed comment

    We provide an rsync on some machines, but not on all. A new rsync isn't a requirement.

    That means that the current patch is incomplete. We either need to "ssh+rsync --version" to check the remote version as well, or call rsync and look for the error message.

  10. Ian Hinder reporter
    • removed comment

    ssh connections to HPC systems can be incredibly slow - maybe up to 10 seconds for something like Kraken. We shouldn't make more of these than are necessary. One option would be to use the ControlMaster feature if we wanted to do this. This would add yet another thing which could go wrong, but it would actually be quite nice because it means you could sync and submit in the same ssh connection.

  11. Erik Schnetter
    • removed comment

    Yes, ssh can be very slow. Hence my suggestion to first try to sync with the full feature set, without checking anything. If this fails, we analyse the return code or error message, and/or try with a reduced feature set. This is as fast as is possible on Kraken.

    We could even skip the initial "ssh mkdir" unless rsync fails, since this won't be necessary in general.

  12. Barry Wardell
    • removed comment

    What about adding an optional entry to the machine database for the rsync version. If it is specified and sufficiently recent, then we use the 'new' rules. If it is too old or not specified, we default to the 'old' rules. Anyone who uses a machine and is sufficiently annoyed by the .pyc files that the old rules sees will then be motivated to update that machine's entry.

  13. Erik Schnetter
    • removed comment

    I prefer MDB entries that are as small as possible. Things that can be done automatically should be done automatically. Only things that are impossible to detect at run time should be in the MDB.

    I assume that rsync returns an error code. It should be easy to construct a command such as

    rsync-with-permissions || rsync-without-permissions

    which would do the "right thing" automatically.

    Or maybe even

    rsync-with-permissions || rsync-without-permissions || { ssh mkdir && { rsync-with-permissions || rsync-without-permissions } }

  14. Ian Hinder reporter
    • removed comment

    This is particularly serious as lonestar, the newest teragrid machine, has rsync 2.6.8. This means that we cannot currently sync to lonestar. I get the error:

    filter rules are too modern for remote rsync.

    You can work around this temporarily by using a version of rsync 3 that I just compiled by adding the following to your defs.local.ini:

    [lonestar] rsynccmd = /home1/00915/hinder/software/rsync-3.0.8/bin/rsync

  15. Barry Wardell
    • removed comment

    I've attached a patch which works around the problem by first trying the new rules and then falling back to the older ones if the first attempt fails with a protocol error. This is essentially implementing Erik's suggestion. OK to apply?

  16. Erik Schnetter
    • removed comment

    The patch is good to apply if it has been tested. (I have not tested it myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.

    I would also reduce the warning to an information, since there are no correctness problems with an old rsync, and in many cases people won't care that much about sync performance. (And those who do will read the output, or should be punished for not doing so.)

  17. Barry Wardell
    • removed comment

    Replying to [comment:16 eschnett]:

    The patch is good to apply if it has been tested. (I have not tested it myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.

    I tested it under 4 conditions:

    • rsync 3.0.7 on my laptop, 2.6.8 on Kraken
    • rsync 3.0.7 on my laptop, 3.0.3 on Kraken
    • rsync 2.6.9 on my laptop, 2.6.7 on Kraken
    • rsync 2.6.9 on my laptop, 3.0.3 on Kraken and it does the correct thing in all four cases. I will commit the patch now.
  18. Log in to comment