quick rsync for simfactory

[Frank Löffler]

Quite often I find myself developing something (code/parfiles) locally and use 'sim sync' to get it to a remote location. Until a recent change in simfactory this was usually quite fast (a couple of seconds at most). In all of these cases I know that I didn't change anything on the remote side. Now 'sim sync' can take a minute even to local machines, depending on their file system load since the remote side has to actually read the complete tree for comparison. And this is without any actual changes.

I propose to have a 'sim sync --fast' option which uses timestamps as before. This should not be the default since it might not catch all changes in every situation, but it would make quick development sessions like this (edit locally, compile and run remotely) feasible again.

Keyword:

[Erik Schnetter]

Simfactory already has "sync --parfiles", which copies only parfiles and is thus much faster.

What about the following idea: Simfactory first uses timestamps to copy files, optimistically assumes that this led to the correct results and begins the build on the remote system, but continues with a full (slower) sync as well. If the full sync exposes a problem, the build is aborted. Otherwise, the build result is accepted.

What is your target system? Would it make sense to set up e.g. DropBox or Google Drive to keep the source tree in sync? These are faster since they begin to copy while you edit, and monitor (via the operating system) which files are being changed.

[Frank Löffler]

Parfiles was just one example. It might be a regular source file as well.

I don't really like the idea of the test for a successful build. A build might be successful even when a file is not updated (yet) and thus the exe contains the wrong code in the end (built earlier than the parallel sync was copying the new file).

In this case the system is supermike at LSU, but that is even one of the faster machines. I don't like setting up a system like DropBox or Google drive on a cluster head node. Not only do I worry about admins disliking this, I also don't like to put my passwords on these machines.

[Frank Löffler]

Also, I tend to use the local simfactory just for the rsync and then the remote simfactory (within a second terminal, logged in remotely) for the build.

[Ian Hinder]

As someone who never edits on the remote system, I would also benefit from such a "quick" option. I would like to be able to set it in my configuration and obtain the old behaviour. I also dislike the proposal for detecting a failed build and syncing in the background. It is too complicated and error-prone, in my opinion.