- removed comment
Simfactory's test mechanism should use softlinks instead of rsync
Issue #1721
open
Simfactory's test mechanism copies all test cases from the source tree into the simulation directory. This takes a long time and a lot of space. This should rarely be necessary, since these test usually run soon after being submitted, so that there is little chance of the source tree becoming outdated.
Keyword:
Comments (2)
-
-
- changed status to open
- removed comment
- Log in to comment
The main reason I added support for running the tests using simfactory was so that I could run them on Kraken, where the compute nodes didn't have access to the home filesystem, so the copying was essential. I am also a little nervous about the assumption that the source tree stays around. The tests may sit in the queue for a while, or you may change branch in the source directory. I agree it would be good to reduce the time taken to copy the data, but I seem to remember that a design goal of simfactory was to be correct, and avoid people making mistakes, at the potential expense of efficiency.
Maybe we could have a "cache" of the required files on the scratch filesystem, and use rsync's link-dest option, to make hard links to that cache. It wouldn't help the first time, but subsequent jobs would benefit.