new python simfactory does not overtake the "-roe" option of Cactus is used with submit

Create issue
Issue #291 closed
anonymous created an issue

I my template for RunScript, I have the line:

[snip] time ${MPIDIR}/bin/mpirun -v --mca btl openib,self --mca mpi_leave_pinned 0 -np @NUM_PROCS@ -npernode @(@PPN_USED@/@NUM_THREADS@)@ @EXECUTABLE@ -roe -L 3 @PARFILE@ [snip]

The option "-roe" will force Cactus to create a log for all processors, and not only for PE0.

The Runscript created by this template only contains the line

time ${MPIDIR}/bin/mpirun -v --mca btl openib,self --mca mpi_leave_pinned 0 -np 8 -npernode 4 /lustre/AEI/alibeck/s imulations/cactus_test-luca-no-openmp/SIMFACTORY/exe/cactus_test-luca-no-openmp -L 3 /lustre/AEI/alibeck/simulation s/cactus_test-luca-no-openmp/output-0003/TOVMHD_CarpetRegrid2_PPM_HLLE_BSSNMoL.par

so the option -roe is missing.

I have used simfactory submit (not create-submit) to start the job. If I use "create-submit" the -roe flag is respected.


Comments (9)

  1. Erik Schnetter
    • removed comment

    The expected behaviour of Simfactory is the following:

    - When a Cactus configuration is built and the --runscript option is given (or if the configuration does not have a run script), then Simfactory's runscript template is copied into the directory containing the Cactus configuration.

    - When a simulation is created, then the run script (and the submit script) accompanying the Cactus configuration is copied into the simulation, and is partly expanded

    - When a restart is submitted, then the simulation's run script is copied into the restart directory, and is fully expanded

    The rationale behind the behaviour is that both Cactus configurations and simulations should be self-consistent if other things on the system change. The submit script and run script very likely depend on the configuration options that were used to build the Cactus configuration, and should therefore remain associated with the executable.

  2. anonymous reporter
    • removed comment

    I principle these ideas are correct. However, I can imagine simulations where one would like to see the content of other PEs at a later stage of the simulations, e.g. if problems occur which do not lead to a error exit pof the slmulation, and where the raeson cannot be identified from the logging of PE0. In such a case it would be really helpful, if one could add ther -roe option. The -roe option is no change in a fundamental part of a simulation, it is only an extension of the logging

  3. Erik Schnetter
    • removed comment

    You can add the -roe option to the corresponding run script, and Simfactory will honour this. There is currently no command to automate this, though.

    If you create a simulation, you need to modify the run script in the Cactus configuration directory. If the simulation already exists, then you need to modify the run script in the simulation.

  4. anonymous reporter
    • removed comment

    Okay, I think this should help. Are you shure that simfactory is looking in the corresponing old output-xxxx directory befor it is starting to run?

  5. Erik Schnetter
    • removed comment

    Internally, Simfactory distinguishes between creating a simulation and submitting a restart. When creating a simulation look for the run script in the Cactus configuration directory, submitting a restart looks in the simulation directory (in the SIMFACTORY subdirectory there). Neither look in the output-NNNN directory; however, this is where the final run script is placed when the restart is submitted.

    You can use the command find -iname "*runscript*" to find where runscripts are stored.

  6. Log in to comment