Parameter files and thornlists could be tested as part of the release process

Create issue
Issue #641 open
Ian Hinder created an issue

There are a number of parameter files and thornlists included as examples with thorns in the ET, and in Cactus. Would it be feasible to have these tested as part of the release process? For example, they should run without crashing or missing thorns, should not trigger NaNs etc, and we should minimise any warnings that are emitted. It would be nice to automate this, though it probably requires a cluster, so would be distinct from the usual test suite procedure, due to requiring more memory than tests.

Something to think about for the next release.


Comments (10)

  1. Ian Hinder reporter

    There are currently 172 example parameter files (arrangements/*/*/par/*.par). Testing these is only feasible automatically. For this, we would need information on how much memory and run time is needed. This could be added as comments to the parameter file (users will also want to know this) which are parsed by whatever system does the testing. We could then test that they at least run without giving any errors. We can't check correctness; for that we hope that the regression tests are sufficient.

    There are 27 thornlists (find arrangements/ -name "*.th" -follow). We should test that these build successfully with the ET checkout. What should we do about thornlists which require non-ET thorns?

    I think that this task is too much to do before the upcoming release, so I am moving the milestone to the next one.

  2. Frank Löffler
    • removed comment

    I only find 8 thornlists within an ET checkout:

    arrangements/CactusIO/IOJpeg/par/ arrangements/CactusIO/IOJpeg/par/ arrangements/EinsteinEvolve/GRHydro/par/ arrangements/Carpet/Carpet/par/ arrangements/Carpet/Carpet/par/ arrangements/Carpet/Carpet/par/ arrangements/McLachlan/doc/ arrangements/McLachlan/doc/

    I don't think it is a problem to include thornlists which include thorns outside of the ET. It is perfectly ok to have thorns being part of the ET, but also usable and interesting in connections with thorns outside of the ET. The same in theory applies to testsuites, but there the issue is size. This isn't an issue with a thornlist.

    As for the example parameter files - yes, they should be - marked as being important for an ET release (if so) - include information on - how much memory they need (could we have Carpet give a 'good guess' and use that?) - categorize them regarding their runtime (giving an exact runtime isn't possible or usefull, but it would be nice to have something like ({quick, moderate, long}).

    Concerning format, Cactus already has a '!DESC' tag, so following this we could use for example:

    !DESC Short description !TAGS ET_2012_05, ET_2012_11 !MEM 2048MB !TIME quick

  3. Ian Hinder reporter
    • removed comment

    Yes you are correct - only 8 thornlists. I was using the checkout on my laptop which has some additional arrangements added. We also have some Cactus thornlists ( though these are no longer linked from, as far as I can tell.

    I agree about allowing thornlists which contain non-ET thorns, but we should try to modify the thornlists to only include ET thorns if possible.

    It would be nice to be able to run Cactus on a parameter file and have it give a good guess about the memory needed. This would need to be done on the head node before qsub is run, which means Cactus would need to be able to run there. It might not be possible to run Cactus on the head node due to mpirun not working there. So for the purpose of this ticket, I would include the approximate size of the run in the parameter file.

    What does "TAGS" mean in your example?

  4. Frank Löffler
    • removed comment

    I would include in TAGS whatever the thorn author finds that parameter file interesting for. The ET release tags might be one option (to indicate that this file should be usable with that release), but others could be added if desired. We could use it to find out which of the parameter files should be tested with the ET in the first place.

  5. Frank Löffler
    • removed comment

    Since we don't have the infrastructure for that in place yet and no time now: milestone bump.

  6. Ian Hinder reporter
    • changed status to open
    • marked as
    • changed milestone to ET_2015_11
    • removed comment

    As discussed at the ET workshop 2015, this is actually a serious issue for new users. They need examples to copy, and if many of them are broken, they are not the best people to fix them, and may be unwilling to ask on the mailing list. Increasing priority to major. I propose that we put effort into this ticket so that all the examples which don't use non-ET thorns can run before the November release.

  7. Log in to comment