- removed comment
move PARAMCHECK before WRAGH
currently PARAMCHECK runs after WRAGH which means that the lev0 data and grid arrays as well as grid scalars have storage (but the grid is not yet set up properly since basegrid did not yet run).
This means that memory to hold level 0 has been allocated which can be quite large (eg for mulitpach runs where level 0 encompases all multipatch grids). Therefore using cactus_sim -P parfile.par to test parameters and parameter values before submitting a job (which is good practise) can potentially cripple the head node on which one tries this.
Would it be possible to to swap the order (and make sure to call CCTKi_FinaliseParamWarn right after PARAMCHECK) or is do we require that storage exists during PARAMCHECK?
For the code that provides the current ordering, see CallSetup in Carpet/src/Initialise.cc and src/main/CactusDefaultInitialise.c (in the flesh).
Keyword:
Comments (5)
-
-
reporter - removed comment
Not sure I follow. I want to only change the order fo WRAGH and PARAMCHECK both of which are traversed from within CallSetup (lines 183 and 186):
182 // Register coordinates 183 ScheduleTraverse (where, "CCTK_WRAGH", cctkGH); 184 185 // Check parameters 186 ScheduleTraverse (where, "CCTK_PARAMCHECK", cctkGH); 187 } END_MGLEVEL_LOOP;
I have not tried this, but wouldn't moving WRAG before ScheduleGHInit by a bad idea?
-
- removed comment
WRAGH is supposed to be before PARAMCHECK, so that one can set up certain data structures that may be required for parameter checking later on.
However, you are right that the grid structure should not be allocated before PARAMCHECK. Thus moving all the routines that handle the grid structure setup, schedule initialization, etc. should be called after PARAMCHECK. This may create some fall-out and/or may require some changes to schedule routines, but it should be possible.
-
reporter - removed comment
ok. This will require some rewriting of code inside of Carpet I think. Right now the docs for the time bins ion page D28 of the user guide) say:
CCTK WRAGH This timebin is executed when all parameters are known, but before the driver thorn constructs the grid. It should only be used to set up information that is needed by the driver. CCTK PARAMCHECK This timebin is for thorns to check the validity of parameter combinations. This bin is also executed before the grid hierarchy is made, so that routines scheduled here only have access to the global grid size and the parameters.
so no one in PARAMCHECK may expect access to grid functions and technically grid arrays and grid scalars are also not mentioned. Checking the various schedule.ccl there are routines scheduled in WRAGH (eg CarpetRegrid2_Initialise) that access grid scalars. So we will need to turn on grid arrays and grid scalars before WRAGH (and hope that those are "small" which I have at least one thorn for which this is not true since it allocates 3d distributed grid arrays). Probably best committed after the release.
-
- removed comment
Carpet does internally not distinguish between grid functions and grid arrays. There is a distinction between variables with global storage (defined in top-level statements in the scheduler) and variables with temporal storage (defined in schedule groups or scheduled routines). From this point of view, it would be cleaner to not allocate any storage before paramcheck.
CarpetRegrid2 could run later, early in the basegrid bin.
Instead of moving things from wragh to basegrid, we could introduce a new bin "basebasegrid" that is run before basegrid, but only in global mode. This may help move schedule entries around.
- Log in to comment
This may work. You would want to move "CallSetup" to before "CCTKi_ScheduleGHInit". Worth a try...