CarpetRegrid2 sets up grids which are not consistent

Create issue
Issue #419 closed
Frank Löffler created an issue

The attached parfile creates a grid using CarpetRegrid2 (hg version), which CarpetReduce later warns about:

INFO (CarpetReduce): Simulation domain volume: 4096 INFO (CarpetReduce): Reduction weight sum: 4468.125 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ

The grid info is:

INFO (Carpet): Grid structure (superregions, grid points): [0][0][0] exterior: [0,0,0] : [21,21,21] ([22,22,22] + PADDING) 10648 [1][0][0] exterior: [3,3,3] : [40,40,40] ([38,38,38] + PADDING) 54872 [2][0][0] exterior: [9,9,9] : [42,42,42] ([34,34,34] + PADDING) 39304 [3][0][0] exterior: [21,21,21] : [52,52,52] ([32,32,32] + PADDING) 32768 [4][0][0] exterior: [45,45,45] : [72,72,72] ([28,28,28] + PADDING) 21952 INFO (Carpet): Grid structure (superregions, coordinates): [0][0][0] exterior: [-48,-48,-48] : [288,288,288] : [16,16,16] [1][0][0] exterior: [-24,-24,-24] : [272,272,272] : [8,8,8] [2][0][0] exterior: [-12,-12,-12] : [120,120,120] : [4,4,4] [3][0][0] exterior: [-6,-6,-6] : [56,56,56] : [2,2,2] [4][0][0] exterior: [-3,-3,-3] : [24,24,24] : [1,1,1]

Keyword:

Comments (15)

  1. Erik Schnetter
    • changed status to open
    • removed comment

    The attached patch re-organises some of CarpetRegrid2's internals and may cure this problem.

    Note that there was a recent correction to CarpetReduce that may also influence this ticket.

  2. Roland Haas
    • removed comment

    The attached patch re-organises some of CarpetRegrid2's internals and may cure this problem.

    Note that there was a recent correction to CarpetReduce that may also influence this ticket.

    I also get inconsistent grid setups out of CarpetRegrid2. This time with the McLachlan qc0 example. The patch to CarpetRegrid2 gives me:

    Property N13CarpetRegrid29rotsym180E WARNING level 0 in thorn CarpetRegrid2 processor 0 host phys44230.physics.gatech.edu (line 45 of /localdata/rhaas3/Cactus_devel/configs/devel/build/CarpetRegrid2/property.cc): -> Property does not hold after being enforced

  3. Roland Haas
    • removed comment

    I get the same Property N13CarpetRegrid29rotsym180E level with the newest version of Carpet (from today, incl. the AMR patches). If I turn off rotatingsymmetry180 and double the grid size then the property error goes away (unsurprisingly) but the simulation volume does not agree with the reduction weight sum. Even when I turn off all symmetries (incl. Reflection) the volumes do not agree. Something still seems to be wrong at least for the qc0 parameter file.

  4. Erik Schnetter
    • removed comment

    The attached parameter file tov_16.par has a problem; the simulation domain is too small. In other words, with buffer zones, the first refined level is too large. Enlarging the domain from 256 to 288 cures this problem, and the volume is then correct.

  5. Roland Haas
    • removed comment

    the error has changed but I still get an error message and an abort when using RoatingSymemtry180. This time CarpetRegrid2 eventually fails with:

    WARNING level 0 in thorn CarpetRegrid2 processor 0 host phys44230.physics.gatech.edu (line 383 of /localdata/rhaas3/Cactus_devel/configs/devel/build/CarpetRegrid2/regrid.cc): -> Could not enforce grid structure properties; giving up

    after a couple of

    INFO (CarpetRegrid2): Regridding levels 1 and up WARNING level 1 in thorn CarpetRegrid2 processor 0 host phys44230.physics.gatech.edu (line 389 of /localdata/rhaas3/Cactus_devel/configs/devel/build/CarpetRegrid2/regrid.cc): -> Could not enforce grid structure properties in this round, starting another round

    The error persists if I turn off the z symmetry. Without RotatingSymmetry180 (but with z symmetry) it works fine (please ignore the attached -norot file, it is incorrect [I had made a mistake when turning off RotatingSymmetry]).

  6. Roland Haas
    • removed comment

    Replying to [comment:6 eschnett]:

    The attached parameter file tov_16.par has a problem; the simulation domain is too small. In other words, with buffer zones, the first refined level is too large. Enlarging the domain from 256 to 288 cures this problem, and the volume is then correct.

    Adding:

    CarpetRegrid2::min_distance = 0 also fixes the issue. Is there a reason that CarpetRegrid2 defaults to enforcing a min_distance of 4 grid points between the refinement levels? I found this a surprising (and so did apparently Frank) when I first ran into it (in the WeylScal4 testsuite)? It seems strange that CarpetRegrid2 would modify the grid setup the user describe if this is not required to produce a well nested / standards conforming grid structure. If this is parameter is new and could also be set to default to zero, is it to late to bring up the suggestion to have min_distance's default changed 0?

  7. Erik Schnetter
    • removed comment

    This parameter is not new, and it always defaulted to four. This value comes from a "gut feeling" of mine, which is probably not the best way to set a default.

    While changing this parameter may change people's grid structures, I think it is not too late to do this for the Mercurial version. If you propose/discuss this on the mailing list we can change it.

  8. Bruno Mundim
    • removed comment

    I am facing the same issue for RotatingSymmetry90 as Roland reported for RotatingSymmetry180. Even setting

    CarpetRegrid2::min_distance = 0

    doesn't solve this issue.

  9. Erik Schnetter
    • removed comment

    Bruno

    When extending the domain for Frank's parameter file, I cannot reproduce the problem. Please attach a parameter file that shows you problem.

  10. Bruno Mundim
    • removed comment

    Hi Erik,

    I have just attached one parameter file as example. Other parameter files that used to work now fail too. Maybe this check is too restrictive? Pi symmetry may not be conserved to machine precision depending on how your mesh spacing is set... In any case, I don't know why this is happening. Thanks for the help!

  11. Erik Schnetter
    • changed status to resolved
    • removed comment

    The grid structure consistency check looks at the grid structure, not at the values.

    It turns out the detected inconsistency was outside of the simulation domain, i.e. it was harmless. (Take the refined box you have and rotate it by 180 degrees. Part of it will then be to the left of the y axis, i.e. outside of the domain. Any inconsistency there is irrelevant, since there won't be any time evolution there anyway.)

    This problem has been corrected in the repository.

  12. Log in to comment