Carpet with Periodic Boundary Conditions

Create issue
Issue #558 closed
Hal Finkel created an issue

The current Carpet driver is not compatible with the periodic boundary conditions (as provided by the Periodic thorn or otherwise).

Erik's comment on the User's list was: "It is straightforward to implement periodicity in the grid structure -- one needs to take the current grid structure, shift it in the 26 directions, and take the logical union of all 27 grid structures. This will then be automatically clipped. The most complex part is calculating by how much to shift."

Please implement support for periodic BCs in Carpet. This will directly enable production science runs.

Keyword: carpet
Keyword: periodic

Comments (26)

  1. Hal Finkel reporter
    • removed comment

    Unfortunately, that does not seem to work. I tried setting: CarpetRegrid2::symmetry_periodic_x = yes CarpetRegrid2::symmetry_periodic_y = yes CarpetRegrid2::symmetry_periodic_z = yes and with and without Periodic::periodic = yes. When the regridding meets the boundary, odd values appear.

  2. Hal Finkel reporter
    • removed comment

    It looks like something is wrong with the testing condition, because even when the regridding meets the boundary, I never see the "making regions periodic in the..." message with veryverbose turned on.

  3. Erik Schnetter
    • removed comment

    I believe the offset (integer domain size) is calculated wrong. I attach a new patch.

  4. Hal Finkel reporter
    • removed comment

    Unfortunately, the new patch also does not seem to work. Also with this version, I never see the "making regions periodic in the..." message with veryverbose turned on.

    I'll post my stdout.log file.

  5. Hal Finkel reporter
    • removed comment

    We seem to be getting closer. Do we need to insert a boundary_clip() operation after the fixup in each direction? I ask because... Refinement level 1: making regions periodic in the x direction... New regions are bboxset<CCTK_INT4,3>:{size=164454,setsize=63,set=[([... Refinement level 1: making regions periodic in the y direction... New regions are bboxset<CCTK_INT4,3>:{size=493362,setsize=189,set=[([

    and it is still running after many hours, seemingly stuck while executing: CarpetRegrid2::periodic<1>::test_impl(...), specifically executing the line: ibset const difference = symmetrised - regions.AT(rl); (so this is in the post-enforcement check for the y direction)

  6. Hal Finkel reporter
    • removed comment

    It did eventually move on to doing CarpetRegrid2::periodic<2>::test_impl, so it was not in an endless loop, but it then started processing a box set with some 1.8k members.

    If I understand the problem correctly, then inserting extra calls to: properties.push_back (new boundary_clip()); after the queueing of periodic<0>() and periodic<1>(), should help, but that does not seem like a general solution (because it implies extra overhead even when the periodic b.c. handing is turned off).

  7. Erik Schnetter
    • removed comment

    Inserting an extra call to clip should not be a performance issue, because these routines are only called during regridding, not during evolution.

    One alternative may be to modify the shifting to

    Shift boxes upwards and downwards by one period symmetrised |= bb.shift(+ioffset / bb.stride()) & baseextent; symmetrised |= bb.shift(-ioffset / bb.stride()) & baseextent;

    which does part of what the clipping does.

    The underlying problem, of course, is Carpet's inefficient implementation of bboxsets. This really needs to improve.

  8. Hal Finkel reporter
    • removed comment

    Inserting the extra calls to: properties.push_back (new boundary_clip()); does allow the regridding to proceed in a reasonable amount of time, but, unfortunately, something is still wrong. I've attached new image from right after it regrids on the boundary. As you can see, the grid now does seem periodic, but bogus data seems nevertheless to be copied into the level 1 cells.

  9. Frank Löffler
    • changed status to open
    • removed comment

    As long as something is wrong with the patch, it should probably not be up for review.

  10. Eloisa Bentivegna
    • removed comment

    I would also be interested in getting this to work. I've tried to reproduce the problem using a slight variant of ks-mclachlan.par (attached). It seems that the problem is not connected to the new patch, which as far as I understand seems correct, but to some other problem in Carpet which emerges when the refinements are touching the outer boundaries. In brief, some boundary points are poisoned starting from the first iteration.

  11. Erik Schnetter
    • removed comment

    I tried this parameter file, and I did not see any nans. How many processes do you use? Which points of which variables contain poison at what iteration?

    I do see a problem with the reduction weight that seems to be set up wrong.

  12. Erik Schnetter
    • removed comment

    I've attached a parameter file that uses cell centering instead of vertex centering. This is more natural for periodic boundaries, because upper and lower boundary can be treated in the same way. Can you try this parameter file?

  13. Eloisa Bentivegna
    • removed comment

    I see the same behavior with the CC parfile. I'm running with one process, one thread.

  14. Erik Schnetter
    • removed comment

    Eloisa, you mentioned poison. Poison is much easier to verify and debug than looking at png images. Can you tell me when and where you are seeing poison, and how many processes you use?

  15. Eloisa Bentivegna
    • removed comment

    Indeed, it is poison. I've now realized that the issue with periodic boundaries is not the same as the one that arises on actual physical boundaries, reported above. I'm now concentrating on the issue with periodic boundaries: the poison is on all the boundary points of refinement levels with rl>0. I'm looking into thorn Periodic.

  16. Ian Hinder
    • removed comment

    Specifically, it seems to be a problem with the Gammas, A, B, etc, (which are initialised using finite differencing) on the UPPER intersection of the symmetry and refinement boundary. This seems very similar to what happens with multipatch, where the intersection of the outer and interpatch (symmetry) boundary is not initialised. We said this was due to the order in which things happen.

    So here we are seeing problems with the intersection of refinement boundaries and either symmetry or outer boundaries (probably two separate issues), and with multipatch we see problems with the intersection of symmetry and outer boundaries, to complete the set!

  17. Erik Schnetter
    • removed comment

    I don't see poison nor nans in my runs. Can you attach an output file with that problem?

  18. Eloisa Bentivegna
    • removed comment

    Please find the 1D output of ml_gamma below. Please notice that the poison can occur both on the upper and on the lower faces; I've slightly modified the parameter file to show that.

  19. Eloisa Bentivegna
    • removed comment

    Erik, could you give me a location where I can find the Formaline tars for your setup? Maybe this is the fastest way to figure out what's different.

  20. Eloisa Bentivegna
    • removed comment

    I've looked into this in more depth and I've found at least two major issues that prevent the original setup from working correctly:

    1) Slab (used by Periodic for filling the symmetry boundaries) doesn't seem to work when moving data between two components of the grid that are owned by the same process (this is also stated in the documentation); as a result, when a refinement level is split into two components (as would have to be the case for one that overflows on one side of the domain and has a chuck appearing on the opposite face) and they are both handled by the same process, the boundaries are simply not filled. If these points existed when the initial data was set, they keep those values forever. If the refinement appears later, it will contain poison. This problem can be reproduced with ks-mclachlan.par, and I'll create a new ticket for Slab; 2) The original example also tries using true AMR. I'm quite unfamiliar on how this should be done, and would like to understand exactly at which point the level_mask needs to be set. With the current location (preregrid), the grid structure will be different depending on whether poisoning is used, which seems to point out that something is used before the user thorn has the chance to set it. Notice that the question is complicated by the fact that the level_mask is set from a derivative of a grid function, so that it will take a few operations (interior loop, syncs, boundary conditions, …) to fill the whole domain with the correct values. The ET list is probably a good place for this discussion.

    Regarding the original request of making Carpet aware of periodic boundaries, as far as I can test Erik's third patch works and closes this ticket.

  21. Erik Schnetter
    • changed status to resolved
    • removed comment

    ad 1: Yes, this is correct. This used to not be a problem (in 2003), and comes back to bite now. Apologies for not implementing a test for this -- all other symmetry thorns do have such a test. There is now a ticket for this, #694.

    ad 2: PreRegrid is the place. Let's discuss on the list.

    Closing the ticket since the original issue seems to be solved.

  22. Log in to comment