Add four more parameters for carpet

Issue #2601 resolved

Liwei Ji created an issue 2022-03-17

Hi all,

we have added four more parameters to carpet:

uniformly_split_interior_points_x
uniformly_split_interior_points_y
uniformly_split_interior_points_z
symmetrically_distribute_points_y

The first three parameters are used to distribute the grid points across processors uniformly according to interior points only.

The fourth parameter is used to distribute the (even number of) grid points across (even number of) processors with the first and second half being translational symmetric to each other in y-dir, when the grid points are not divisible by the number of processors in y-dir.

Attached please find the diff with the master branch. Thanks!

diff.out

Comments (16)

Roland Haas
Please create a pull request.

Why only add these 4 parameters and not parameter for all xyz and uniform / symmetric combinations?
- 2022-03-31T02:12:02+00:00
Liwei Ji reporter
Right. I will add 6 parameters and create a pull request then
- 2022-03-31T14:11:53+00:00
Roland Haas
Pull request is here: https://bitbucket.org/eschnett/carpet/pull-requests/50/add-six-pars
- 2022-04-03T16:01:30+00:00
Roland Haas
Please review.
- 2022-05-10T21:09:29+00:00
Liwei Ji reporter
Hi Roland, I’m not sure how to review. Could you help me with more details? Thanks
- 2022-05-10T23:43:31+00:00
Roland Haas
There are some suggestions on the “How to review a patch” page of the wiki (somewhat hidden, since it is listed in the “maintainers” section of the front page only):

https://docs.einsteintoolkit.org/et-docs/How_to_Review_a_Patch
- 2022-05-11T04:00:13+00:00
Roland Haas
Please review.
- 2022-06-15T13:48:13+00:00
Liwei Ji reporter
Hi @Roland Haas ,
I made some mistakes and ended up creating a new pull request

https://bitbucket.org/eschnett/carpet/pull-requests/51/carpet-add-par-to-split-interior-points

sorry for the troubles and huge delay.

I think I have modify the pull request according to you suggestions.

Could you please take a look? Thanks.
- 2022-06-16T11:51:59+00:00

Roland Haas

So I have trouble coming up with a test parfile that passes. This one:

ActiveThorns = "
        Boundary
        Carpet
        CarpetIOASCII
        CarpetLib
        CarpetRegrid2
        CartGrid3D
        CoordBase
        IOUtil
        InitBase
        MoL
        SymBase
"

$ghost_size = 1
$boundary_size = 1
$num_levels = 1

IOASCII::out1D_vars = "
        grid::x
"

IO::out_dir       = $parfile
IO::out_fileinfo  = "none"
IO::parfile_write = no

IOASCII::out1D_every = 1
IOASCII::out1D_d     = no
IOASCII::out1D_y     = no
IOASCII::out1D_z     = no

Carpet::max_refinement_levels    = 2
Carpet::use_buffer_zones         = yes
Carpet::prolongation_order_space =  1
Carpet::prolongation_order_time  =  1

Carpet::init_fill_timelevels = yes

CartGrid3D::type         = "CoordBase"
CartGrid3D::domain       = "full"
CartGrid3D::avoid_origin = no

CoordBase::xmin = -1.
CoordBase::xmax = +1 - 0.125

CoordBase::ymin = -0.125
CoordBase::ymax = +0.125
CoordBase::zmin = -0.125
CoordBase::zmax = +0.125
CoordBase::dx   =  0.125
CoordBase::dy   =  0.125
CoordBase::dz   =  0.125

driver::ghost_size  = $ghost_size
Cactus::cctk_itlast = 0

Carpet::domain_from_CoordBase = yes
Carpet::split_interior_points_x = yes
Carpet::processor_topology = "manual"
Carpet::processor_topology_3d_x = 4
Carpet::processor_topology_3d_y = 1
Carpet::processor_topology_3d_z = 1

CoordBase::boundary_size_x_lower = $boundary_size
CoordBase::boundary_size_y_lower = $boundary_size
CoordBase::boundary_size_z_lower = $boundary_size
CoordBase::boundary_size_x_upper = $boundary_size
CoordBase::boundary_size_y_upper = $boundary_size
CoordBase::boundary_size_z_upper = $boundary_size

CarpetRegrid2::regrid_every   = 0
CarpetRegrid2::verbose        = yes

CarpetRegrid2::num_centres  = 1
CarpetRegrid2::num_levels_1 = $num_levels
CarpetRegrid2::position_x_1 = 0.0
CarpetRegrid2::radius_1[1]  = 0.2

fails with assert failures on cub (essentially). Namely (adding some debug code):

ipos [0,0,0] nprocs_dir [4,1,1] cub [9,6,6] rub0 [32,6,6]
ipos [1,0,0] nprocs_dir [4,1,1] cub [17,6,6] rub0 [32,6,6]
ipos [2,0,0] nprocs_dir [4,1,1] cub [23,6,6] rub0 [32,6,6]
ipos [3,0,0] nprocs_dir [4,1,1] cub [30,6,6] rub0 [32,6,6]
cactus_sim: /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/Carpet/src/Recompose.cc:1246: void Carpet::SplitRegions_AsSpecified(const cGH*, std::vector<CarpetLib::region_t>&, std::vector<CarpetLib::region_t>&): Assertion `all(not(ipos == nprocs_dir - 1) or cub == rub0)' failed.

when run on the minimum number of MPI ranks (4) that is supports.

Could you provide a working parfile as a testcase?

2022-06-19T01:34:15+00:00

Liwei Ji reporter

Hi @Roland Haas , here is a test case

ActiveThorns = "
        Boundary
        Carpet
        CarpetIOASCII
        CarpetLib
        # CarpetRegrid2
        CartGrid3D
        CoordBase
        IOUtil
        InitBase
        MoL
        SymBase
"

$ghost_size = 1
$boundary_size = 1
$num_levels = 1

IOASCII::out1D_vars = "
        grid::x
"

IO::out_dir       = $parfile
IO::out_fileinfo  = "none"
IO::parfile_write = no

IOASCII::out1D_every = 1
IOASCII::out1D_d     = no
IOASCII::out1D_y     = no
IOASCII::out1D_z     = no

Carpet::max_refinement_levels    = 1
Carpet::use_buffer_zones         = yes
Carpet::prolongation_order_space =  1
Carpet::prolongation_order_time  =  1

Carpet::init_fill_timelevels = yes

CartGrid3D::type         = "CoordBase"
CartGrid3D::domain       = "full"
CartGrid3D::avoid_origin = no

CoordBase::xmin = -1.
CoordBase::xmax = +1 - 0.125

CoordBase::ymin = -0.125
CoordBase::ymax = +0.125
CoordBase::zmin = -0.125
CoordBase::zmax = +0.125
CoordBase::dx   =  0.125
CoordBase::dy   =  0.125
CoordBase::dz   =  0.125

driver::ghost_size  = $ghost_size
Cactus::cctk_itlast = 0

Carpet::domain_from_CoordBase = yes
#Carpet::split_interior_points_x = yes
Carpet::processor_topology = "manual"
Carpet::processor_topology_3d_x = 4
Carpet::processor_topology_3d_y = 1
Carpet::processor_topology_3d_z = 1

CoordBase::boundary_size_x_lower = $boundary_size
CoordBase::boundary_size_y_lower = $boundary_size
CoordBase::boundary_size_z_lower = $boundary_size
CoordBase::boundary_size_x_upper = $boundary_size
CoordBase::boundary_size_y_upper = $boundary_size
CoordBase::boundary_size_z_upper = $boundary_size
CoordBase::boundary_shiftout_x_lower  = 1
CoordBase::boundary_shiftout_x_upper  = 1
#CoordBase::boundary_shiftout_y_lower  = 1
#CoordBase::boundary_shiftout_y_upper  = 1
#CoordBase::boundary_shiftout_z_lower  = 1
#CoordBase::boundary_shiftout_z_upper  = 1

#CarpetRegrid2::regrid_every   = 0
#CarpetRegrid2::verbose        = yes

#CarpetRegrid2::num_centres  = 1
#CarpetRegrid2::num_levels_1 = $num_levels
#CarpetRegrid2::position_x_1 = 0.0
#CarpetRegrid2::radius_1[1]  = 0.2

I made the following changes:

remove CarpetRegrid2, it seems that the new pars are not consistent with it,
set Carpet::max_refinement_levels = 1 the new pars only can work with no refinement case for now,
add CoordBase::boundary_shiftout_x_lower = 1 and CoordBase::boundary_shiftout_x_upper = 1, for some reason only when I turn on these. The boundary ghost points are counted as global points. And these option are on in SphericalNR.

After these changes, when Carpet::split_interior_points_x = yes, I got

ipos = [0,0,0], nprocs_dir = [4,1,1], cub = [5,3,3], rub0 = [18,3,3]
ipos = [1,0,0], nprocs_dir = [4,1,1], cub = [9,3,3], rub0 = [18,3,3]
ipos = [2,0,0], nprocs_dir = [4,1,1], cub = [13,3,3], rub0 = [18,3,3]
ipos = [3,0,0], nprocs_dir = [4,1,1], cub = [18,3,3], rub0 = [18,3,3]

when Carpet::split_interior_points_x = no, I got

ipos = [0,0,0], nprocs_dir = [4,1,1], cub = [5,3,3], rub0 = [18,3,3]
ipos = [1,0,0], nprocs_dir = [4,1,1], cub = [10,3,3], rub0 = [18,3,3]
ipos = [2,0,0], nprocs_dir = [4,1,1], cub = [14,3,3], rub0 = [18,3,3]
ipos = [3,0,0], nprocs_dir = [4,1,1], cub = [18,3,3], rub0 = [18,3,3]

basally, the distribution of (clb, cub) change in x change from [(0,5), (5,9), (9,13), (13,18)] to [(0,5), (5,10), (10,14), (14,18)]

2022-06-20T12:23:24+00:00

Roland Haas
- assigned issue to
  
  Roland Haas
- 2022-07-14T13:16:42+00:00
Roland Haas
Thank you. I think with those things should be good to go. With the addition of adding this parfile as a test case to Carpet. Shiftout is ok since without the shiftout the first boundary point is exactly on the boundary (eg at -1.0) and there are not boundary_size points outside of the nominal domain (but boundary_size - 1). This is actually intentional since it helps with convergence to have the boundary point stay fixed.

Weren’t there plans at one point for SphericalNR to support / use mesh refinement on top of the spherical coordinates? In that case one would have to update this code it would seem.
- 2022-07-14T13:22:14+00:00
Liwei Ji reporter
Hi Roland, thanks. I will add the above test par to Carpet/test.

We are thinking about add support of mesh refinement to SphericalNR at some point. But not relatively soon. When that happens, we will update this part to make it also works with mesh refinement. Thanks for pointing it out.

‌

In order to get the size of boundary points, I have tried with the par granularity_boundary. It is always 0, even thought in the test par, we have set CoordBase::boundary_size_x_lower = $boundary_size. Should we set granularity_boundary explicitly in the par file or use ghost_width[i] instead for the moment?
- 2022-07-16T17:37:25+00:00
Liwei Ji reporter
Hi Roland. I have update the pull request according to the discussion we had before. Could you please take a look again? thanks
- 2022-07-24T15:24:00+00:00
Roland Haas
Applied as git hash 8d84af01 "Carpet: update comments for split interior" of carpet
- 2022-07-26T19:27:41+00:00
Roland Haas
- changed status to resolved
- 2022-07-26T19:27:51+00:00
Log in to comment

Assignee: Roland Haas

Type: enhancement

Priority: major

Status: resolved

Component: Carpet

Milestone: –

Version: development version

Votes: 0

Watchers: 1