Add four more parameters for carpet
Hi all,
we have added four more parameters to carpet
:
uniformly_split_interior_points_x
uniformly_split_interior_points_y
uniformly_split_interior_points_z
symmetrically_distribute_points_y
The first three parameters are used to distribute the grid points across processors uniformly according to interior points only.
The fourth parameter is used to distribute the (even number of) grid points across (even number of) processors with the first and second half being translational symmetric to each other in y-dir, when the grid points are not divisible by the number of processors in y-dir.
Attached please find the diff with the master
branch. Thanks!
Comments (16)
-
-
reporter Right. I will add 6 parameters and create a pull request then
-
Pull request is here: https://bitbucket.org/eschnett/carpet/pull-requests/50/add-six-pars
-
Please review.
-
reporter Hi Roland, I’m not sure how to review. Could you help me with more details? Thanks
-
There are some suggestions on the “How to review a patch” page of the wiki (somewhat hidden, since it is listed in the “maintainers” section of the front page only):
https://docs.einsteintoolkit.org/et-docs/How_to_Review_a_Patch
-
Please review.
-
reporter Hi @Roland Haas ,
I made some mistakes and ended up creating a new pull requesthttps://bitbucket.org/eschnett/carpet/pull-requests/51/carpet-add-par-to-split-interior-points
sorry for the troubles and huge delay.
I think I have modify the pull request according to you suggestions.
Could you please take a look? Thanks.
-
So I have trouble coming up with a test parfile that passes. This one:
ActiveThorns = " Boundary Carpet CarpetIOASCII CarpetLib CarpetRegrid2 CartGrid3D CoordBase IOUtil InitBase MoL SymBase " $ghost_size = 1 $boundary_size = 1 $num_levels = 1 IOASCII::out1D_vars = " grid::x " IO::out_dir = $parfile IO::out_fileinfo = "none" IO::parfile_write = no IOASCII::out1D_every = 1 IOASCII::out1D_d = no IOASCII::out1D_y = no IOASCII::out1D_z = no Carpet::max_refinement_levels = 2 Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 1 Carpet::prolongation_order_time = 1 Carpet::init_fill_timelevels = yes CartGrid3D::type = "CoordBase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = no CoordBase::xmin = -1. CoordBase::xmax = +1 - 0.125 CoordBase::ymin = -0.125 CoordBase::ymax = +0.125 CoordBase::zmin = -0.125 CoordBase::zmax = +0.125 CoordBase::dx = 0.125 CoordBase::dy = 0.125 CoordBase::dz = 0.125 driver::ghost_size = $ghost_size Cactus::cctk_itlast = 0 Carpet::domain_from_CoordBase = yes Carpet::split_interior_points_x = yes Carpet::processor_topology = "manual" Carpet::processor_topology_3d_x = 4 Carpet::processor_topology_3d_y = 1 Carpet::processor_topology_3d_z = 1 CoordBase::boundary_size_x_lower = $boundary_size CoordBase::boundary_size_y_lower = $boundary_size CoordBase::boundary_size_z_lower = $boundary_size CoordBase::boundary_size_x_upper = $boundary_size CoordBase::boundary_size_y_upper = $boundary_size CoordBase::boundary_size_z_upper = $boundary_size CarpetRegrid2::regrid_every = 0 CarpetRegrid2::verbose = yes CarpetRegrid2::num_centres = 1 CarpetRegrid2::num_levels_1 = $num_levels CarpetRegrid2::position_x_1 = 0.0 CarpetRegrid2::radius_1[1] = 0.2
fails with
assert
failures on cub (essentially). Namely (adding some debug code):ipos [0,0,0] nprocs_dir [4,1,1] cub [9,6,6] rub0 [32,6,6] ipos [1,0,0] nprocs_dir [4,1,1] cub [17,6,6] rub0 [32,6,6] ipos [2,0,0] nprocs_dir [4,1,1] cub [23,6,6] rub0 [32,6,6] ipos [3,0,0] nprocs_dir [4,1,1] cub [30,6,6] rub0 [32,6,6] cactus_sim: /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/Carpet/src/Recompose.cc:1246: void Carpet::SplitRegions_AsSpecified(const cGH*, std::vector<CarpetLib::region_t>&, std::vector<CarpetLib::region_t>&): Assertion `all(not(ipos == nprocs_dir - 1) or cub == rub0)' failed.
when run on the minimum number of MPI ranks (4) that is supports.
Could you provide a working parfile as a testcase?
-
reporter Hi @Roland Haas , here is a test case
ActiveThorns = " Boundary Carpet CarpetIOASCII CarpetLib # CarpetRegrid2 CartGrid3D CoordBase IOUtil InitBase MoL SymBase " $ghost_size = 1 $boundary_size = 1 $num_levels = 1 IOASCII::out1D_vars = " grid::x " IO::out_dir = $parfile IO::out_fileinfo = "none" IO::parfile_write = no IOASCII::out1D_every = 1 IOASCII::out1D_d = no IOASCII::out1D_y = no IOASCII::out1D_z = no Carpet::max_refinement_levels = 1 Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 1 Carpet::prolongation_order_time = 1 Carpet::init_fill_timelevels = yes CartGrid3D::type = "CoordBase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = no CoordBase::xmin = -1. CoordBase::xmax = +1 - 0.125 CoordBase::ymin = -0.125 CoordBase::ymax = +0.125 CoordBase::zmin = -0.125 CoordBase::zmax = +0.125 CoordBase::dx = 0.125 CoordBase::dy = 0.125 CoordBase::dz = 0.125 driver::ghost_size = $ghost_size Cactus::cctk_itlast = 0 Carpet::domain_from_CoordBase = yes #Carpet::split_interior_points_x = yes Carpet::processor_topology = "manual" Carpet::processor_topology_3d_x = 4 Carpet::processor_topology_3d_y = 1 Carpet::processor_topology_3d_z = 1 CoordBase::boundary_size_x_lower = $boundary_size CoordBase::boundary_size_y_lower = $boundary_size CoordBase::boundary_size_z_lower = $boundary_size CoordBase::boundary_size_x_upper = $boundary_size CoordBase::boundary_size_y_upper = $boundary_size CoordBase::boundary_size_z_upper = $boundary_size CoordBase::boundary_shiftout_x_lower = 1 CoordBase::boundary_shiftout_x_upper = 1 #CoordBase::boundary_shiftout_y_lower = 1 #CoordBase::boundary_shiftout_y_upper = 1 #CoordBase::boundary_shiftout_z_lower = 1 #CoordBase::boundary_shiftout_z_upper = 1 #CarpetRegrid2::regrid_every = 0 #CarpetRegrid2::verbose = yes #CarpetRegrid2::num_centres = 1 #CarpetRegrid2::num_levels_1 = $num_levels #CarpetRegrid2::position_x_1 = 0.0 #CarpetRegrid2::radius_1[1] = 0.2
I made the following changes:
- remove
CarpetRegrid2
, it seems that the new pars are not consistent with it, - set
Carpet::max_refinement_levels = 1
the new pars only can work with no refinement case for now, - add
CoordBase::boundary_shiftout_x_lower = 1
andCoordBase::boundary_shiftout_x_upper = 1
, for some reason only when I turn on these. The boundary ghost points are counted as global points. And these option are on inSphericalNR
.
After these changes, when
Carpet::split_interior_points_x = yes
, I gotipos = [0,0,0], nprocs_dir = [4,1,1], cub = [5,3,3], rub0 = [18,3,3] ipos = [1,0,0], nprocs_dir = [4,1,1], cub = [9,3,3], rub0 = [18,3,3] ipos = [2,0,0], nprocs_dir = [4,1,1], cub = [13,3,3], rub0 = [18,3,3] ipos = [3,0,0], nprocs_dir = [4,1,1], cub = [18,3,3], rub0 = [18,3,3]
when
Carpet::split_interior_points_x = no
, I gotipos = [0,0,0], nprocs_dir = [4,1,1], cub = [5,3,3], rub0 = [18,3,3] ipos = [1,0,0], nprocs_dir = [4,1,1], cub = [10,3,3], rub0 = [18,3,3] ipos = [2,0,0], nprocs_dir = [4,1,1], cub = [14,3,3], rub0 = [18,3,3] ipos = [3,0,0], nprocs_dir = [4,1,1], cub = [18,3,3], rub0 = [18,3,3]
basally, the distribution of
(clb, cub)
change inx
change from[(0,5), (5,9), (9,13), (13,18)]
to[(0,5), (5,10), (10,14), (14,18)]
- remove
-
-
assigned issue to
-
assigned issue to
-
Thank you. I think with those things should be good to go. With the addition of adding this parfile as a test case to Carpet. Shiftout is ok since without the shiftout the first boundary point is exactly on the boundary (eg at -1.0) and there are not
boundary_size
points outside of the nominal domain (butboundary_size - 1
). This is actually intentional since it helps with convergence to have the boundary point stay fixed.Weren’t there plans at one point for SphericalNR to support / use mesh refinement on top of the spherical coordinates? In that case one would have to update this code it would seem.
-
reporter Hi Roland, thanks. I will add the above test par to
Carpet/test
.We are thinking about add support of mesh refinement to SphericalNR at some point. But not relatively soon. When that happens, we will update this part to make it also works with mesh refinement. Thanks for pointing it out.
In order to get the size of boundary points, I have tried with the par
granularity_boundary
. It is always0
, even thought in the test par, we have setCoordBase::boundary_size_x_lower = $boundary_size
. Should we setgranularity_boundary
explicitly in the par file or useghost_width[i]
instead for the moment? -
reporter Hi Roland. I have update the pull request according to the discussion we had before. Could you please take a look again? thanks
-
-
- changed status to resolved
- Log in to comment
Please create a pull request.
Why only add these 4 parameters and not parameter for all xyz and uniform / symmetric combinations?