I had been using the Mayer version (2019_10) of the EinsteinToolkit to do some BNS simulations, but not until very recently did I realise that there is some discrepancy in the outputs between simulations run with different number of cores. The discrepancy starts to grow after quite a while and at some point the relative error could be sever tens of percents.
I have got mainly two set of output after running the same simulation with different number of cores: i) 8 / 12 / 32 / 36 ii) 16 / 17 / 24 / 48 within each set, the outputs agree well up to 10^(-10) digit for a long enough simulation time. I simply do not know which data should I trust...
And when I tried to use more cores to test (e.g. 64), the code crashed almost immediately with the following errors: (a bunch of) [ml=0 rl=0 c=6] The following grid structure consistency check failed: The communicated region must be contained in the active part of the domain comm <= domain_active and Cactus/configs/ET2/build/CarpetLib/dh.cc:2123: -> The grid structure is inconsistent. It is impossible to continue.
The relevant code for the grid settings are:
CoordBase::xmin = 0.00 CoordBase::ymin = -492.80 CoordBase::zmin = 0.00 CoordBase::xmax = +492.80 CoordBase::ymax = +492.80 CoordBase::zmax = +492.80 CoordBase::dx = 22.4 CoordBase::dy = 22.4 CoordBase::dz = 22.4 ... Carpet::max_refinement_levels = 8 Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 Carpet::refinement_centering = "vertex" ... CarpetRegrid2::num_levels_2 = 6 CarpetRegrid2::position_x_2 = -34.0 CarpetRegrid2::radius_2 =240.0 CarpetRegrid2::radius_2 =120.0 CarpetRegrid2::radius_2 = 52.0 CarpetRegrid2::radius_2 = 40.0 CarpetRegrid2::radius_2 = 30.0
Do anyone encounter this kind of problem before? I thought the rule of thumb is to use 2^N number of cores but it turns out that the output for using 16 or 32 cores are quite different though.. what could be the issue here? Please direct me if there is tickets concerning similar issue.. Any comments would also be appreciated.