CarpetIOHDF5 incorrectly recovers grid scalars and arrays with DISTRIB=CONSTANT

Create issue
Issue #1609 open
Roland Haas created an issue

Due to the way that grid variables with DISTRIB=constant are handled, a recent change to Carpet (which is required to run on larger core counts) caused these variables to not be correctly recovered (the change was to add a new dimension rather than extend an existing one).

There were two things missing: grid scalars are handled as zero dimensional arrays in the flesh but as 1d arrays with 1 element in the driver, restoring variables (arrays or scalars) did not take the new way of handling DISTRIB=constant into account.

Since I have already committed fixes before and now have fixes for the fixes (sorry), it would be good if someone was to review them.

The typical failure mode is that grid scalars/arrays are not fully recovered which in the case of grid scalars is silent (since the whole variable is missing which we silently ignore) leaving them with undefined values.

Keyword:

Comments (7)

  1. Roland Haas reporter
    • removed comment

    A test case turned out not that straightforward since the incorrect data would only ever appear on processes other than process 0 so Carpet would never write it do disk. Also contrary to what I originally stated, grid SCALARS are indeed still read correctly (since special code for group.dim==0 was taken out and thus they actually behave as before), for grid arrays however the effect is present. Attached is a demo parfile cp.par. To demonstrate do:

    mpirun -n 2 cactus_bns_all -roe cp.par >cp.log
    mpirun -n 2 cactus_bns_all -roe cp.par >rec.log
    

    and notice that while rec.log lists

    INFO (CarpetIOHDF5):   reading 'CARPETREGRID2::radius[0]' from dataset 'CARPETREGRID2::radius[0] it=0 tl=0'
    

    this entry is missing from CCTK_Proc1.log.

    With the patches CCTK_Proc1.log als contains this line, ie the grid array is successfully read on process 1.

    I don't know how to make a test case that the test system would support out of this.

  2. Erik Schnetter
    • changed status to open
    • removed comment

    You could write a non-trivial value to the grid array, and after recovery, calculate both min and max via a local reduction, and output this via IOASCII.

  3. Roland Haas reporter
    • marked as
    • removed milestone
    • removed comment

    No, the bug has been fixed. The ticket was re-opened to provide a test case.

  4. Log in to comment