Reg. HDF5 output of QuasiLocalMeasures variables

[Vaishak Prasad]

Greetings

I am currently facing an issue when attempting to output QuasiLocalMeasures variables in HDF5 (in a BBH simulation similar to the gallery one) files instead of ASCII. I only face this issue when I request any QuasiLocalMeasures variables to be output in IOHDF5.

‌

I am getting the same error message as in 005361 (http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .

I am attaching the parameter file, the backtrace and the out and err files from the run. Requesting your help.

(ETK messages archive num. 007565)

Thanking you in advance

Vaishak P

[Roland Haas]

Reproducer parfile.

[Roland Haas]

Summarizing my end of the conversation on mailing list (http://lists.einsteintoolkit.org/pipermail/users/2020-August/007565.html):

To make it fail I need more than 1 refinement level active. It fails on both 1 and 4 MPI ranks (that I have tested on).

Adding a bit of debug output:

std::cerr << "active1: " << active0 << " outputslab: " << outputslab << std::endl;

I get:

active1: bboxset<CCTK_INT4,3>(set<bbox>:{([12,12,12]:[136,136,136]:[4,4,4]/[3,3,3]:[34,34,34]/[32,32,32]/32768)},stride:[4,4,4],offset:[0,0,0]) outputslab: ([0,0,0]:[38,75,0]:[1,1,1]/[0,0,0]:[38,75,0]/[39,76,1]/2964)
cactus_sim: /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&<int, 3>::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.

which seems to indicate the issue is really that one should not try to use "active" for anything other than grid functions by adding a:

if (groupdata.grouptype == CCTK_GF)

before the block of code that I suggested be commented out ("active" only makes sense for grid functions, since it refers to [basically] the non-buffer points due to mesh refinement).

[Roland Haas]

Pull request is here: https://bitbucket.org/eschnett/carpet/pull-requests/38/carpetiohdf5-only-store-active-region-for

[Roland Haas]

Please review.

[Roland Haas]

@Vaishak Prasad Turns out that you can also use:

iohdf5::out_vars = "QuasiLocalMeasures::qlm_newman_penrose"

which handles output of 2D arrays fine (it just cannot create a 2D slice through a 3D grid function).

[Vaishak Prasad]

Hi,

iohdf5::out_vars = "QuasiLocalMeasures::qlm_newman_penrose"

Is not working. So are these:

‌

QuasiLocalMeasures::qlm_shapes
QuasiLocalMeasures::qlm_coordinates
QuasiLocalMeasures::qlm_tetrad_l
QuasiLocalMeasures::qlm_tetrad_n
QuasiLocalMeasures::qlm_tetrad_m
QuasiLocalMeasures::qlm_killing_vector
QuasiLocalMeasures::qlm_killed_twometric
QuasiLocalMeasures::qlm_invariant_coordinates
QuasiLocalMeasures::qlm_3determinant
QuasiLocalMeasures::qlm_weyl_scalars
QuasiLocalMeasures::qlm_ricci_scalars
QuasiLocalMeasures::qlm_twometric

‌

Out of these (qlm_tetrad_m, qlm_weyl_scalars and qlm_newman_penrose) are complex variable types.

‌

The run exists with signal 6 with the error:

terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

‌

Thanks

‌

‌

[Roland Haas]

I cannot reproduce that failure.

The attached file qlm-ks-outvars.par uses

IOhdf5::out_every = 1
iohdf5::out_vars = "
QuasiLocalMeasures::qlm_newman_penrose
QuasiLocalMeasures::qlm_shapes
QuasiLocalMeasures::qlm_coordinates
QuasiLocalMeasures::qlm_tetrad_l
QuasiLocalMeasures::qlm_tetrad_n
QuasiLocalMeasures::qlm_tetrad_m
QuasiLocalMeasures::qlm_killing_vector
QuasiLocalMeasures::qlm_killed_twometric
QuasiLocalMeasures::qlm_invariant_coordinates
QuasiLocalMeasures::qlm_3determinant
QuasiLocalMeasures::qlm_weyl_scalars
QuasiLocalMeasures::qlm_ricci_scalars
QuasiLocalMeasures::qlm_twometric
"

and runs fine for me. Namely:

mpirun -n 2 exe/cactus_sim qlm-ks-outvars.par

produces a file qlm-ks-outvars/qlm_npmu[0].h5 which contains a dataset:

QUASILOCALMEASURES::qlm_npmu[0]\ it=4\ tl=0 Dataset {76/76, 39/39}
    Attribute: carpet_mglevel scalar
        Type:      native int
        Data:  0
    Attribute: cctk_bbox {4}
        Type:      native int
        Data:  1, 1, 1, 1
    Attribute: cctk_nghostzones {2}
        Type:      native int
        Data:  0, 0
    Attribute: group_timelevel scalar
        Type:      native int
        Data:  0
    Attribute: iorigin {2}
        Type:      native int
        Data:  0, 0
    Attribute: level scalar
        Type:      native int
        Data:  0
    Attribute: name scalar
        Type:      32-byte null-terminated ASCII string
        Data:  "QUASILOCALMEASURES::qlm_npmu[0]"
    Attribute: time scalar
        Type:      native double
        Data:  0.05
    Attribute: timestep scalar
        Type:      native int
        Data:  4
    Location:  1:58684
    Links:     1
    Chunks:    {76, 39} 47424 bytes
    Storage:   47424 logical bytes, 19682 allocated bytes, 240.95% utilization
    Filter-0:  shuffle-2 OPT {16}
    Filter-1:  deflate-1 OPT {6}
    Type:      struct {
                   "real"             +0    native double
                   "imag"             +8    native double
               } 16 bytes

Can you try and modify the file qlm-ks-outvars.par to fail in the way you observe, so that there is a quick reproducer?

[Roland Haas]

Also, if you could include out and err file as well as backtrace for the failure, that would be helpful.

[Vaishak Prasad]

I am able to run the par file 'qlm-ks-outvars.par' as is. I will try to reprouce the error and report back to you.

[Vaishak Prasad]

I am attaching the backtrace, err, out, par files for the gallery run. I am still working on qlm-ks-outvars.

‌

[Vaishak Prasad]

I could not get qlm-ks-outvars to fail in the way the bbh case does. I am able to output QLM variables of any dimensionality.

I believe the bbh case is failing due to some other reason.

[Roland Haas]

Unless objected I will apply https://bitbucket.org/eschnett/carpet/pull-requests/38/carpetiohdf5-only-store-active-region-for after 2020-09-24.

[Roland Haas]

Applied as git hash c049b72e "CarpetIOHDF5: only store active region for grid functions in sliced output" of carpet