1D HDF5 output nonfunctional with multipatch - std::out_of_range

Create issue
Issue #446 closed
Ian Hinder created an issue

I have tried to run the Llama/LlamaWaveToy/par/Kerr-Schild_Gaussian.par parameter file with the Mercurial version of Carpet. I get the error:

terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check

This does not happen with the Git version of Carpet.

The backtrace is

Backtrace from rank 0 pid 25881: 1. /lib64/libc.so.6(gsignal+0x35) [0x2b35dfd24265] 2. /lib64/libc.so.6(abort+0x110) [0x2b35dfd25d10] 3. gnu_cxx::verbose_terminate_handler()(/usr/lib64/libstdc++.so.6) 4. /usr/lib64/libstdc++.so.6 [0x2b35df69ddb6] 5. /usr/lib64/libstdc++.so.6 [0x2b35df69dde3] 6. /usr/lib64/libstdc++.so.6 [0x2b35df69deca] 7. std::throw_out_of_range(char const)(/usr/lib64/libstdc++.so.6) 8. CarpetIOHDF5::GetAllActive(dh const, gh const, int, int, bboxset<int, 3>&)(./SIMFACTORY/cactus_hg_datura) 9. CarpetIOHDF5::IOHDF5<1>::OutputDirection(_cGH const, int, std::string, std::string, vect<int, 1> const&, bool, bool)(./SIMFACTORY/cactus_h g_datura) a. CarpetIOHDF5::IOHDF5<1>::OutputVarAs(_cGH const, char const, char const)(./SIMFACTORY/cactus_hg_datura) b. CarpetIOHDF5::IOHDF5<1>::OutputGH(_cGH const)(./SIMFACTORY/cactus_hg_datura) c. Carpet::OutputGH(_cGH const)(./SIMFACTORY/cactus_hg_datura) d. ./SIMFACTORY/cactus_hg_datura [0x128ddeb] e. Carpet::Initialise(tFleshConfig)(./SIMFACTORY/cactus_hg_datura) f. ./SIMFACTORY/cactus_hg_datura(main+0x99) [0x4d5649] 10. /lib64/libc.so.6(libc_start_main+0xf4) [0x2b35dfd11994] 11. ./SIMFACTORY/cactus_hg_datura [0x4d5369]

Keyword:

Comments (12)

  1. anonymous
    • removed comment

    Ian: does only 1d HDF5 output fail or also 2d? (3d does not matter, it uses a different code)

  2. Ian Hinder reporter
    • removed comment

    Anonymous,

    2D also fails if I enable it:

    Backtrace from rank 0 pid 27706: 1. /lib64/libc.so.6(gsignal+0x35) [0x2ae9836dd265] 2. /lib64/libc.so.6(abort+0x110) [0x2ae9836ded10] 3. gnu_cxx::verbose_terminate_handler()(/usr/lib64/libstdc++.so.6) 4. /usr/lib64/libstdc++.so.6 [0x2ae983056db6] 5. /usr/lib64/libstdc++.so.6 [0x2ae983056de3] 6. /usr/lib64/libstdc++.so.6 [0x2ae983056eca] 7. std::throw_out_of_range(char const*)(/usr/lib64/libstdc++.so.6) 8. CarpetIOHDF5::GetAllActive(dh const*, gh const*, int, int, bboxset<int, 3>&)(./SIMFACTORY/cactus_hg_datura) 9. CarpetIOHDF5::IOHDF5<2>::OutputDirection(_cGH const*, int, std::string, std::string, vect<int, 2> const&, bool, bool)(./SIMFACTORY/cactus_h g_datura) a. CarpetIOHDF5::IOHDF5<2>::OutputVarAs(_cGH const*, char const*, char const*)(./SIMFACTORY/cactus_hg_datura) b. CarpetIOHDF5::IOHDF5<2>::OutputGH(_cGH const*)(./SIMFACTORY/cactus_hg_datura) c. Carpet::OutputGH(_cGH const*)(./SIMFACTORY/cactus_hg_datura) d. ./SIMFACTORY/cactus_hg_datura [0x12927db] e. Carpet::Initialise(tFleshConfig*)(./SIMFACTORY/cactus_hg_datura) f. ./SIMFACTORY/cactus_hg_datura(main+0x99) [0x4d5679] 10. /lib64/libc.so.6(libc_start_main+0xf4) [0x2ae9836ca994] 11. ./SIMFACTORY/cactus_hg_datura [0x4d5399]

  3. Roland Haas
    • removed comment

    Replying to [comment:4 hinder]:

    This is for grid functions.

    hmm, then I have no idea what causes it (and am unable to reproduce it with for example the output_buffers.par file in CarpetIOHDF5/par). I can reproduce an out-of-range exception when I try to output grid arrays, eg. SphericalSurface::sf_radius. I fixed the bug responsible for that and pushed the change.

  4. Erik Schnetter
    • removed comment

    This part of Carpet, i.e. handling the grid structure, is where hg and git Carpet differ most, and also where there is the most development internally in Carpet.

    Could the routine GetAllActive be replaced by just accessing level_dboxes::active?

    Ian: Do you have a line number associated with this backtrace?

    This error usually indicates that a std::vector is accessed out of range. Apart from inlining, this would be the .at(rl) that may have been corrected by Rolands' patch.

  5. Ian Hinder reporter
    • removed comment

    Running under a debugger, I get

    1. 0 0x00007fff873615d6 in kill ()
    2. 1 0x00007fff87401cd6 in abort ()
    3. 2 0x00000001034ded95 in gnu_cxx::verbose_terminate_handler ()
    4. 3 0x00000001034dcc0a in cxxabiv1::terminate ()
    5. 4 0x00000001034dcc53 in std::terminate ()
    6. 5 0x00000001034dcd56 in cxa_throw ()
    7. 6 0x0000000103472786 in std::throw_out_of_range ()
    8. 7 0x00000001007b275e in CarpetIOHDF5::GetAllActive (dd=<value temporarily unavailable, due to optimizations>, hh=<value temporarily unavailable, due to optimizations>, ml=<value temporarily unavailable, due to optimizations>, rl=<value temporarily unavailable, due to optimizations>, allactive=<value temporarily unavailable, due to optimizations>) at stl_vector.h:634
    9. 8 0x00000001007ad93c in CarpetIOHDF5::IOHDF5<1>::OutputDirection (cctkGH=<value temporarily unavailable, due to optimizations>, vindex=<value temporarily unavailable, due to optimizations>, alias=<value temporarily unavailable, due to optimizations>, basefilename=<value temporarily unavailable, due to optimizations>, dirs=<value temporarily unavailable, due to optimizations>, is_new_file=<value temporarily unavailable, due to optimizations>, truncate_file=true) at OutputSlice.cc:564
    10. 9 0x00000001007aef7b in std::string::_M_rep () at /sw/lib/gcc4.4/include/c++/4.4.4/bits/basic_string.h:483
    11. 10 0x00000001007aef7b in CarpetIOHDF5::IOHDF5<1>::OutputVarAs (cctkGH=0x104a03880, varname=<value temporarily unavailable, due to optimizations>, alias=0x103b747e0 "u") at basic_string.h:503
    12. 11 0x00000001007af255 in CarpetIOHDF5::IOHDF5<1>::TriggerOutput (cctkGH=0x104a03880, vindex=285) at OutputSlice.cc:366
    13. 12 0x00000001007af32a in CarpetIOHDF5::IOHDF5<1>::OutputGH (cctkGH=0x104a03880) at OutputSlice.cc:242
    14. 13 0x0000000100c5c660 in std::vector<Carpet::Timer*, std::allocator<Carpet::Timer*> >::size () at /sw/lib/gcc4.4/include/c++/4.4.4/bits/stl_vector.h:54
    15. 14 0x0000000100c5c660 in std::vector<Carpet::Timer*, std::allocator<Carpet::Timer*> >::_M_range_check () at OutputGH.cc:633
    16. 15 std::vector<Carpet::Timer*, std::allocator<Carpet::Timer*> >::at () at /sw/lib/gcc4.4/include/c++/4.4.4/bits/stl_vector.h:652
    17. 16 0x0000000100c5c660 in Carpet::OutputGH (cctkGH=<value temporarily unavailable, due to optimizations>) at stl_vector.h:55
    18. 17 0x0000000100c5565d in CallAnalysis (cctkGH=0x104a03880) at Initialise.cc:1307
    19. 18 0x0000000100c59380 in Carpet::Initialise (fc=<value temporarily unavailable, due to optimizations>) at Initialise.cc:141
    20. 19 0x000000010001990d in main (argc=2, argv=0x7fff5fbfe760) at flesh.cc:80

    The OutputSlice.cc:564 line is the GetAllActive call here:

    re-compute the active (non-buffered) region ibset allactive; GetAllActive (dd, hh, m, rl, allactive);

  6. Erik Schnetter
    • removed comment

    The error occurs in GetAllActive. It would be good to get better line number information. Maybe you can use compiler options to provide these; otherwise, disabling inlining, disabling optimisation, or adding printf statements in GetAllActive may help. Maybe gcc would provide better debug output than Intel.

    What are the values of m, rl, and c that are passed into GetAllActive?

  7. Roland Haas
    • removed comment

    Hello Ian,

    The OutputSlice.cc:564 line is the GetAllActive call here:

    re-compute the active (non-buffered) region ibset allactive; GetAllActive (dd, hh, m, rl, allactive);

    Hmm, so this happens even with my initial patch it seems. GetAllActive's result is really only needed when you set output_buffer_points (or whatever it is actually called) to "no". This happens only for MultiPatch runs (and only for the "extra" non-refined patches)? If you need a quick fix so that you can actually run something (and don't need output_buffer_zones) then the attached patch disables GetAllActive when output_buffer_points="no" (which is the default). I put in the GetAllActive function (it's more or less a copy of a piece of dh::regrid). But I cannot reproduce your error with grid functions (ie. all of out[0123]D_var = "grid::x") works fine for me). Grid array output also seems to work at least for out[012]D_var = "sphericalsurface::sf_radius". This really seems to be a multipatch only issue.

  8. Log in to comment