resubmit failed to run

Issue #2433 closed
Former user created an issue

recently everytime I was trying to restart from the checkpoint, I got error message like this

OPENMP DISPLAY ENVIRONMENT END
cactus_1108: /home/qwang/ET/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:70: CCTK_REAL8 CarpetLib::th::get_time(int, int, int) const: Assertion `tl >= 0 and tl < timelevels' failed.
cactus_1108: /home/qwang/ET/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:70: CCTK_REAL8 CarpetLib::th::get_time(int, int, int) const: Assertion `tl >= 0 and tl < timelevels' failed.
cactus_1108: /home/qwang/ET/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:70: CCTK_REAL8 CarpetLib::th::get_time(int, int, int) const: Assertion `tl >= 0 and tl < timelevels' failed.

Comments (8)

  1. Roland Haas

    Hello,

    if you are using a version of the ET from before ET_2020_05 then you most likely are seeing (a variant of) issue #380.

    My recollection is that there is a workaround in older versions (ET_2020_05 should not be affected) was to ensure that (before checkpointing, I am not sure if one can change these after the simulation starts) ensure that:

    ADMBase::metric_timelevels = 3
    

    is in your parfile.

    if the error persists, please attach the full out and err files as well as the parfile to this ticket.

  2. Former user Account Deleted

    Thanks!

    I checked my parameter file and I have this line:

    ADMBase::metric_timelevels = 3

    Sorry that I have deleted the old one and started a new one due to the disk storage. I could the err and out files once this one done (might need one day).

  3. Roland Haas

    Hello Qingwen Wang,

    I see. So it likely is not exactly this issue, though any other grid function that gets interpolated in time (due to eg Scalar or Basic output or interpolation in space) can trigger the same error message. Without more information it is impossible to say which variable it is. You may even have to find out where Carpet / CarpetLib call get_time and add some debug output to report which variables it tries to access (get_time no longer knows, though it may be useful to pass in this information just for more detailed error messages).

    Yours,
    Roland

  4. Roland Haas

    Things that come to mind would be:

    • ADMBase::shift_timelevels if you are using PunctureTracker
    • ADMBase::lapse_timelevels (possibly) if you are using AHFinderDirect
    • and pretty much anything in ioScalar_vars and ioBasic_vars

  5. Former user Account Deleted

    Hi Roland,

    Thanks for quick response!

    • ADMBase::shift_timelevels if you are using PunctureTracker
    • ADMBase::lapse_timelevels (possibly) if you are using AHFinderDirect
    • and pretty much anything in ioScalar_vars and ioBasic_vars

    I checked that I didn't use any of these. Just remembered that though I don't have the out and err files yet but I attached my parameter file if useful.

    Best Regards, Qingwen Wang

  6. Roland Haas

    I am afraid I cannot find any attachment in the ticket right now. The variables that I listed were only (common) examples. Any variables that has a number of timelevels not 1 or identical to the number of timelevels required by Carpet’s time prolongation (3 for 2nd order accurate prolongation in time) can trigger this.

  7. Log in to comment