The number of timelevels for the AHFinder mask needs to be set to something intelligent.

Create issue
Issue #380 resolved
Ian Hinder created an issue

The number of timelevels for AHFinderdirect::ahmask has to be set to something intelligent. Setting it to some fixed number doesn't work, because the number of timelevels for a given time-integrator might be different and Carpet chokes in that situation. At the moment it's set to how-many timelevels ADMBase::metric has, but that's also not good because that might be 1, while ahmask should have more. We cannot couple AHFinderDirect to any evolution thorn or Carpet as well. We might have to introduce a new parameter, but I would like to avoid this if somehow possible (which is why I coupled it to the metric in the first place, hoping this would be ok, which is apparently not really the case).

Keyword:

Comments (17)

  1. Roland Haas
    • removed comment

    Ian, it is the same issue for initialization (tried to run qc0 yesterday and had the same problem). The same workaround could be applied, though it is more hairy to do so in this case since qc0 is an actual evolution. The root cause is init_3_timelevels which does a ScheduleTraverse (where, "CCTK_POSTSTEP", cctkGH); (in initialise_3tl_evolve_IIb). I found however that I would get this error even later on in the simulation. My guess would be that since AHFinderDirect looks for horizons every 128 iterations (and coarse_step = 256) it could be looking at points on refinement level 0 (r>64 maybe when looking for the surface with areal radius 50?).

  2. Erik Schnetter
    • removed comment

    I don't think that changing the scheduling is the correct approach here. Something wants to interpolate ahmask, and the ahmask isn't set up to allow this. Either ahmask needs 3 time levels, or it must not be interpolated (or it must be interpolated with a different operator).

  3. Roland Haas
    • removed comment

    I agree with Erik in that scheduling is not the solution (in fact it does not help anyway) for the apparent horizon finding during the actual evolution. There (for the run I ran) I reduced the finding frequency of AHFinderDirect to the coarsest time step (for all horizons, though maybe only the larger surfaces were required). This kind of corresponds to not interpolating ahmask.

    Scheduling during ANALYSIS helps with Carpet::init_3_levels since Carpet calls CCTK_POSTSTEP during CCTK_INITIAL time when presumably cctk_iteration = 0 (but eg. cctk_time = -delta_t) so all of AHFinder's checks say "run me now". (I think from what I understand of how Carpet and the interpolation works.) I am not sure if I'd be happy with a full three-timelevel grid function just so that I can use it once during initial data. At least the my hydro runs tend to be starved for memory so I like avoiding extra grid functions (even though it is the cleanest solution to the problem). Maybe one coud make the number of timelevels a parameter?

    One could avoid scheduling in ANALYSIS by using Carpet::fill_timelevels instead of init_3_levels.

  4. Erik Schnetter
    • removed comment

    If the mask is only needed during initialisation, and you are tight on memory, consider deallocating the mask after it is no longer needed.

    fill_timelevels is the wrong approach for QC0, since this is not a stationary configuration. This will lead to a very large error initialliy. In fact, even init_3_timelevels is not good enough since it is only second order accurate.

    This is a vacuum run; why is the ahmask necessary here at all?

  5. Roland Haas
    • removed comment

    AHFinder uses this grid function (which is zero outside of the horizon non-zero inside and in a buffer zone) to find out if it is using points from within the horizon. There is a also a noshrink option to AHFinder which prevents horizons from ever shrinking (and uses the mask to detect this). From looking at it (had to because this actually caused aborts for me at one point) this seems related to excision (expansion.cc line 928).

    I don't think that the mask is required for anything else (in vacuum, it is very useful to have around for hydro).

    The mask is used all the time not just during initialization so I cannot turn it off. I wanted to say that there are two issues: ahmask usage during evolution where you can set find_every = coarsest_step and the one during initialization where find_every will not help.

    I am happy with whatever solution is decided upon. I just looked at AHFinder's schedule.ccl and it already allocates several timelevels based on a parameter: ADMBase::metric_timelevels So setting this to 3 should solve the issue in a nice way.

  6. Erik Schnetter
    • removed comment

    I think ahmask is used only for excision, and since we don't use excision in our vacuum simulations we don't need it. Without excision, horizons can shrink, and there is no reason to disallow using points from inside the horizon.

    AHFinderDirect has a few parameters called use_mask, set_mask_* etc., and these should all be false in the sample QC0 simulations. Apparently they are not.

    In the short term, I would follow your suggestion and increase the number of time levels, since this seems safer.

    In the long run, I would like to add a facility to completely disable this mask. In our vacuum simulations, the mask is set to all zeros anyway.

  7. Ian Hinder reporter
    • removed milestone
    • removed comment

    No solution is forthcoming for this release - removing milestone.

  8. Roland Haas

    This still happens and can be triggered in qc0-mclachlan.par by changing

    diff --git a/par/qc0-mclachlan.par b/par/qc0-mclachlan.par
    index fbb6742..3d3e633 100644
    --- a/par/qc0-mclachlan.par
    +++ b/par/qc0-mclachlan.par
    @@ -114,9 +114,9 @@ CoordBase::zmin =    0.00
     CoordBase::xmax = +120.00
     CoordBase::ymax = +120.00
     CoordBase::zmax = +120.00
    -CoordBase::dx   =    2.00
    -CoordBase::dy   =    2.00
    -CoordBase::dz   =    2.00
    +CoordBase::dx   =    2.00*1.5
    +CoordBase::dy   =    2.00*1.5
    +CoordBase::dz   =    2.00*1.5
    
     CoordBase::boundary_size_x_lower     = 3
     CoordBase::boundary_size_y_lower     = 3
    @@ -262,7 +262,7 @@ ADMBase::initial_dtlapse = "zero"
     ADMBase::initial_dtshift = "zero"
    
     # needed for AHFinderDirect
    -ADMBase::metric_timelevels = 3
    +ADMBase::metric_timelevels = 1
    
     TwoPunctures::par_b          =  1.168642873
     TwoPunctures::par_m_plus     =  0.453
    @@ -270,6 +270,10 @@ TwoPunctures::par_m_minus    =  0.453
     TwoPunctures::par_P_plus [1] = +0.3331917498
     TwoPunctures::par_P_minus[1] = -0.3331917498
    
    +twopunctures::npoints_phi = 6
    +twopunctures::npoints_A = 6
    +twopunctures::npoints_B = 6
    +
     #TODO# TwoPunctures::grid_setup_method = "evaluation"
    
     TwoPunctures::TP_epsilon = 1.0e-2
    @@ -354,7 +358,7 @@ Multipole::l_max = 4
    
     ActiveThorns = "AHFinderDirect"
    
    -AHFinderDirect::find_every = 128
    +AHFinderDirect::find_every = 1 #28
    
     AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES = no
    

    which fails after 1 step with:

    ERROR from host fdea4908.spdns.org process 1
      while executing schedule bin CCTK_POSTSTEP, routine AHFinderDirect::AHFinderDirect_find_horizons
      in thorn CarpetInterp, file /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetInterp/src/interp.cc:1514:
      -> Grid function "AHFINDERDIRECT::ahmask" has only 1 active time levels on refinement level 1; this is not enough for time interpolation
    cactus_sim: /home/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.
    

    The “best” solution seems to me to actually change AHFinderDirect::ahmask to have TIMELEVELS=1 in interface.ccl and STORAGE: ahmask in schedule.ccl. This indicates to Carpet that the grid function in question is constant in time and thus it will not complain about it when trying to interpolate it in time. This is a lie of course but justified I think. Reasons for this:

    • the grid function is a mask that is 0 on the outside of horizons, 1 inside based on SpaceMask::emask and that one has only a single timelevel and is reset by AHFinderDirect in POSTREGRID. Before each horizon find ahmask is set based on emask.
    • both grid functions use prolongation=”none” tags thus there is no interpolation in space or time during prolongation for either one
    • AHFinderDirect does not find horizons in postregrid but does still use the last-found horizon to initialize emask which sets it to the last found horizons, it does not copy this over to ahmask leaving ahmask unitialized after a regrid (!) on the past timelevels that are not set before horizon finding happens. This will cause warnings of the type “one or more points on the trial horizon surface point is/are in an excised region“ be produced if mask_is_noshrink is set (which is the only case in which the result from interpolating ahmask is actually used for anything).
    • since with mask_is_noshrink the valid mask on future timelevels is always smaller than the valid mask on past ones, ignoring past ones during interpolation does not affect the check for inside / outside the valid region

    To demonstrate the “shrink” one can use the attached modified parameter file ahfinder-test.par

    The pull request https://bitbucket.org/einsteintoolkit/einsteinanalysis/pull-requests/10/rhaas-singlelevel/diff reverts changes setting the number of timelevels of ahmask to a value greater than 1, which avoids it becoming undefined after a regrid in past timelevels and adds explicit InterpNumTimelevels tags to use a single timelevel for interpolation.

  9. Log in to comment