The number of timelevels for the AHFinder mask needs to be set to something intelligent.

[Ian Hinder]

The number of timelevels for AHFinderdirect::ahmask has to be set to something intelligent. Setting it to some fixed number doesn't work, because the number of timelevels for a given time-integrator might be different and Carpet chokes in that situation. At the moment it's set to how-many timelevels ADMBase::metric has, but that's also not good because that might be 1, while ahmask should have more. We cannot couple AHFinderDirect to any evolution thorn or Carpet as well. We might have to introduce a new parameter, but I would like to avoid this if somehow possible (which is why I coupled it to the metric in the first place, hoping this would be ok, which is apparently not really the case).

Keyword:

[Roland Haas]

Ian, it is the same issue for initialization (tried to run qc0 yesterday and had the same problem). The same workaround could be applied, though it is more hairy to do so in this case since qc0 is an actual evolution. The root cause is init_3_timelevels which does a ScheduleTraverse (where, "CCTK_POSTSTEP", cctkGH); (in initialise_3tl_evolve_IIb). I found however that I would get this error even later on in the simulation. My guess would be that since AHFinderDirect looks for horizons every 128 iterations (and coarse_step = 256) it could be looking at points on refinement level 0 (r>64 maybe when looking for the surface with areal radius 50?).

[Frank Löffler]

So - would it be sufficient to change the parameter file, for the release?

[Erik Schnetter]

I don't think that changing the scheduling is the correct approach here. Something wants to interpolate ahmask, and the ahmask isn't set up to allow this. Either ahmask needs 3 time levels, or it must not be interpolated (or it must be interpolated with a different operator).

[Roland Haas]

I agree with Erik in that scheduling is not the solution (in fact it does not help anyway) for the apparent horizon finding during the actual evolution. There (for the run I ran) I reduced the finding frequency of AHFinderDirect to the coarsest time step (for all horizons, though maybe only the larger surfaces were required). This kind of corresponds to not interpolating ahmask.

Scheduling during ANALYSIS helps with Carpet::init_3_levels since Carpet calls CCTK_POSTSTEP during CCTK_INITIAL time when presumably cctk_iteration = 0 (but eg. cctk_time = -delta_t) so all of AHFinder's checks say "run me now". (I think from what I understand of how Carpet and the interpolation works.) I am not sure if I'd be happy with a full three-timelevel grid function just so that I can use it once during initial data. At least the my hydro runs tend to be starved for memory so I like avoiding extra grid functions (even though it is the cleanest solution to the problem). Maybe one coud make the number of timelevels a parameter?

One could avoid scheduling in ANALYSIS by using Carpet::fill_timelevels instead of init_3_levels.

[Erik Schnetter]

If the mask is only needed during initialisation, and you are tight on memory, consider deallocating the mask after it is no longer needed.

fill_timelevels is the wrong approach for QC0, since this is not a stationary configuration. This will lead to a very large error initialliy. In fact, even init_3_timelevels is not good enough since it is only second order accurate.

This is a vacuum run; why is the ahmask necessary here at all?

[Roland Haas]

AHFinder uses this grid function (which is zero outside of the horizon non-zero inside and in a buffer zone) to find out if it is using points from within the horizon. There is a also a noshrink option to AHFinder which prevents horizons from ever shrinking (and uses the mask to detect this). From looking at it (had to because this actually caused aborts for me at one point) this seems related to excision (expansion.cc line 928).

I don't think that the mask is required for anything else (in vacuum, it is very useful to have around for hydro).

The mask is used all the time not just during initialization so I cannot turn it off. I wanted to say that there are two issues: ahmask usage during evolution where you can set find_every = coarsest_step and the one during initialization where find_every will not help.

I am happy with whatever solution is decided upon. I just looked at AHFinder's schedule.ccl and it already allocates several timelevels based on a parameter: ADMBase::metric_timelevels So setting this to 3 should solve the issue in a nice way.

[Erik Schnetter]

I think ahmask is used only for excision, and since we don't use excision in our vacuum simulations we don't need it. Without excision, horizons can shrink, and there is no reason to disallow using points from inside the horizon.

AHFinderDirect has a few parameters called use_mask, set_mask_* etc., and these should all be false in the sample QC0 simulations. Apparently they are not.

In the short term, I would follow your suggestion and increase the number of time levels, since this seems safer.

In the long run, I would like to add a facility to completely disable this mask. In our vacuum simulations, the mask is set to all zeros anyway.

[Frank Löffler]

We now set the number of timelevels for the metric to three, which solves the problem for now. I change the description to match the problem which is not really solved.

[Ian Hinder]

No solution is forthcoming for this release - removing milestone.

[Erik Schnetter]

Is this still an issue?