init_3_timelevels produces nans on past timelevels of the initial slice

[Yosef Zlochower]

I have been running into an issue where I when I interpolate an evolved function I get nans even though there are no nans on the gridfunction itself. The nans appear to arise when the past timelevels (at iteration 0) are initialized by init_3_timelevels and occur near the buffer regions. As the evolution proceeds, these bad past timelevels are overwritten with seemingly good data.

The attached thorn demonstrates the issue by evolving the trivial system

dt_f = 1

In visit you can see nans on the “_p_p” timelevel at iteration 1 (not iteration 0??). The nans go away later in the evolution.

[Roland Haas]

I started to look into this. I certainly still happens, and can be reproduced with a slightly simplified partfile (2 reflevels, 1 mpi rank, euler instead of rk4 time stepping, 0 timesteps) and shows nan in the prolongation region of reflevel 1 on timelevel 1 (but not tl 2 it seems so really only when interpolation in time is required). I will poke around a bit more.

[Roland Haas]

simplified demo thorn

[Roland Haas]

I am actually not sure if this ever worked as advertised. The code in question is:

   initialise_3tl_flip_timelevels(cctkGH);
   initialise_3tl_evolve(cctkGH);
   initialise_3tl_evolve(cctkGH);
   // TODO: May want to restrict where possible (i.e. if the time
   // refinement factor is one)
   initialise_3tl_recycle(cctkGH);

which is bad because initialise_3tl_evolve steps all reflevel by one step, so that in step 1 when say reflevel 1 wants to do prolongation for the buffer zones there are only 2 valid timelevels on reflevel 0 (namley tl=0 and tl=1) and thus interpolation in time picks up nan from tl=2.

A fix is proposed in: https://bitbucket.org/eschnett/carpet/commits/7147f842569c37719d1c97d5f6ddd7e2587a7be4 essentially moving the reflevel loop outside of initialise_3tl_evolve. Still has issues with the restriction involved but I don’t think those can be avoided.

Comments would be welcome.

[Roland Haas]

Pull request is here: https://bitbucket.org/eschnett/carpet/pull-requests/49/carpet-evolve-level-by-level-in-init3tl

[Roland Haas]

Please review.

[Roland Haas]

@Yosef Zlochower any news on this?

[Roland Haas]

@Yosef Zlochower any news on this?

[Roland Haas]

@Yosef Zlochower says (private email): The fix works for me. I no longer get Nans in the output files.

[Roland Haas]

Applied as git hash cfb236a4 "Carpet: evolve level by level in init3tl" of carpet