CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing

Create issue
Issue #1497 closed
Ian Hinder created an issue

The old tests CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p, which are almost identical apart from running on 1 or 2 processes, are both failing with the error

----------------------------------------------------------------
Iteration      Time | WAVEMOL::phi | *VEMOL::phit | *VEMOL::phix
                    |        norm2 |        norm2 |        norm2
----------------------------------------------------------------
        0     0.138 |         -nan |         -nan |         -nan
WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org
  (line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc): 
  -> Internal error: extrapolation in time.  variable=WAVEMOL::phi  time=0.20000000000000001  times=[0.40000000000000002,0.67500000000000004,0.27500000000000002]
WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org
  (line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc): 
  -> Internal error: extrapolation in time.  variable=WAVEMOL::phi  time=0.20000000000000001  times=[0.40000000000000002,0.67500000000000004,0.27500000000000002]

This test is newly failing because the dependent thorn WaveMoL has only just been added to the toolkit. The tests had a parameter setting WaveMoL::num_timelevels = 3 which didn't correspond to any WaveMoL parameter, so previously the tests would crash earlier. I removed that parameter setting, and now the tests fail with this "extrapolation in time" error. I don't understand this failure. The times array seems to be non-monotonic [0.4, 0.675, 0.275] perhaps indicating that it might not have been initialised correctly? The coarse grid timestep should be 0.1*0.25 = 0.025, so I don't see where these times come from during iteration 1. The NaNs in the norms are probably poison, which probably wasn't enabled when this test was written, so nobody noticed.

Keyword:

Comments (9)

  1. Roland Haas
    • removed comment

    the incorrect timestep can possibly be due to the same issue that Eloisa had when initializing her cosmological simulations namely that cctk_time and cctk_delta_time are set to "generic" values (namely 0.0 and 1.0) early on in SetupGH and then set to the correct values once the Time thorn initializes itself.

    Grid arrays are created before grid functions so possibly the fix that was made for grid functions does not cover the grid array case.

    This is just a guess of course. Could be solidified by adding a printf for cctk_delta_time to SetupGH.cc/set_state and/or changing the initial delta_time to say 17.4 instead of 1.0.

    Uhm, well. ok. WAVEMOL::phi is likely a grid function and not a grid array. Still my money would be on SetupGH.

  2. Erik Schnetter

    Please not that this also currently prevents Pandora from running the tests; Pandora times out when running the test suite so that no test results are available.

  3. Erik Schnetter
    • assigned issue to
    • removed comment

    I think I found the reason; init_three_timelevels seems to be at fault. I am now testing the new parameter file.

  4. Frank Löffler
    • removed comment

    While I agree that it is probably best to disable this for the interpolation test: Doesn't this hide a problem with init_3_timelevels?

  5. Erik Schnetter
    • removed comment

    Yes, there is probably something wrong with global interpolation that requires time interpolation and init_3_timelevels. Most likely, one should not interpolate in time while the init_3_timelevels algorithm is running, as it (by design) modifies the time levels. There may also be a more severe bug hiding.

  6. Log in to comment