- changed milestone to ET_2014_05
- removed comment
CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing
The old tests CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p, which are almost identical apart from running on 1 or 2 processes, are both failing with the error
----------------------------------------------------------------
Iteration Time | WAVEMOL::phi | *VEMOL::phit | *VEMOL::phix
| norm2 | norm2 | norm2
----------------------------------------------------------------
0 0.138 | -nan | -nan | -nan
WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org
(line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc):
-> Internal error: extrapolation in time. variable=WAVEMOL::phi time=0.20000000000000001 times=[0.40000000000000002,0.67500000000000004,0.27500000000000002]
WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org
(line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc):
-> Internal error: extrapolation in time. variable=WAVEMOL::phi time=0.20000000000000001 times=[0.40000000000000002,0.67500000000000004,0.27500000000000002]
This test is newly failing because the dependent thorn WaveMoL has only just been added to the toolkit. The tests had a parameter setting WaveMoL::num_timelevels = 3 which didn't correspond to any WaveMoL parameter, so previously the tests would crash earlier. I removed that parameter setting, and now the tests fail with this "extrapolation in time" error. I don't understand this failure. The times array seems to be non-monotonic [0.4, 0.675, 0.275] perhaps indicating that it might not have been initialised correctly? The coarse grid timestep should be 0.1*0.25 = 0.025, so I don't see where these times come from during iteration 1. The NaNs in the norms are probably poison, which probably wasn't enabled when this test was written, so nobody noticed.
Keyword:
Comments (8)
-
-
- removed comment
the incorrect timestep can possibly be due to the same issue that Eloisa had when initializing her cosmological simulations namely that cctk_time and cctk_delta_time are set to "generic" values (namely 0.0 and 1.0) early on in SetupGH and then set to the correct values once the Time thorn initializes itself.
Grid arrays are created before grid functions so possibly the fix that was made for grid functions does not cover the grid array case.
This is just a guess of course. Could be solidified by adding a printf for cctk_delta_time to SetupGH.cc/set_state and/or changing the initial delta_time to say 17.4 instead of 1.0.
Uhm, well. ok. WAVEMOL::phi is likely a grid function and not a grid array. Still my money would be on SetupGH.
-
- changed status to open
- assigned issue to
- removed comment
Peter is looking into this.
-
- marked as critical
- removed comment
Please not that this also currently prevents Pandora from running the tests; Pandora times out when running the test suite so that no test results are available.
-
- assigned issue to
- removed comment
I think I found the reason; init_three_timelevels seems to be at fault. I am now testing the new parameter file.
-
- changed status to resolved
- removed comment
The problem went away when I switched off init_three_timelevels.
-
- removed comment
While I agree that it is probably best to disable this for the interpolation test: Doesn't this hide a problem with init_3_timelevels?
-
- removed comment
Yes, there is probably something wrong with global interpolation that requires time interpolation and init_3_timelevels. Most likely, one should not interpolate in time while the init_3_timelevels algorithm is running, as it (by design) modifies the time levels. There may also be a more severe bug hiding.
- Log in to comment