1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

Inclusion of sgrid importer in Einstein Toolkit

Issue #2747 resolved
Samuel Cupp created an issue

SGRID is a code for generating binary neutron star data with arbitrary masses, spins and orbital eccentricities. This thorn provides an interface to import SGRID initial data into the Einstein Toolkit.

Comments (75)

  1. Michal Pirog 
      <div class="preview-container wiki-content"><!-- loaded via ajax --></div>
  <div class="mask"></div>
</div>

    </div> </form>

  3. Michal Pirog

    Hi there. I have a silly question. Can you explain to me where and how I can upload my files here to make it available to whoever is involved?

  4. Samuel Cupp reporter

    In the upper right of the ticket page, there should be a more option that drops down to give the attach file option. What exactly are you trying to upload?

  6. Samuel Cupp reporter

    Is the thorn not publicly available? There’s no way to include a thorn that isn’t public as far as I’m aware. A public repo needs to be set up asap in order for it to be able to be checked out by the toolkit. I didn’t realize this was all still private.

  7. Michal Pirog

    Publicly available repo is here:

    https://github.com/wofti/CactusSgrid.git

    The thorn itself is at the directory "DNSData" ("BNSData" is an old version)

    We are using the thornlist "dns.th" from the directory "useful_files" as an argument for "GetComponents".

    Finally, we compile Cactus with the option list "dns.cfg" from the directory "useful_files".

    PS: "README" is to update. And I need to prepare some .par file for testing. Again: where exactly should I upload this thorn?

  8. Roland Haas

    You don’t have to upload it anywhere, really. All that is needed for it to be “in the Einstien Toolkit” is that there is a correct checkout line of it in the official ET thornlist in the ET “manifest” repository (since, strictly speaking, this is all the ET is: a collated list of externally developed software).

    If you however would like to have the code hosted somewhere that is not e.g. your pesonal github or Bitbucket account (eg b/c you may change jobs, or b/c you would need to grant at least some access to the ET release chair / a maintainer) it can also be moved into an appropriate repository in https://github.com/einsteintoolkit e.g. the thorn could go into “EinsteinInitialdata”. Downside: you lose some “branding” since your name no longer appears in the URL, upside you do not have to eg handle access and the ET maintainers will take care of the all steps required for each release.

    Examples for “include” are eg the Outflow thorn in EinsteinAnalysis. Examples for “not include” are eg Carpet.

  9. Michal Pirog

    I was convinced that there is some specific place where files must be uploaded. Anyway, I'm still confused. There is a deadline today which says:

    "2023-09-27 (-10wk): all codes proposed for inclusion in master branch (still under review) [dates pushed back for delayed release]"

    What should I/we do to not miss this deadline.

  10. Samuel Cupp reporter

    This means adding the thorns to the einsteintoolkit.th in the master branch of the manifest (link). As long as the thorns are in a publicly accessible repo, it should be fine. Note that we usually make branches and tags for every single release on these repositories (see, for example, the plethora of ET_YYYY_MM branches in wvuthorns). This either means giving the ET maintainers permission to create branches in the repository (and possibly letting any future release managers/maintainers have access) or be willing to create these branches and tags on behalf of the ET. Generally, letting at least a few of the senior maintainers have commit access will help ensure smooth future releases in case people get sick or something at the same time as a release.

    So right now, the most important thing is making sure we put in the correct information. Based on what is in the linked git repo, it is

    !TARGET   = $ARR
!TYPE     = git
!URL      = https://github.com/wofti/CactusSgrid.git
!REPO_PATH = ..
!CHECKOUT =
CactusSgrid/DNSdata

    If that’s all we need to add, then I can do that today.

    @Roland Haas what does the $2 that most thorns use mean for repo_path? I ask because I don’t know if the .. in the above is what GetComponents normally expects or not.

  11. Michal Pirog

    First: I hope it's all we need to add, and I'll be more than happy if you do it. Thanks!

    Second: It's Wolfgang's repo, so I think we should ask him. I'm writing an email to him right now.

  12. Roland Haas

    GetComponents supports some very limited replacement variables in the URL, AUTH_URL and REPO_PATH variables, namely it will take the <ARRANGEMENT>/<THORN> entry of each thorn and will assign <ARRANGEMENT> to $1 (ie the first thingy) and $2 will be set to <THORN> (ie the second component). More or less like Perl with do when one used a pattern m!([^/]*)/(.*)!.

    There is also some magic in how REPO_PATH behaves depending on whether it does or does not contain a $1 or $2.

    If it does contain a $1 or $2 then the relative path inside of the repository will be exactly as given by REPO_PATH (after variable expansion). If on the other hand no $1 or $2 is found, then the relative path in the repo is <REPOS_PATH>/<ARRANGEMENT>/<THORN>.

    What needs to go into REPO_PATH thus hugely depends on the git repository layout. $2 often works nicely because people tend to put all thorns into the root of the git repo. .. is somewhat unusual.

  13. Samuel Cupp reporter

    When I try to compile, I get errors like

    /usr/bin/ld: /home/sam/grav/toolkit/tmp/Cactus/configs/sim/scratch/external/SGRID/lib/libsgrid.a(Coordinates.o): in function `xyz_of_AnsorgNS':
Coordinates.c:(.text+0xa48e): undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: Coordinates.c:(.text+0xa4dc): undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: Coordinates.c:(.text+0xac01): undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: /home/sam/grav/toolkit/tmp/Cactus/configs/sim/scratch/external/SGRID/lib/libsgrid.a(Coordinates.o): in function `dABphi_dxyz_AnsorgNS':
Coordinates.c:(.text+0xb8cc): undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: Coordinates.c:(.text+0xb91c): undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: Coordinates.c:(.text+0xd14c): undefined reference to `GOMP_critical_name_start'
/usr/bin/ld: /home/sam/grav/toolkit/tmp/Cactus/configs/sim/scratch/external/SGRID/lib/libsgrid.a(Coordinates.o): in function `xyz_of_AnsorgNS':
Coordinates.c:(.text+0xac9b): undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: /home/sam/grav/toolkit/tmp/Cactus/configs/sim/scratch/external/SGRID/lib/libsgrid.a(Coordinates.o): in function `dABphi_dxyz_AnsorgNS':
Coordinates.c:(.text+0xd27b): undefined reference to `GOMP_critical_name_end'
/usr/bin/ld: /home/sam/grav/toolkit/tmp/Cactus/configs/sim/scratch/external/SGRID/lib/libsgrid.a(DNSgrid.o): in function `Interp_Var_From_Grid1_To_Grid2_star._omp_fn.0':
DNSgrid.c:(.text+0x205): undefined reference to `omp_get_num_threads'
/usr/bin/ld: DNSgrid.c:(.text+0x20c): undefined reference to `omp_get_thread_num'

    Is ExternalLibraries/SGRID not properly hooking into the OMP in the toolkit?

  15. Samuel Cupp reporter

    That seemed to fix it. Thanks. I’ll merge the new thorns into master to see if it works in the test suite.

  16. Samuel Cupp reporter

    @Roland Haas Currently, DNSdata has a README, but no doc/documentation.tex. That means nothing will appear for it on the website, correct?

  20. Roland Haas

    @Liwei Ji did you take a look at the formal criteria when reviewing the SGRID importer and its associated code?

  22. Michal Pirog

    Samuel, Roland,

    I know I need to get that documentation.tex done, but I'm swamped with teaching right now. Can you give me a deadline to write it to keep this submission on track? I'm guessing it's ASAP, but do you need it by tonight, this weekend, or within the week?

  23. Samuel Cupp reporter

    The reviews are supposed to be done by next Wednesday. We can a little lenient about when the documentation is provided (CarpetX has already made a deal with me to get it in a little late). However, having at least one test parfile is strongly encouraged. This would go in DNSdata/test/ and runs automatically anytime the testsuite is run.

  24. Liwei Ji

    I have looked at the files in CactusSgrid/DNSdata and they look good to me (good style, comments and so on). Is there anything else I need to do (Sorry that I haven’t review any code before and have no experience). I think the only miss pieces are documentations and tests.

  25. Samuel Cupp reporter

    @Roland Haas what do other ID thorns do for tests? I know most just rely on evolution thorns which use them for tests, but this one obviously doesn’t have an existing test in any evolution thorns. I would like to see a test for this thorn, but if it needs a file to read I don’t know the right way to do that.

  26. Michal Pirog

    It is an initial data reader. I believe we need to provide a sample of the initial data to be read, interpolate onto the Cactus grid, and start the Cactus evolution. There are three files, weighing up to 80MB (in the lowest resolution version). My intention was to upload them to the "DNSdata/test/" directory. What do you think?

  27. Roland Haas

    right. TwoPunctures just generates some ID at very low resolution. The Meudon_Bin_Foo thorns do not have tests. FLRWSolver uses a special “no data file needed” setup for its test. CarpetIOHDF5 has some recover tests (and ID reader) in carpet/CarpetIOHDF5/test/input_initial_data.par and some data files for that (15k data). Same with einsteininitialdata/ReadInterpolate/test/synthetic.par (1MB data). 80MB is definitely too large. Ideally the files would <1MB (they don’t need to be physically sane, so very few modes and few should be sufficient).

  29. Roland Haas

    That would make it the largest file use by any test. 

    0.179688 MB ./carpet/CarpetIOHDF5/test/newsep/grid.coordinates.file_0.h5
0.179688 MB ./carpet/CarpetIOHDF5/test/newsep/grid.coordinates.xyz.file_0.h5
0.191406 MB ./pittnullcode/SphericalHarmonicReconGen/test/CceR0158m.h5
0.996094 MB ./einsteininitialdata/ReadInterpolate/test/synthetic_write/checkpoint.chkpt.it_0.h5
1.32812 MB ./carpet/CarpetIOHDF5/test/CarpetWaveToyCheckpoint_test.it_64.h5
1.39844 MB ./pittnullcode/SphericalHarmonicRecon/test/metric_test_case.h5
11.0977 MB ./einsteinanalysis/AHFinderDirect/test/checkpointML-EE/checkpoint.chkpt.it_1.h5

    which I would consider a doubtful distinction. As said the file does not actually have to be a solution to anything real. So setting up eg Minkowski space using a single domain is fine. Though something with some actually non-zero values and spatially varying would of course be nice (but linear basis functions would be enough).

    It does not have to evolve, just read in and eg write out 1d ASCII output.

  30. Michal Pirog

    There is a new directory DNSdata/test/ now. It contains minimal initial data for Einstein Toolkit testing. They are: "BNSdata_properties.txt", "checkpoint.0" (4.2M), and "test.par". All of them are necessary, but only "checkpoint.0" has a size in MB, the other two are in KB. The file "et.par" is a parfile for Cactus. It needs a path to this directory to be set as: DNSdata::sgrid_datadir = "path_to_the_directory/test". I'll appreciate your feedback here.

  31. Roland Haas

    the way to do it is to put a clever relative path in there that will work when Cactus runs the testsuite. Eg see repos/carpet/CarpetIOHDF5/test/input_initial_data.par which contains: 

    IO::filereader_ID_dir   = "../../../arrangements/Carpet/CarpetIOHDF5/test/input_initial_data"

    ‌

    which will not work if eg you cd into the test directory and run the parfile manually using cactus_sim foo.par (since the relative locations are different).

  32. Michal Pirog

    I believe it's done. At least I did it in the analogous way.

    BTW: "...when Cactus runs the testsuite...". Can anybody explain how it works, and what is to do in this matter?

  33. Roland Haas

    Cactus runs the testsuite for compiled configuration “sim” when you run:

    make sim-testsuite

    or use the --testsuite option of simfactory (really only useful for the half-yearly testing).

    They are also automatically run for each commit and results are shown on:

    https://einsteintoolkit.github.io/tests

    Right now it seems no SGRID tests are actually run. What happens if you do make sim-testsuite?

  34. Michal Pirog

    Apparently, I'm doing something fundamentally wrong because 

    make sim-testsuite

    does not take CactusSgrid into consideration. Extract from an output:

    ========================================================================

  Summary for configuration sim

    Time                     -> Thu Oct 12 22:09:09 EDT 2023
    Host                     -> koko-login001.hpc.fau.edu
    Processes                -> 2
    User                     -> mpirog

    Total available tests    -> 350
    Unrunnable tests         -> 15
    Runnable tests           -> 335
    Total number of thorns   -> 262
    Number of tested thorns  -> 99
    Number of tests passed   -> 325
    Number passed only to
               set tolerance -> 182
    Number failed            -> 10

  Tests passed:

    tov (from ADMMass)
    tov_carpet (from ADMMass)
    test_AHF_1 (from AHFinder)
    test_AHF_2 (from AHFinder)
    Kerr (from AHFinderDirect)
    Kerr-Cartoon (from AHFinderDirect)
    Kerr-Cartoon-EE (from AHFinderDirect)
    Kerr-EE (from AHFinderDirect)
    Kerr-definition-expansion (from AHFinderDirect)
    Kerr-definition-expansion-EE (from AHFinderDirect)
    Kerr-definition-expansion-product (from AHFinderDirect)
    Kerr-definition-expansion-product-EE (from AHFinderDirect)
    Kerr-definition-inner-expansion (from AHFinderDirect)
    Kerr-definition-inner-expansion-EE (from AHFinderDirect)
    Kerr-definition-mean-curvature (from AHFinderDirect)
    Kerr-definition-mean-curvature-EE (from AHFinderDirect)
    Kerr-modification-radius (from AHFinderDirect)
    Kerr-modification-radius-EE (from AHFinderDirect)
    Kerr-rotating-180 (from AHFinderDirect)
    Kerr-rotating-180-EE (from AHFinderDirect)
    Kerr-rotating-90 (from AHFinderDirect)
    Kerr-rotating-90-EE (from AHFinderDirect)
    Kerr-selection-areal-radius (from AHFinderDirect)
    Kerr-selection-areal-radius-EE (from AHFinderDirect)
    Kerr-selection-areal-radius-definition-expansion-product (from AHFinderDirect)
    Kerr-selection-areal-radius-definition-expansion-product-EE (from AHFinderDirect)
    Kerr-selection-mean-coordinate-radius (from AHFinderDirect)
    Kerr-selection-mean-coordinate-radius-EE (from AHFinderDirect)
    checkpointML-EE (from AHFinderDirect)
    misner1.2-025 (from AHFinderDirect)
    recoverML-EE (from AHFinderDirect)
    BaikalVacuum_EE_O8_sgw3d (from BaikalVacuum)
    BaikalVacuum_PreSync (from BaikalVacuum)
    boostedpuncture (from CT_MultiLevel)
    constraints_spherical (from CT_MultiLevel)
    poisson (from CT_MultiLevel)
    64k2 (from Carpet)
    kasner (from Carpet)
    kasner_amr (from Carpet)
    no-overlap (from Carpet)
    overlap (from Carpet)
    presync (from Carpet)
    test_restrict_sync (from Carpet)
    CarpetEvolutionMask_test (from CarpetEvolutionMask)
    CarpetEvolutionMask_test_off (from CarpetEvolutionMask)
    buffer_mask (from CarpetEvolutionMask)
    compact (from CarpetIOASCII)
    coords (from CarpetIOASCII)
    grid-coords (from CarpetIOASCII)
    newsep (from CarpetIOASCII)
    CarpetWaveToyNewRecover_test_1proc (from CarpetIOHDF5)
    CarpetWaveToyRecover_test_1proc (from CarpetIOHDF5)
    CarpetWaveToyRecover_test_2proc (from CarpetIOHDF5)
    newsep (from CarpetIOHDF5)
    nobuffers (from CarpetIOHDF5)
    presyncRecover (from CarpetIOHDF5)
    coords (from CarpetIOScalar)
    coords2 (from CarpetIOScalar)
    coords3 (from CarpetIOScalar)
    coords4 (from CarpetIOScalar)
    newsep (from CarpetIOScalar)
    carpetintegrate (from CarpetIntegrateTest)
    carpetintegrate-linear (from CarpetIntegrateTest)
    waveinterp-1p (from CarpetInterp)
    waveinterp-2p (from CarpetInterp)
    test_cc_horest_o3 (from CarpetProlongateTest)
    test_cc_horest_o5 (from CarpetProlongateTest)
    test_cc_o0 (from CarpetProlongateTest)
    test_cc_o1 (from CarpetProlongateTest)
    test_cc_o2 (from CarpetProlongateTest)
    test_cc_o3 (from CarpetProlongateTest)
    test_cc_o4 (from CarpetProlongateTest)
    test_cc_o5 (from CarpetProlongateTest)
    test_cc_rest_o0 (from CarpetProlongateTest)
    test_cc_rest_o1 (from CarpetProlongateTest)
    test_cc_rest_o2 (from CarpetProlongateTest)
    test_cc_rest_o3 (from CarpetProlongateTest)
    test_cc_rest_o4 (from CarpetProlongateTest)
    test_cc_rest_o5 (from CarpetProlongateTest)
    test_cc_tvd (from CarpetProlongateTest)
    test_cc_tvd_hi (from CarpetProlongateTest)
    test_cc_tvd_x (from CarpetProlongateTest)
    test_cc_tvd_y (from CarpetProlongateTest)
    test_cc_tvd_z (from CarpetProlongateTest)
    test_interp (from CarpetProlongateTest)
    test_o1 (from CarpetProlongateTest)
    test_o11 (from CarpetProlongateTest)
    test_o3 (from CarpetProlongateTest)
    test_o5 (from CarpetProlongateTest)
    test_o7 (from CarpetProlongateTest)
    test_o9 (from CarpetProlongateTest)
    nonstaggered (from CarpetReduce)
    nonstaggered_excl (from CarpetReduce)
    periodic_weight (from CarpetReduce)
    staggered (from CarpetReduce)
    regrid2_2D (from CarpetRegrid2)
    regrid2_2cen (from CarpetRegrid2)
    regrid2_2cen_minfrac (from CarpetRegrid2)
    regrid2_granularity (from CarpetRegrid2)
    carpet (from Cartoon2D)
    schw-0050 (from Cartoon2D)
    test_cartoon_2 (from Cartoon2D)
    test_cartoon_3 (from Cartoon2D)
    test7patch (from Coordinates)
    test_ah (from Dissipation)
    test_ob (from Dissipation)
    E2xeon_test_dbh (from DistortedBHIVP)
    test_dbh (from DistortedBHIVP)
    eos_omni (from EOS_Omni)
    GaugeWave (from EinsteinExact_Test)
    KS-tilted-EE (from EinsteinExact_Test)
    KerrSchild (from EinsteinExact_Test)
    Minkowski (from EinsteinExact_Test)
    ModifiedSchwarzschildBL (from EinsteinExact_Test)
    ShiftedGaugeWave (from EinsteinExact_Test)
    Vaidya2 (from EinsteinExact_Test)
    Schwarzschild_EF (from Exact)
    bowl-init (from Exact)
    de_Sitter (from Exact)
    qc0-mclachlan (from Extract)
    GRHydro_test_shock (from GRHydro)
    GRHydro_test_shock_hllc (from GRHydro)
    GRHydro_test_shock_mp5 (from GRHydro)
    GRHydro_test_shock_ppm (from GRHydro)
    GRHydro_test_shock_weno (from GRHydro)
    GRHydro_test_tov_ppm_ML (from GRHydro)
    GRHydro_test_tov_ppm_ML_disable_internal_excision (from GRHydro)
    GRHydro_test_tov_ppm_no_trp_ML (from GRHydro)
    balsara1_1d (from GRHydro)
    balsara1_1d_dc (from GRHydro)
    balsara1_2d (from GRHydro)
    balsara1_2d_dc (from GRHydro)
    balsara2_1d (from GRHydro)
    balsara2_2d (from GRHydro)
    balsara3_1d (from GRHydro)
    balsara3_2d (from GRHydro)
    balsara4_1d (from GRHydro)
    balsara4_2d (from GRHydro)
    balsara5_1d (from GRHydro)
    balsara5_2d (from GRHydro)
    cylexp_tvd_mc2_hlle (from GRHydro)
    cylexp_tvd_mc2_hlle_dc (from GRHydro)
    test_one_hybrid (from GRHydro)
    test_tov_carpet_refined_nosync (from GRHydro)
    tov_carpetevolutionmask (from GRHydro)
    tov_carpetevolutionmask2 (from GRHydro)
    tov_slowsector (from GRHydro)
    magtov_poloidal (from GRHydro_InitData)
    magtov_poloidal_poloidal_P_p_2 (from GRHydro_InitData)
    GiRaFFE_tests_AlfvenWave (from GiRaFFE)
    GiRaFFE_tests_AlignedRotator (from GiRaFFE)
    GiRaFFE_tests_DegenAlfvenWave (from GiRaFFE)
    GiRaFFE_tests_ExactWald (from GiRaFFE)
    GiRaFFE_tests_FFEBreak (from GiRaFFE)
    GiRaFFE_tests_FastWave (from GiRaFFE)
    GiRaFFE_tests_MagnetoWald (from GiRaFFE)
    GiRaFFE_tests_SplitMonopole (from GiRaFFE)
    GiRaFFE_tests_ThreeWave (from GiRaFFE)
    test_restrict_sync (from HighOrderWaveTest)
    tov_mass_test (from Hydro_Analysis)
    two_tov (from Hydro_Analysis)
    diag_flip_pugh_eno (from Hydro_InitExcision)
    diag_pugh_eno (from Hydro_InitExcision)
    sphere_pugh_ppm (from Hydro_InitExcision)
    x_flip_pugh_eno (from Hydro_InitExcision)
    x_pugh_eno (from Hydro_InitExcision)
    rnsA2 (from Hydro_RNSID)
    kerr (from IDAnalyticBH)
    test_bl (from IDAnalyticBH)
    test_misner (from IDAnalyticBH)
    test_axibrill (from IDAxiBrillBH)
    test_axibrill_nostagger (from IDAxiBrillBH)
    E2xeon_test_axioddbh (from IDAxiOddBrillBH)
    test_axioddbh (from IDAxiOddBrillBH)
    ConstraintViolate (from IDConstraintViolate)
    test_pw_ADM_dir-x (from IDLinearWaves)
    test_pw_ADM_dir-z (from IDLinearWaves)
    test_waveell (from IDScalarWaveElliptic)
    test_recover (from IOHDF5)
    wavetoy (from InterpToArray)
    wavetoy2 (from InterpToArray)
    LeanBSSN_BY (from LeanBSSNMoL)
    LeanBSSN_BY-spin (from LeanBSSNMoL)
    LeanBSSN_BY-spin-no-gauge (from LeanBSSNMoL)
    LeanBSSN_BY-spin-no-lapse (from LeanBSSNMoL)
    LeanBSSN_BY-spin-no-shift (from LeanBSSNMoL)
    LeanBSSN_schw (from LeanBSSNMoL)
    llamawavetoy_6patch (from LlamaWaveToy)
    llamawavetoy_7patch (from LlamaWaveToy)
    test-minkowski-carpet-1lev (from LoopControl)
    ML_ADMConstraints (from ML_BSSN_Test)
    ML_BSSN_EE_sgw3d (from ML_BSSN_Test)
    ML_BSSN_EE_sgw3d_rhs (from ML_BSSN_Test)
    ML_BSSN_MP_O8_bh (from ML_BSSN_Test)
    ML_BSSN_NewRad (from ML_BSSN_Test)
    ML_BSSN_O8_sgw3d (from ML_BSSN_Test)
    ML_BSSN_sgw3d (from ML_BSSN_Test)
    ML_BSSN_sgw3d_harmonic (from ML_BSSN_Test)
    ML_BSSN_sgw3d_harmonic_phi (from ML_BSSN_Test)
    ML_BSSN_sgw3d_rhs (from ML_BSSN_Test)
    ML_CCZ4_sgw3d (from ML_CCZ4_Test)
    ML_CCZ4_sgw3d_rhs (from ML_CCZ4_Test)
    gaussian-RK4 (from ML_WaveToy_Test)
    gaussian-RK45 (from ML_WaveToy_Test)
    gaussian-RK45-adaptive (from ML_WaveToy_Test)
    memspeed (from MemSpeed)
    test_22 (from Multipole)
    test_31 (from Multipole)
    test_44 (from Multipole)
    test_driscollhealy (from Multipole)
    test_midpoint (from Multipole)
    test_rads (from Multipole)
    test_simpson (from Multipole)
    test_trapezoidal (from Multipole)
    test_vars (from Multipole)
    teukolsky (from NPScalars)
    LeanBSSN_Ei_mu0.4_c0.05 (from NPScalars_Proca)
    nancount (from NaNChecker)
    moving_tov (from Outflow)
    papi (from PAPI)
    ml-gw1d-small (from PeriodicCarpet)
    ml-gw1d-small-amr (from PeriodicCarpet)
    testperiodicinterp (from PeriodicCarpet)
    LeanBSSN_Ei_mu0.4_c0.05 (from ProcaEvolve)
    ML_BSSN_Ei_mu0.4_c0.05 (from ProcaEvolve)
    LeanBSSN_Ei_mu0.4_c0.05 (from Proca_simpleID)
    qlm-bl (from QuasiLocalMeasures)
    qlm-ks (from QuasiLocalMeasures)
    qlm-ks-EE (from QuasiLocalMeasures)
    qlm-ks-boosted (from QuasiLocalMeasures)
    qlm-ks-shifted (from QuasiLocalMeasures)
    qlm-ks-tilted (from QuasiLocalMeasures)
    qlm-minkowski (from QuasiLocalMeasures)
    admhydro (from ReadInterpolate)
    synthetic (from ReadInterpolate)
    E2xeon_test_rdbh (from RotatingDBHIVP)
    test_rdbh (from RotatingDBHIVP)
    Kerr-EE (from RotatingSymmetry180)
    Kerr-rotating-180-EE (from RotatingSymmetry180)
    Kerr-rotating-180-staggered-EE (from RotatingSymmetry180)
    Kerr-staggered-EE (from RotatingSymmetry180)
    KerrSchild-rotating-180-EE (from RotatingSymmetry180)
    Kerr-rotating-90-EE (from RotatingSymmetry90)
    Kerr-rotating-90-staggered-EE (from RotatingSymmetry90)
    KerrSchild-rotating-90-EE (from RotatingSymmetry90)
    wavetoyc (from SampleBoundary)
    slabtest (from SlabTest)
    regression_test (from SphericalHarmonicRecon)
    SpEC-dat-test (from SphericalHarmonicReconGen)
    SpEC-h5-test (from SphericalHarmonicReconGen)
    test_one_boost_max (from TOVSolver)
    test_one_static_max (from TOVSolver)
    test_tov_carpet (from TOVSolver)
    test_two_av (from TOVSolver)
    test_two_max (from TOVSolver)
    file (from TerminationTrigger)
    signal (from TerminationTrigger)
    walltime (from TerminationTrigger)
    arrays (from TestArrays)
    arrays0 (from TestArrays)
    TestComplex (from TestComplex)
    TestComplexPow (from TestComplex)
    test_fpointer_null (from TestFpointerNULL)
    TestAvg (from TestGlobalReduce)
    TestAvg_dest0 (from TestGlobalReduce)
    TestMax (from TestGlobalReduce)
    TestMax_dest0 (from TestGlobalReduce)
    TestMin (from TestGlobalReduce)
    TestMin_dest0 (from TestGlobalReduce)
    TestScalar (from TestGlobalReduce)
    TestSum (from TestGlobalReduce)
    TestSum_dest0 (from TestGlobalReduce)
    TestWeightAvg (from TestGlobalReduce)
    TestWeightMax (from TestGlobalReduce)
    TestWeightMin (from TestGlobalReduce)
    TestWeightSum (from TestGlobalReduce)
    gaussianMax (from TestGlobalReduce)
    gaussianMax_dest0 (from TestGlobalReduce)
    gaussianSum (from TestGlobalReduce)
    gaussianSum_dest0 (from TestGlobalReduce)
    test_local_interp_2 (from TestLocalInterp2)
    TestAvg (from TestLocalReduce)
    TestMax (from TestLocalReduce)
    TestSum (from TestLocalReduce)
    testloop (from TestLoop)
    testloop (from TestLoopControl)
    ICN (from TestMoL)
    ICN-avg (from TestMoL)
    RK2 (from TestMoL)
    RK2-MR-2-1 (from TestMoL)
    RK2-central (from TestMoL)
    RK3 (from TestMoL)
    RK4 (from TestMoL)
    RK4-MR-2-1 (from TestMoL)
    RK4-RK2 (from TestMoL)
    RK45 (from TestMoL)
    RK45CK (from TestMoL)
    RK65 (from TestMoL)
    RK87 (from TestMoL)
    expressions (from TestPar)
    ReadWriteCarpet (from TestReadWrite)
    trigger (from Trigger)
    bhns_eval (from TwoPunctures)
    bhns_interp (from TwoPunctures)
    twopunctures (from TwoPunctures)
    twopunctures_carpet (from TwoPunctures)
    twopunctures_kerrproca_c11 (from TwoPunctures_KerrProca)
    vectors (from Vectors)
    test_binary_1 (from WaveBinarySource)
    gaussian (from WaveMoL)
    wavetoy1df77 (from WaveToy1DF77)
    test_WaveToy2D (from WaveToy2DF77)
    test_rad (from WaveToyC)
    wave_output (from WaveToyC)
    test_rad (from WaveToyCXX)
    test_custom (from WaveToyExtra)
    test_rad (from WaveToyF77)
    test_rob (from WaveToyF77)
    test_rad (from WaveToyF90)
    test_wavef90_flat (from WaveToyF90)
    test_wavef90_zero (from WaveToyF90)
    test_rad (from WaveToyFreeF90)
    teukolsky (from WeylScal4)
    teukolskyID (from WeylScal4)
    teukolskyParity (from WeylScal4)

  Tests failed:

    Baikal_PreSync (from Baikal)
    magnetizedTOV-Baikal (from Baikal)
    FishboneMoncrief_IllinoisGRMHD-LLR (from FishboneMoncriefID)
    large_Afield_BNS (from Seed_Magnetic_Fields)
    magnetized_explosionTOV (from Seed_Magnetic_Fields)
    magnetized_explosionTOV (from Seed_Magnetic_Fields_BNS)
    magnetized_explosionTOV (from VolumeIntegrals_GRMHD)
    magnetized_explosionTOV (from VolumeIntegrals_vacuum)
    magnetized_explosionTOV (from particle_tracerET)
    magnetized_explosionTOV (from smallbPoynET)

========================================================================
------------------------------------------------------------------------

    ‌

  35. Samuel Cupp reporter

    This test is trying to use NSTracker, which I don't think is part of the Einstein Toolkit. My guess is that its one of the unrunnable tests mentioned at the top. Can you just get rid of that thorn?

    Also, the parfile says its going to run to a final time of 2500, which (unless this runs very quickly) is incorrect for a test. This isn’t going to run long, if at all, and it should be a minimal code test, not a physics test. So the grid doesn’t have to be well resolved. You can almost certainly move the outer boundary closer, and maybe remove some refinement levels. I’d say the iterations should probably be less than 10, especially if you’re doing full GRMHD.

  36. Samuel Cupp reporter

    Additionally, the test shouldn’t try to output 2D or 3D data, since that’ll be way too much for the testsuite to compare against. I’d recommend removing all the extra wave extraction, 2D/3D output, TerminationTrigger, etc. Basically make it as lightweight as possible.

    To get the test to run successfully, you also need to have a directory DNSdata/test/CactusSgrid_test. The expected output for the test should live here.

  37. Samuel Cupp reporter

    If you want to keep the current parfile as an example for people, you could add DNSdata/par/ and put a production-level parfile example there.

  38. Michal Pirog

    I created the directory "CactusSgrid_test" as you suggested. Interestingly, when I change its name to, something else I'm getting feedback:

    "Parameter file CactusSgrid_test in thorn DNSdata but no output directory"

    It means that this test procedure is aware of the thorn and the test. However, regardless if the name of this directory is set properly or not, it is not tested. I don't see it on the list of tested thorns, and it's not in the directory "TEST". And the correct directory (CactusSgrid_test) is empty.

    Is it not tested, or it's failing the test? The feedback is:

        Time                     -> Tue Oct 17 10:40:38 EDT 2023
    Host                     -> koko-login003.hpc.fau.edu
    Processes                -> 2
    User                     -> mpirog

    Total available tests    -> 352
    Unrunnable tests         -> 16
    Runnable tests           -> 336
    Total number of thorns   -> 261
    Number of tested thorns  -> 100
    Number of tests passed   -> 336
    Number passed only to
               set tolerance -> 188
    Number failed            -> 0

  Tests passed:

    [here goes the list of the tests but it's not here]

    Is it among "Unrunnable tests"?

  40. Samuel Cupp reporter

    I uploaded a parfile that seems to work for me. DNSdata appears when I use make sim-testsuite and I can run the test. Note that I haven't let it finish, and I haven't checked to see if the requested grid hierarchy is small enough to be reasonable for a test. I changed the length of the simulation to iterations with only 2 iterations, and I got rid of NSTracker which isn't part of the ET. I also removed some other thorns like TerminationTrigger.
    This probably isn’t exactly right, but you can compare my changes with the original and see if you need anything I removed or if there’s anything else you can strip away.

  41. Michal Pirog

    I did it just without NSTracker. It helped. Now there is:

    Tests passed:

    [...]

    CactusSgrid_test (from DNSdata)

    [...]

  42. Samuel Cupp reporter

    Ok. Are there any expected output files? If the directory with a name matching the parfile is empty, I think it succeeds so long as the run finishes regardless of the data. @Roland Haas can correct me if that is wrong.

  43. Michal Pirog

    There is an output. I need to turn it off. And I need to prepare .par file and push it. And I'm super busy now because of teaching and I can take care about it tomorrow late evening/night. Will it be ok?

  46. Samuel Cupp reporter

    That works for me.

    Once the test is properly set up, it should be easy to tell if its working or not since the CI banner at the top of the website will be passing (green), pending (yellow), or failing (red).

  47. Roland Haas

    The test counts as “passed” if there are no output files provided and the code exits cleanly (exit code 0). having some output can of course be useful to diagnose what failed (and also that the numbers are correct). Eg if the ID reader puts the metric where the curvature is expected that likely still gives exit code 0 but is of course completely wrong.

  48. Michal Pirog

    I pushed a new .par file and the test works well. There is an issue to discuss. The thorn reads the Sgrid ID, does the job, and leaves the data stored to be used again (for the sake of efficiency, to not repeat this "job"). Is it ok?
    We can write it that way and remove this data immediately, but it is not going to change the fact, that before Cactus can take the data they have to be (temporarily or not) written.

  49. Samuel Cupp reporter

    The test is failing because if failed to find the BNS_properties.txt file. It looks like you hard-coded the path for the file to something on your machine. Can you try switching it to the ../../../arrangements/CactusSgrid/DNSdata/test/sgrid_id?

  51. Samuel Cupp reporter

    Oh, sorry. I didn’t see your message before commenting. I’m not sure I understand. Are you talking about the parfile in test/sgrid_id? How exactly does it interact with the test parfile? Do users normally have to run 2 different parfiles to use the thorn?

  52. Michal Pirog

    The complete set of ID contains three files:

    1. BNSdata_properties.txt - sgrid configuration summary
    2. checkpoint.0 - main file with ID
    3. sgrid_id.par - sgrid parfile

    All of them are required to be there. The only thing you need for Cactus to run (of for the test) is a path to the directory where all there are located. It is one line you mentioned before:

    DNSdata::sgrid_datadir           = "../../../arrangements/CactusSgrid/DNSdata/test/sgrid_id"

    The file 

    "../../../arrangements/CactusSgrid/DNSdata/test/sgrid_id/sgrid_id.par"

    is not Cactus parfile. It is sgrid parfile and simply must be there.

  54. Roland Haas

    It's in the Cactus flesh repo, repos/flesh/doc/latex/cactus.sty . However, when writing Cactus documentation in the doc directory of a thorn, usually you can put in (well you should not have to put in anything assuming you start from either the template generated by make newthorn or copy and pasted an existing documentation.tex file [and you are doing that aren’t you?]):

    \usepackage{../../../../doc/latex/cactus}

    The unusual case where this fails if when the directory structure does not match the expected one, usually b/c your repository structure is not REPO_ROOT/Thorn/doc/documentation.tex but eg instead REPO_ROOT/CactusCode/Thorn/doc/documentation.tex (eg done by RePrimAnd). Even in that case you should still put in the path shown above, even if this makes running:

    cd repos/foo/docs
pdflatex documentation.tex

    fail. Reason being that in fact you are not expected to run pdflatex yourself, but instead run make FOO-ThornDoc which will need that path (and will fail otherwise and thus the web-documtentaion will fail and your thorn won’t be seen be anyone).

  55. Michal Pirog

    OK, the thorn's documentation is seating at:

    ~/Cactus/repos/CactusSgrid/DNSdata/doc/documentation.tex

    The path is:

    \usepackage{../../../../doc/latex/cactus}

    I'm in the directory "Cactus" and I'm trying:

    A) 

    make DNSData-ThornDoc
Creating thorn documentations...
Created thorn documentations in doc/ThornDoc directory.
Done.

    B) 

    make CactusSgrid-ThornDoc
Creating thorn documentations...
Created thorn documentations in doc/ThornDoc directory.
Done.

    C)

    make CactusSgrid/DNSData-ThornDoc
Creating thorn documentations...
Created thorn documentations in doc/ThornDoc directory.
Done.

    There is no such thing like "ThornDoc" at "Cactus/doc". Also, 

    ~/Cactus/repos/CactusSgrid/DNSdata/doc/

    does not contain .pdf file. I'm looking for .pdf file to see the output of the text I have written.

    Any help?

  56. Roland Haas

    the documentation that make DNSData-ThornDoc creates (this being the correct command) is, as the command reports put into doc/ThornDoc. Though in your case, you have a typo (make is case sensitive): the thorn is DNSdata (lowercase “d”), not DNSData. So try: 

    make DNSdata-ThornDoc

    which will report: 

    Creating thorn documentations...
  Processing thorn CactusSgrid/DNSdata...
  Parsing ccl files...
    ERROR: Could not create documentation (check for latex errors)
  Created thorn documentations in doc/ThornDoc directory.
  Done.

    and you can find the PDF file in doc/ThornDoc/CactusSgrid/DNSdat/documentation.pdf (if it builds, which it does not). 

    ! LaTeX Error: Command \tableWidth already defined.
               Or name \end... illegal, see p.192 of the manual.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

l.74 ...length{\descWidth} \newlength{\tableWidth}
                                                   \newlength{\maxVarWidth} ...

    ‌

    I would very strongly (ie things will fail horribly if you don’t…) to not use extra packages and if really, really, really necessary use \newcommand and not \def to define new commands (also mind the comments in the template TeX file). Also read-the-docs: http://einsteintoolkit.org/usersguide/UsersGuide.html#x1-115000C1.8.4

    :-)

  57. Michal Pirog

    Though in your case, you have a typo (make is case sensitive): the thorn is DNSdata (lowercase “d”), not DNSData.

    I'm so embarrassed... It works... Thank you (for both, an advice and patience).

  58. Michal Pirog

    Now I know I need to prepare the documentation as soon as possible, obviously. What about the upcoming test? What is the current situation regarding it? Is there anything else I should be doing or preparing?

  60. Samuel Cupp reporter

    The testing involves people running the testsuite on various machines/clusters. So far, no one has said that the sgrid test is failing. I’ll ask Liwei to look over the documentation.

  62. Michal Pirog

    I have tried, of course. However, I'm not sure what to expect after this test. Could any of you take a look at it and give me your feedback?

  63. Samuel Cupp reporter

    Hi Michal. I was looking over your test. It runs fine on my machine, and we’ll mention here if any of the test runners experience a failure. One thing I would suggest is that you activate the scalar output. You can just set it to the last iteration (currently 10)

    IOScalar::outScalar_every      = 10 # every_coarse

    Then, the test will validate not just that the code runs, but also that the results match.

  64. Samuel Cupp reporter

    You’ll also have to add the output files to the DNSdata/test/DNSdata_test directory.

  66. Roland Haas

    The test fails on the test server: https://einsteintoolkit.github.io/tests/build_810.html

    The reason is that the test data includes data from SystemStatistics which includes eg the amount of free memory. See the diff output linked on the webpage: https://github.com/EinsteinToolkit/tests/blob/gh-pages/records/version_810/sim_810_1/DNSdata/DNSdata_test.diffs

    Tests should not include data that will change each time, see eg https://docs.einsteintoolkit.org/et-docs/Adding_a_test_case for how a test should be structured.

  68. Samuel Cupp reporter

    Sorry, I didn’t notice you were outputting a statistics file. You should also remove the SystemStatistics variable from the IO in the parfile so the output generated by the test matches what’s in the test directory.

  70. Samuel Cupp reporter

    Thanks! The only other changes I would like to suggest are
    1. there are several variables that are outputted but don’t have files in the test dir (HydroBase::w_lorentz, SphericalSurface::sf_radius, and ML_ADMConstraints::ML_Ham); I’d just remove these from the parfile, but if you want to check them please add the corresponding output files
    2. we aim to have the tests be as light-weight as possible; to that end, I’d suggest cutting down the number of iterations for the test to two or three scalar outputs
    two outputs would correspond to setting

    Cactus::terminate   = "iteration"
Cactus::cctk_itlast = 512

    Changing this also means replacing all the files in DNSdata/test/DNSdata_test with the new output.

    Thanks for working with us on these changes. If the test fails on any machines, we will let you know.

  71. Roland Haas

    I’d remove the output file and thorn SystemStatistics from the ActiveThorns line (to run with as few thorns as possible).

  72. Samuel Cupp reporter

    He already removed the SystemStatistics output, if that’s what you mean. But yeah, removing the thorn completely is a reasonable request.

  73. Michal Pirog

    I’d remove the output file and thorn SystemStatistics from the ActiveThorns line

    That is what I did. I’m working on the other suggestions now.

  76. Log in to comment
Assignee
Type
enhancement
Priority
major
Status
resolved
Component
EinsteinToolkit thorn
Milestone
ET_2023_11
Version
–
Votes
0
Watchers
4
Jira Software: the preferred issue tracker for Bitbucket. Join the team!