Test "Poisson equation" example

Create issue
Issue #2173 resolved
Roland Haas created an issue

Before each release, check that http://einsteintoolkit.org/gallery/poisson/index.html still works and produces correct output.

Keyword: None

Comments (51)

  1. Roland Haas reporter
    • removed comment

    I re-ran the example using current master code and got this graph for the error (terr value, plotted as line number vs. column 2):

    However looking at the values of the coefficient (or the solution in psi for that matter) reveals that not all is well:

    namely of the 5 Gaussian blobs mentioned in the parfile only 2 show up.

    Some digging reveals that this is b/c of a not-backwards compatible change to CT_Analytic in d07b160 - (tag: ET_2015_11_v0) Reduced number of terms of gaussian[] superposition in Analytic.m in the first version of the thorn in the toolkit.

    This is in principle ok since it never supported more than 2 Gaussian blobs while in the ET but we likely should

    1. remove the non-functional parameters from param.ccl
    2. regenerate the gallery example with the correct data
  2. anonymous
    • removed comment

    We agreed in the ET call today to: 1. remove the non-functional parameters from the thorn 1. regenerate the example data using the current thorn. Will compare that the result are comparable to what was produced by the original code when using the same number of Gaussians.

  3. Roland Haas reporter
    • removed comment

    In the call on 2018-0802 a suggestion was made to mabye re-instantiate the 5 Gaussians. param.ccl right now claims that 20 gaussians are supported, so supporting 5 is not actually fixing the "bug" but instead only making the gallery example work.

    I new pull request enabling 20 gaussians is here:

    I have quickly checked on runtime and find that run time of the generated routine is:

    number of Gaussians run time total run time (s)
    2 139 287
    5 305 582
    20 869 1030

    So basically the CT_Analytic function's runtime is scaling almost linearly with the number of Gaussians. Note that this is a version of the poisson.par file where the max. number of iterations in the ML scheme has been reduced to 2 from 200 so presumably in a real run of the parfile CT_Analytic would not take >50% of the runtime since it only runs once and not once per grid sweep.

    I will thus prepare and test a commit allowing up to 5 Gaussians which lets one run the gallery example as shown.

    Only relative numbers matter as the absolute numbers would change with processor speed etc.

  4. Roland Haas reporter
    • removed comment

    Commands to create tarball and upload to bitbucket:

    tar czf poisson-$(date +%Ym%d).tar.gz --exclude cactus-source --exclude SIMFACTORY --exclude 'checkpoint*.h5' --exclude 'ct_multilevel-err*.h5' --exclude 'ct_multilevel-residual*.h5' err.p?? poisson
    curl -u 'rhaas@illinois.edu' -X POST https://api.bitbucket.org/2.0/repositories/einsteintoolkit/www/downloads -F files=@"poisson-$(date +%Ym%d).tar.gz"

    removing some of the output files was required as the upload otherwise fails (for the 750MB full tarball).

  5. Roland Haas reporter
    • changed status to resolved
    • removed comment

    I pushed a change to support 5 amp[] values in git hash 2563eba "CT_Analytic: regenrate code" of ctthorns. This makes the result look like what the gallery shows. I have updated the gallery in git hash 71f3c7a "gallery/ns: update plot and data after re-run" of wwww.

  6. Matthew

    I obtained the above results from running the poisson gallery example, using the “terr_norm_eqn0.asc” and “ct_analytic-ct_testc0.file_*.h5” files respectively. Would you still like me to upload all the results onto bitbucket?

  7. helvi witek


    We uploaded the data of the latest test and the corresponding information on the website.

  8. Roland Haas reporter

    I updated the link on the Poisson gallery page to point to the newest tarball, and added Matthew's plot to the page as well.

    Marking as resolved.

  9. helvi witek

    The gallery example ”poisson” was last successfully tested on 11 October 2019. The data is uploaded and the website description updated.

  10. William Gabella

    Tested several times and once I used 1 node, 8 mpi tasks, and 1 cpu per task, with 48 GB it ran in 20-25 minutes. Did not use simfactory, so the output tarball reflects that. Removed big files and added some plot files, a python script to make the err.png graphic, a Readme, and left in the 11 Oct 2019 run script and err.png with _20191011 appended to them. The command file might be useful to other HPCs. Included our Slurm script and the Lmod script it uses to load modules.

    Used a more manual curl command to add to the Downloads directory, see Roland’s above,

    curl -u 'b.gabella@vanderbilt.edu' -X POST https://api.bitbucket.org/2.0/repositories/einsteintoolkit/www/downloads -F files=@"poisson_20200529.tar.gz"

    which is useful getting the output director from the HPC to the Bitbucket Downloads folder. Must have an account and permissions to do this.

    I used git to edit the index.html and to change out the err.png file:

    git clone https://bitbucket.org/einsteintoolkit/www/src/master/

    Find the /master/gallery/poisson folder and update the index.html and any graphics (at least the err.png) you change or add. Then just git after that, always with care,

    git status
    git add index.html
    git add err.png
    git commit -m "some comment"
    git push
    <with login>

    Then clear your browser history and update that Poisson gallery page


    and check your changes were taken.

    I found it useful to download my tarball with wget (my browser seems to gunzip it but leave the name XXX.tar.gz, but it is no longer gzipped. Checked it was right with

    wget https://bitbucket.org/einsteintoolkit/www/downloads/poisson_20200529.tar.gz

    and that downloaded a gzipped tarball. Go figure.

    Could not update the VisIt graphic, did not have access, will still try. At the top of this issue is a *.session file for VisIt that should be useful.

  11. William Gabella

    [Warning save your long entry before attaching any files…just lost a ton of text when I attached psi2.session. Ack!]

    Have been unable to install Visit either on my laptop or on our HPC cluster. Found a docker image for version 2.12.3 (current as of now is 3.1.2) at https://github.com/symerio/visit-docker . Cannot run docker on my laptop because Fedora 32 uses CGRoups V2 and currently runc from docker throws an error. I can run podman as

    podman run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix:rw symerio/visit

    or with more -v local directories mapped to container directories.

    From the HPC cluster, where we use singularity, I use the command

    singularity run docker://symerio/visit

    from within the Poisson data directory, the one containing the two plotted data files ct_analytic-ct_testc0.file_0.h5 and ct_multilevel-psi.file_0.h5 . Copy the psi2.session file, and run above command, and restore the session from File > Restore Session and select the psi2.session file. This should recreate the images with the sources in white and the slice through the Psi potential with color contours. With the new versions of the Toolkit, running this gallery example, I get an image that is indistinguishable to the eye from the previous one.

  12. William Gabella

    Re-ran Poisson example today and then did a diff of the *.asc files and they agree but maybe for the last digit. Run looks fine with new ET.

    Updated the *.png files and they look identical. Used Visit via singularity for psi.png and used the plotErr.py python script for err.png (had to rename it).

    Was able to git push the web pages but cannot do the curl command from the hpc:

    gw345/gabe :<esims/poissFreeze3 370 >curl -u 'b.gabella@vanderbilt.edu' -X POST https://api.bitbucket.org/2.0/repositories/einsteintoolkit/www/downloads -F files=@"poisson_20201119.tar.gz"
    Enter host password for user 'b.gabella@vanderbilt.edu':
    Because your Atlassian account admin requires single sign-on, you'll need to authenticate with an app password. Create an app password at https://bitbucket.org/account/admin/app-passwords

    Host password was NOT my user password. Thought this worked seamlessly last time:)

    Trying to upload the 300 MB poisson_20201119.tar.gz via the web page after I am logged into ET bitbucket.

  13. William Gabella

    Logging out and back on did not help.

    Tried also using the web page at the Downloads folder to “Add Files” and it takes a long time and then shows a “You must solve the captcha.” without every showing me a captcha.

  14. William Gabella

    Oops, forgot that psi_4x3.png was used by the gallery page. Just added that to poisson directory via git. Cropped the psi.png to be 1200x900 pixels.

  15. William Gabella

    Okay the “host password” using my userid and not my SSO email, worked today. Maybe something was buggered yesterday evening or this is the magic incantation. Curl command was

    curl -u 'wegabella' -X POST https://api.bitbucket.org/2.0/repositories/einsteintoolkit/www/downloads -F files=@"poisson_20201119.tar.gz"
    Enter host password for user 'wegabella':

    And I used the “app password” made with my Bitbucket, SSO account with full email “b.gabella@vanderbilt.edu” and selecting all permissions…maybe too much.

    In “Your profile and settings” > Personal Settings> App Passwords. Create one and in this case because of the difficulty I had I selected all permissions. It warns you but, you cannot look at the password after the one time it shows it to you, so copy it down.

  16. Miguel Gracia

    I have updated the parfile with the Cactus::presync_mode = "mixed-error" option.

    I ran the example using a machine configured with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64Gb of ram.

  17. Miguel Gracia

    I updated the website and uploaded the results. I ran the example using a machine configured with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64Gb of ram.

  18. Log in to comment