1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

Inclusion of kuibit

Issue #2538 resolved
Gabriele Bozzola created an issue

kuibit is a Python package for post-processing simulations. The tool comes with several features for analysis and visualization. For some of the most common operations (e.g., 2D plot of a grid variable, extraction of gravitational waves, ...), users can directly obtain the result without writing any code. The number of such ready-made scripts grows with every release of kuibit. For everything else, kuibit has a large number of features (https://sbozzolo.github.io/kuibit/features.html) that can be used in scripts or notebooks.

kuibit is designed to be user-friendly: it has rich documentation (https://sbozzolo.github.io/kuibit/) examples ready to be used, and tutorials. kuibit is also designed to be developer- and maintainer- friendly: it is thoroughly commented, and it implements several continuous integration pipelines to test, document, and publish the package automatically. The high quality of kuibit is also recognized by its publication in the Journal of Open Source Software (https://joss.theoj.org/papers/10.21105/joss.03099)

kuibit is largely inspired by Wolfgang Kastaun's PyCactus. kuibit shares the same overall design with PyCactus, and in some cases, the implementation details too. kuibit would not exist without PyCactus.

As far as I know, 5-10 people are actively using kuibit as their main tool to interface with the output of simulations. The feedback I got is overwhelmingly positive: users that are new to the Einstein Toolkit praise the simplicity of use and the comfort of working in Python; experienced users claim that kuibit allows them to focus on the science instead of the technical details.

kuibit has already been used for publications, see https://inspirehep.net/literature?sort=mostrecent&size=25&page=1&q=refersto%3Arecid%3A1858070.

kuibit repo: https://github.com/Sbozzolo/kuibit

Comments and feedback are welcomed.

Comments (28)

  6. Gabriele Bozzola reporter

    I gave two overviews on kuibit (first, second), but I have never given a more hands-on tutorial. If needed, I can give that too at any time.

  7. Gabriele Bozzola reporter

    How would kuibit be distributed with the Einstein Toolkit? Would users have to install it with pip? Would the repo be included?

    It would be nice to distribute the repo because they contain the examples, which, in my opinion, are the fastest way for new users to start visualizing their simulations. On the flip side, installing with pip kuibit is the easiest way to obtain a working copy of kuibit.

    ‌

    Also, I wrote a page (first steps) to introduce users to kuibit. I don’t know if it is useful or not, so @Yosef Zlochower , if you have no prior experience with kuibit, feedback on the on-boarding process would be appreciated!

  11. Gabriele Bozzola reporter

    Thanks for pointing out the broken link. It will be automatically fixed upon release of version 1.3.0 (version 1.2.1 didn’t go through the continuous delivery pipeline, so the example tarball was not generated).

    How did you download the example? If I look at the files in the archive for version 1.2.0, no one calls add_grid_strcture_to_parser (which was introduced later).

    ‌

  13. Gabriele Bozzola reporter

    In that case, you should get kuibit from git directly as well.

    pip install git+https://github.com/Sbozzolo/kuibit.gitshould work

  14. Gabriele Bozzola reporter

    After the call this morning, I took the following step:

    • Update the CI/CD pipeline. Now, when a release is tagged on GitHub, the documentation is archived to sbozzolo.github.io/kuibit/VERSION. Hence, it will be always possible to refer to the documentation for a specific stable version. sbozzolo.github.io/kuibit/ will point to the documentation generated from master. This will be the latest stable. sbozzolo.github.io/kuibit/dev will be the documentation from the next version.
    • Tagged a new release, 1.3.0. Hopefully, things should be working as expected now. It is on PyPi.

    If installed with pip, kuibit should bring all the dependencies with it EXCEPT:

    • numba (optially needed to compute mismatch between gravitational waves)
    • pycbc, lalsuite (optially needed to test mismatch between gravitational waves)
    • mayavi

    mayavi is used in some examples for 3D rendering. However, the project does not seem maintained anymore and it not really compatible with recent versions of Python/vtk. mayavi is a dependency in the next version kuibit, but this might change in the future.

    The following is an idea for distributing kuibit with the upcoming release of the Einstein Toolkit. If GetComponent can checkout a specific commit/tag, it can be used to download the kuibit repo for version 1.3.0. The main advantage of this is that users would have immediately all the examples on their machine. The examples are scripts that can be used to make plots and analysis without writing any code. Therefore, I think that they would be very useful for beginners. Then, users should install kuibit 1.3.0 with pip install --user kuibit==1.3.0. We can optionally recommend users to add CACTUS/kuibit/examples/bin to their PATH, so that making some plots becomes very easy. When a new version of kuibit is released, users can update it with pip and update the examples checking out the corresponding tag (if they wish to do so). The link to the documentation will be sbozzolo.github.io/kuibit/1.3.0

    ‌

  15. Yosef Zlochower

    Current version may not build if user hasn’t separately installed hdf5 build environment (h5py usually can install without this). Can we add examples showing how to make 2D figures like BNS movie in gallery?

  16. Roland Haas

    A bit late but here’s a pull request that installs parts of kuibit into utils/Analysis/kuibit (same location as POWER got installed to): https://bitbucket.org/einsteintoolkit/manifest/pull-requests/10/einsteintoolkitth-add-parts-of-kuibit-to

    I added kubit’s source code as well just so that is there (and in case users already have the prereqs installed eg on their laptops) but my recollection was that either:

    pip install git+https://github.com/Sbozzolo/kuibit.git@1.3.0

    or (eventually):

    pip install git+https://github.com/Sbozzolo/kuibit.git@ET_2021_11

    to have all prerequisites installed (note the @ sign to specify the tagged version to get).

    For the new user notebook we can provide a list of OS package manager packages to install to be able to use kuibit from the copy in the ET.

  17. Roland Haas

    @Gabriele Bozzola for inclusion in in the ET, at least one of the ET maintainers (http://einsteintoolkit.org/maintainers-credits.html) should have write permission to the repository. Is that already the case? In this case @Steven R. Brandt may be the best person to give access to since he is release mananger ie will have to create tags or ensure that they have been created otherwise.

  18. Gabriele Bozzola reporter

    @Gabriele Bozzola

    for inclusion in in the ET, at least one of the ET maintainers (http://einsteintoolkit.org/maintainers-credits.html) should have write permission to the repository. Is that already the case? In this case 

    @Steven R. Brandt

    may be the best person to give access to since he is release mananger ie will have to create tags or ensure that they have been created otherwise.

    You are Steven were invited to join the GitHub repo.

    Current version may not build if user hasn’t separately installed hdf5 build environment (h5py usually can install without this). 

    This comes from the fact that kuibit requires h5py < 3 at the moment. I am trying to relax this assumption, but the dependency solver goes crazy. I will have to fiddle around to find a way out of this. In case, I will push a version 1.3.1 to fix this.

    Can we add examples showing how to make 2D figures like BNS movie in gallery?

    I can help those that are running the gallery examples to make visualizations with kuibit. For example, you should be able to make the 2D movie we have in the gallery with kuibit without writing any single line of code:

    mopi -m examples/mopi_movies/grid_var --resolution 500 --plane xy --variable rho_b --colorbar --datadir simulation --interpolation-method bicubic --logscale --vmin -7 --vmax -1 --parallel --outdir movie -x0 -30 -30 -x1 30 30

    I added kubit’s source code as well just so that is there (and in case users already have the prereqs installed eg on their laptops) but my recollection was that either:

    Correct, it is not trivial to install kuibit directly from the repo (one needs `poetry`, and even then, it is still not simple). The best way is with pip. This also simplifies upgrading kuibit.

    Hence, we should provide instructions to download kuibit with pip and a link to the documentation relative to the version.

    For the new user notebook we can provide a list of OS package manager packages to install to be able to use kuibit from the copy in the ET.

    Unless I fail to work around the h5py problem, I think and hope that no extra package should be required, unless one wants to do 3D rendering with mayavi.

    ‌

    I can also give a tutorial on kuibit at some point.

  22. Roland Haas

    You can request up to one (not a hard rule but strongly encouraged) citations that people should use when using RePrimAnd, as well as multiple (any number, but you may provide indication under which circumstances which ones are suggested) suggested citations. Those will be listed on: https://www.einsteintoolkit.org/citation.html

    To provide the citation please provide bibtex entries to be included in einsteintoolkit.bib (in the manifest repo) that should look something like this:

    @Misc{Kranc:web,
  suggested-for ="McLachlan/ML_BSSN",
  key =          {Kranc},
  title =        {{Kranc}: {Kranc} Assembles Numerical Code},
  url =          {http://kranccode.org/},
}

@Misc{EinsteinToolkit:2021_05,
  requested-for ={EinsteinToolkit},
  author       = {Zachariah Etienne and Steven R. Brandt and Peter Diener and William E. Gabella and Miguel Gracia-Linares and Roland Haas and Atul Kedia and Miguel Alcubierre and Daniela Alic and Gabrielle Allen and Marcus Ansorg and Maria Babiuc-Hamilton and Luca Baiotti and Werner Benger and Eloisa Bentivegna and Sebastiano Bernuzzi and Tanja Bode and Gabriele Bozzola and Brockton Brendal and Bernd Bruegmann and Manuela Campanelli and Federico Cipolletta and Giovanni Corvino and Samuel Cupp and Roberto De Pietri and Harry Dimmelmeier and Rion Dooley and Nils Dorband and Matthew Elley and Yaakoub El Khamra and Joshua Faber and Toni Font and Joachim Frieben and Bruno Giacomazzo and Tom Goodale and Carsten Gundlach and Ian Hawke and Scott Hawley and Ian Hinder and E. A. Huerta and Sascha Husa and Sai Iyer and Daniel Johnson and Thorsten Kellermann and Andrew Knapp and Michael Koppitz and Pablo Laguna and Gerd Lanferman and Frank L{\"o}ffler and Joan Masso and Lars Menger and Andre Merzky and Jonah Maxwell Miller and Mark Miller and Philipp Moesta and Pedro Montero and Bruno Mundim and Andrea Nerozzi and Scott C. Noble and Christian Ott and Ravi Paruchuri and Denis Pollney and David Radice and Thomas Radke and Christian Reisswig and Luciano Rezzolla and David Rideout and Matei Ripeanu and Lorenzo Sala and Jascha A Schewtschenko and Erik Schnetter and Bernard Schutz and Ed Seidel and Eric Seidel and John Shalf and Ken Sible and Ulrich Sperhake and Nikolaos Stergioulas and Wai-Mo Suen and Bela Szilagyi and Ryoji Takahashi and Michael Thomas and Jonathan Thornburg and Malcolm Tobias and Aaryn Tonita and Paul Walker and Mew-Bing Wan and Barry Wardell and Leonardo Werneck and Helvi Witek and Miguel Zilh{\~a}o and Burkhard Zink and Yosef Zlochower},
  title        = {The Einstein Toolkit},
  month        = may,
  year         = 2021,
  note         = {To find out more, visit http://einsteintoolkit.org},
  publisher    = {Zenodo},
  version      = {The "Lorentz" release, ET\_2021\_05},
  doi          = {10.5281/zenodo.4884780},
  url          = {https://doi.org/10.5281/zenodo.4884780}
}

@article{Noble:2005gf,
      requested-for = {WVUThorns/IllinoisGRMHD},
      author         = "Noble, Scott C. and Gammie, Charles F. and McKinney,
                        Jonathan C. and Del Zanna, Luca",
      title          = "{Primitive variable solvers for conservative general
                        relativistic magnetohydrodynamics}",
      journal        = "Astrophys. J.",
      volume         = "641",
      pages          = "626-637",
      doi            = "10.1086/500349",
      year           = "2006",
      eprint         = "arXiv:astro-ph/0512420",
      SLACcitation   = "%%CITATION = ASTRO-PH/0512420;%%",
}

    the preferred entry format is SPIRES-like, in that SPIRES keys are strongly recommended but we list all authors. The top of einsteintoolkit.th explains the convention: https://bitbucket.org/einsteintoolkit/manifest/raw/master/einsteintoolkit.bib

  23. Gabriele Bozzola reporter

    You can use this citation key:

    @article{Bozzola:2021hus,
    requested-for = {kuibit},
    author = "Bozzola, Gabriele",
    title = "{kuibit: Analyzing Einstein Toolkit simulations with Python}",
    eprint = "2104.06376",
    archivePrefix = "arXiv",
    primaryClass = "gr-qc",
    doi = "10.21105/joss.03099",
    journal = "J. Open Source Softw.",
    volume = "6",
    number = "60",
    pages = "3099",
    year = "2021"
}

    kuibit relies upon numpy, scipy, and h5py

    @article{Harris:2020xlr,
 suggested-for = {kuibit},
 author        = {Charles R. Harris and K. Jarrod Millman and St{\'{e}}fan J.
                 van der Walt and Ralf Gommers and Pauli Virtanen and David
                 Cournapeau and Eric Wieser and Julian Taylor and Sebastian
                 Berg and Nathaniel J. Smith and Robert Kern and Matti Picus
                 and Stephan Hoyer and Marten H. van Kerkwijk and Matthew
                 Brett and Allan Haldane and Jaime Fern{\'{a}}ndez del
                 R{\'{i}}o and Mark Wiebe and Pearu Peterson and Pierre
                 G{\'{e}}rard-Marchant and Kevin Sheppard and Tyler Reddy and
                 Warren Weckesser and Hameer Abbasi and Christoph Gohlke and
                 Travis E. Oliphant},
    title = "{Array programming with NumPy}",
    eprint = "2006.10256",
    archivePrefix = "arXiv",
    primaryClass = "cs.MS",
    doi = "10.1038/s41586-020-2649-2",
    journal = "Nature",
    volume = "585",
    number = "7825",
    pages = "357--362",
    year = "2020"
}
@article{Virtanen:2019joe,
 suggested-for = {kuibit},
  author  = {Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and
            Haberland, Matt and Reddy, Tyler and Cournapeau, David and
            Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and
            Bright, Jonathan and {van der Walt}, St{\'e}fan J. and
            Brett, Matthew and Wilson, Joshua and Millman, K. Jarrod and
            Mayorov, Nikolay and Nelson, Andrew R. J. and Jones, Eric and
            Kern, Robert and Larson, Eric and Carey, C J and
            Polat, {\.I}lhan and Feng, Yu and Moore, Eric W. and
            {VanderPlas}, Jake and Laxalde, Denis and Perktold, Josef and
            Cimrman, Robert and Henriksen, Ian and Quintero, E. A. and
            Harris, Charles R. and Archibald, Anne M. and
            Ribeiro, Ant{\^o}nio H. and Pedregosa, Fabian and
            {van Mulbregt}, Paul and {SciPy 1.0 Contributors}}, 
    title = "{SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python}",
    eprint = "1907.10121",
    archivePrefix = "arXiv",
    primaryClass = "cs.MS",
    doi = "10.1038/s41592-019-0686-2",
    journal = "Nature Meth.",
    volume = "17",
    pages = "261",
    year = "2020"
}

    Unfortunately, h5py does not have a recommended citation.

    ‌

  24. Roland Haas

    Thank you. You can certainly add the scipy and numpy citations as suggested ones. If you do list citations for kuibit’s dependencies (typically not done, those dependenicies would have their own list) you may also consider the matplotlib citation (which they actually request, https://matplotlib.org/3.1.1/citing.html):

    @Article{Hunter:2007,
  Author    = {Hunter, J. D.},
  Title     = {Matplotlib: A 2D graphics environment},
  Journal   = {Computing in Science \& Engineering},
  Volume    = {9},
  Number    = {3},
  Pages     = {90--95},
  abstract  = {Matplotlib is a 2D graphics package used for Python for
  application development, interactive scripting, and publication-quality
  image generation across user interfaces and operating systems.},
  publisher = {IEEE COMPUTER SOC},
  doi       = {10.1109/MCSE.2007.55},
  year      = 2007
}

    though the entry itself already exists in einsteintoolkit.bib. Usually the suggested citations are secondary publications or publications that describe specific aspects that not all users will use.

    Independent of this, adding the numpy and scipy entries (with or without suggested-for) to einsteintoolkit.bib seems like a good idea. Thanks for looking them up.

  26. Gabriele Bozzola reporter

    Usually the suggested citations are secondary publications or publications that describe specific aspects that not all users will use.

    I see. Without numpy, scipy, and h5py, kuibit would not exist. Therefore, I personally always cite them, and I would recommend anyone that uses kuibit to do the same. (`matplotlib` is not as important in kuibit: you can use 95 % of the available features without it and use your preferred plotting library instead)

  29. Log in to comment
Assignee
Type
enhancement
Priority
major
Status
resolved
Component
–
Milestone
ET_2021_11
Version
development version
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!