Necessary prerequisites

To build Sunrise, you first need to get the following 3rd party libraries:

• The Blitz++ C++ matrix library, current version. NOTE that blitz has switched to using Git. Make sure you don't just update the cvs or mercurial versions, as those repos are no longer being used. Changset d4a8862b671a is confirmed to work. Later versions may cause problems. The v0.10 tar file may work, but this has not been verified. Changeset ab84372f3dce (the current tip as of 12/8/2014) is confirmed to work (with version 11 of the Intel compilers, at least). The previously recommended changeset caused a compilation error in the hilbert.h file of libPJutil.
• The cfitsio FITS library. (If you are using a Fedora distribution, it's available as a yum package. On MacOX, you can get it through fink.)
• The CCfits C++ FITS library, version 2.3 or higher. (Note that Fedora has a package but make sure it's the correct version as otherwise it likely won't work.)
• The Boost C++ utility library. (If you are using a Fedora distribution, it's available as a yum package. On MacOX, you can get it through fink. It takes a long time to build, so if you can find a prebuild version, you should really do so.) Note that Sunrise requires version 1.40.0 or later. Version 1.48.0 is confirmed to work. Later versions may cause problems.
• The Intel Threading Building Blocks library. If you are using a recent Intel compiler, you already have this, otherwise free versions can be downloaded from the TBB web site.
• If you want to be able to use MPI parallelism, then you need Open MPI or another MPI implementation. Note that the MPI library must support multithreaded MPI.
• If you want to be able to load Gadget HDF5 files, you need the HDF5 library and the HDF5 C++ library (libhdf5_cpp). (Again, you can get these with yum and fink.)
• If you want to use the GPU-accelerated version, you need to install the CUDA library and SDK from Nvidia. Sunrise should work with versions 3.2 and higher, but CUDA changes pretty rapidly so it's possible newer versions won't work. Note that you need a compute capability 1.3 or higher device, ie at least a GT200 card. See further notes on building with GPU support at the bottom.
• If you want to run the Sunrise unit tests, you need the UnitTest++ unit testing framework. (Test coverage is pretty small, mostly for the new octree functionality.)

Finally, Sunrise depends on Patrik's utility library. This is available via Git from bitbucket. This library needs the GNU "units" program, in addition to the libraries above. (Actually, it only needs the units.dat file, but the easiest way to get it is to just install the package. Note that as of v2.0, this file is no longer included, so you must download v1.88.)

Compiling Sunrise

Note that you need fairly recent versions of autotools for this to work. I use autoconf 2.61, automake 1.10 and libtool 1.5.24. If you have older and run into trouble, just download them from the gnu site yourself and build them, it takes a few minutes.

In this example, I've used $HOME as the install directory, so that stuff ends up in $HOME/include, $HOME/lib, etc. If you want a different directory, just change the prefix options to configure.

Also, if you are using a nonstandard compiler (ie intel icpc), remember to set the CXX environment variable before you start. Mixing compilers is a recipe for trouble, trust me...

Boost:

Download Boost from SourceForge. Cd to the directory where you put the distribution and do

./bootstrap.sh --without-libraries=python --with-toolset=intel-linux --prefix=$HOME --libdir=$HOME/lib/boost_1_48_0 --includedir=$HOME/include/boost_1_48_0
./b2 -j<n> --user-config=user-config.jam install

This will copy the Boost headers to $HOME/include/boost_1_48_0/boost and will build the libraries and put them in $HOME/lib/boost_1_48_0. You obviously need to set the toolset to whatever compiler you'll be using, see the "getting started" instructions "getting started" for more info or if you run into trouble. Also, <n> should be some reasonable number depending on how many cores your machine has, because building Boost in parallel greatly decreases the build time.

Note that you only need the --user-config option if you are building Boost.MPI. The user-config.jam file should contain "using mpi ;". If you run into problems, google around for building Boost.MPI on your particular platform. As an example below is the user-config.jam used to compile sunrise on NERSC Edison

using mpi : /opt/cray/craype/2.2.1/bin/CC : <find-shared-library>mpich ;
using intel-linux : : : <compileflags>-I/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/include/ <linkflags>-L/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib/ ; # You will need to comment the 'using intel-linux' on project-config.jam if you have it here

If you use the above commands, Boost will install its include files in a directory $PREFIX/include/boost-<version>/boost. For the files to be found without specifically adding that directory to the include directives, I suggest you do ln -s $PREFIX/include/boost-<version>/boost $PREFIX/include/boost.

Note that the current Sunrise trunk requires Boost 1.40.0 or later, and if you are using MPI (see below) you must make sure that the Boost.MPI library is included in your build. Version 1.48.0 is confirmed to work. Later versions may cause problems.

Blitz:

First clone the git repository. (Changesets here may refer to the old hg repo which is no longer used.) Changeset ab84372f3dce is confirmed to work. Later versions may cause problems. The v0.10 tar file may work, but this has not been verified.

NOTE If you are using Sunrise 4.0, you need to use a recent version of blitz (see above), which has switched to using Mercurial. Make sure you don't just update the cvs version, as that repo is no longer being used. Also note that this version of blitz is not backwards-compatible with Sunrise 3.03, and vice versa.

 autoreconf -fiv
 export CXX=icpc
 export CXXFLAGS='-g -O2 -pthread'
 export LDFLAGS=-pthread
 ./configure --prefix=$HOME --enable-threadsafe --disable-cxx-flags-preset --enable-serialization 
 make lib
 make install

If you get errors related to not finding the boost libraries try using --with-boost-libdir=$HOME/lib/boost_1_48_0/ (or wherever you installed boost)

The current version of blitz can use atomic data types from the Intel TBB library instead of mutex locking to make the array reference counts thread safe. If you have TBB installed, you can enable this with --with-tbb and get a slight performance increase.

cfitsio

Untar the files and do:

 export CFLAGS='-pthread -O3'
 export CC=icc
 ./configure --prefix=$HOME
 make
 make install

CCfits

Extract sources to a directory named CCfits (otherwise there will be trouble..)

 export CXXFLAGS='-pthread -O3'
 ./configure --prefix=$HOME --with-cfitsio=$HOME
 make
 make install

(Sometimes I've run into a weird error when linking the library with icpc. If it fails with something like _dso_locate in the message, and the link line has a bunch of gcc-3.3.3 or 3.4 .o files in it, libtool is confused and thinks that you are compiling with an old gcc. In that case you'll have to manually link by pasting the link line and removing all the .o files, and then copy the library which is in .libs/ to where you want it. Then copy the include files (all .h files and the file called just CCfits) to $PREFIX/include/CCfits/.)

TBB

You need a pretty recent version of TBB. If you are using a recent version of icpc (like v12) then this should be sufficient, but you have to find the path to the library and supply it when configuring Sunrise. Usually you can find it by looking at the path to the icpc executable, which will be something like /n/sw/intel/mkl/10.3.1.107/composerxe-2011.3.174/bin/intel64/icpc. In this case, you'll typically find the TBB library and headers at /n/sw/intel/mkl/10.3.1.107/composerxe-2011.3.174/include and lib.

If you aren't using the Intel compiler (or your version is too old -- if you get a compilation error that "concurrent_priority_queue" doesn't exist, then it's too old), you need to download and install it. Compiled versions for Mac and Linux are available at the TBB web site, they can just be downloaded and then pointed to. I don't have much experience with this since I usually use icpc, so you may have to do some research here.

MPI

The MPI functionality in Sunrise was developed with OpenMPI, though in principle any MPI implementation should work. The important thing is that the MPI library must support the MPI_THREAD_MULTIPLE mode. In the case of OpenMPI, this requires specifying this when compiling. Building MPI requires compiling in the appropriate support for whatever communication hardware your particular cluster has, so I suggest you ask your local sysadmin to do this for you.

HDF5

If you want HDF5 support and you can't get a package with the library, you'll have to build it, too. Don't forget to include C++ support.

./configure --enable-cxx --prefix=$HOME
make
make install

libPJutil:

First get libPJutil from bitbucket by doing git clone https://lutorm@bitbucket.org/lutorm/libpjutil.git. (It is usually fine to use the most recent version of this, as it doesn't change much.) Then go into this directory and do:

autoreconf -fiv
export CPPFLAGS=-I$HOME/include (or wherever you need it to point to find blitz; also use "-pthread" if you use it for blitz)
export CXXFLAGS="-O2 -pthread"
export LDFLAGS=-L$HOME/lib (or wherever)
./configure --prefix=$HOME  (if boost is in $HOME/include and $HOME/lib)
make
make install

If your boost install is in some funny place, you can also use --with-boost-libdir=xxx to help the compiler find the libraries.

This library now contains an embedded copy of the GNU "units" program for units conversion. For this to work, the path to the file "units.dat" needs to be defined in the environment variable UNITSFILE. On standard linux systems, this file is in /usr/share/, so in that case you would add

export UNITSFILE=/usr/share/units.dat

to your bash_profile. If you don't have GNU units installed, you need to download it and put this file somewhere. NB. As of v2.0, this file is no longer included, so you must download v1.88.

Sunrise:

Finally. Get sources from the repository. In general you should check out the current version, which is mentioned on the front page, not the trunk. The trunk may be unstable at any given time.

autoreconf -fiv

Then I usually make a little file called "build" that sets the config options because I have to redo it pretty often. It contains:

export CPPFLAGS="-I$HOME/include -I$HOME/include/libPJutil" (yes, you need to separately specify the libPJutil directory, sorry)
export CXXFLAGS="-ggdb  -O3"
export LDFLAGS="-L$HOME/lib"
export CXX=icpc
./configure --prefix=$HOME --with-boost=$HOME (optionally: --with-boost-thread=mt --with-boost-program-options=mt)

The arguments to the --with-boost-xxx options depends on your compiler and system. These days the default boost install seems to install the libraries with no specific suffix, in which case you should only need --with-boost=xxx.

If you get a configure error that says something cryptic like "Error: could not link against!", it's not finding the boost libraries. Look in the directory where the boost libraries are located and you'll find libboost-thread-<yaddayadda>-<.so-stuff>. The <yaddayadda> is what you specify here, it can be blank, or gcc/it depending on which compiler you used. The --with-boost option obviously should point to the directory that contains the boost include and lib directories. If you run into trouble, search the Sunrise discussions and also the boost mail lists.

One final note: By default, Sunrise uses a lot of assertions to make sure nothing goes awry. This takes extra time. If you are doing a bunch of big runs, once you have made sure that it will run correctly, you can add -DNDEBUG to your CXXFLAGS. This will remove all assertions and can make the code run up to twice as fast.

Finally, it's time to compile:

make
make install

success! (hopefully...) If you encounter problems, please let us know (and update the wiki).

TCmalloc

As most C++ programs, Sunrise allocates lots of very small objects on the heap. The standard system malloc/free calls are quite slow and do not scale well with large numbers of threads concurrently allocating, because it has to make use of a global mutex.

The TCmalloc library (Thread-caching malloc) implements a replacement malloc where each thread can allocate small objects without using any locks. Using this library results in a modest performance increase. The configure script searches for libtcmalloc_minimal and uses it if found, so just make sure the path to the library is in your LDFLAGS.

GPU acceleration

If you have a machine with a CUDA-enabled GPU with compute model 1.3 or later (GT200 and on), you can make use of it for the dust emission calculation with the option "--enable-cuda". This functionality is not extensively tested, but I believe it works on both CUDA version 3.2 and 4.0. You must also define the NVCXXFLAGS environment variable so that nvcc can find the location of the CUDA SDK. You also need to add the path to the SDK and CUDA libraries, so the build instructions then look like:

export CPPFLAGS="-I$HOME/include -I$HOME/include/libPJutil -I/usr/local/cuda/sdk/common/inc"
export CXXFLAGS="-ggdb  -O3"
export LDFLAGS="-L$HOME/lib -L/usr/local/cuda/sdk/lib -L/usr/local/cuda/lib"
export CXX=g++
export NVCXXFLAGS="-arch=sm_13 -I/usr/local/cuda/sdk/common/inc"
./configure --prefix=$HOME --with-boost-thread=mt --with-boost-program-options=mt --with-cuda

MPI support

To enable MPI support, use the --with-mpi option to configure. The configuration script uses the mpicxx command to figure out what the MPI compilation options are, so make sure that your mpicxx points to the same compiler you are compiling with.

If you get an error like

mpi_util.cc(260): error: identifier "partial_sum" is undefined

try this https://groups.google.com/forum/#!topic/sunrisemcrx/Y8M7_sNKglk

Arepo

For processing Arepo snapshots directly in the Voronoi mesh, you need to use --enable-arepo when configuring Sunrise and make sure the paths to libarepo.a and the header files are in your LDFLAGS and CPPFLAGS, respectively. You also need to make sure the libraries needed by Arepo (like HDF5, fftw and MPI) are present. For more hints about using Arepo, see RunningWithArepo.

Unit Tests

Sunrise has started to acquire some (long overdue) unit tests. These are implemented using the UnitTest++ framework. Once you've installed this (notes are on the web page), you enable it by using --enable-tests when running configure. Make sure the paths to the include and library files are in CPPFLAGS and LDFLAGS, respectively. The tests are run by doing 'make check` in the unittest directory.

Additional hints

The old recommendation to not use -O3 with icpc no longer applies. There was a bug in the Blitz library that caused this problem which I have since fixed. -O3 should now work fine, whether it makes things significantly faster is more debatable.