install should use GASNet "make install", or reference external GASNET

[Dan Bonachea]

The new install command does not work as advertised.

In particular, it fails to install some of the necessary UPCXX headers, and an incomplete, non-working subset of GASNet.

I have a strong preference that UPCXX install (from a GASNET build) use GASNet's actual install command, rather than recreating that logic with a broken crawl through GASNet's tree that will miss many of the "special" things that GASNet install does. In short, GASNet has production-quality and supported logic whose job it is to install a complete working GASNet in a requested directory, and it's not UPCXX's place to try and second-guess that logic, as that design would be fundamentally fragile.

In the case of a UPCXX install from an GASNET install tree, the UPCXX install should reference the GASNet install.

$ rm -Rf .nobs /home/bonachea/UPC/inst-upcxx ; export CC=gcc CXX=g++ DBGSYM=1 OPTLEV=0 GASNET= GASNET_CONDUIT=udp ; ./install /home/bonachea/UPC/inst-upcxx
Downloading http://gasnet.lbl.gov/EX/GASNet-2017.6.0.tar.gz
Finished    http://gasnet.lbl.gov/EX/GASNet-2017.6.0.tar.gz
Configuring GASNet...

<trim>

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -MM -MT x /home/bonachea/UPC/upcxx/src/upcxx.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -O0 -g -Wall -g3 -Wall -Wpointer-arith -Wwrite-strings -Wmissing-format-attribute -Wno-unused -Wno-unused-parameter -Wno-address -c /home/bonachea/UPC/upcxx/src/upcxx.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/5978a15f84e1431f654eb64c563c8947f661636e.upcxx.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -MM -MT x /home/bonachea/UPC/upcxx/src/future/core.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -MM -MT x /home/bonachea/UPC/upcxx/src/diagnostic.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -MM -MT x /home/bonachea/UPC/upcxx/src/packing.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -MM -MT x /home/bonachea/UPC/upcxx/src/digest.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -MM -MT x /home/bonachea/UPC/upcxx/src/backend/gasnet1_seq/backend.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -MM -MT x /home/bonachea/UPC/upcxx/src/dist_object.cpp

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -O0 -g -Wall -c /home/bonachea/UPC/upcxx/src/future/core.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/63e13ea65cd0cd907a438e490c665d5e0a078e21.core.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -O0 -g -Wall -c /home/bonachea/UPC/upcxx/src/diagnostic.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/c1087af8316d318c46ce433eb86efed7736a3ffc.diagnostic.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -O0 -g -Wall -c /home/bonachea/UPC/upcxx/src/digest.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/60aa323b45f7ca958d397e1184b8275d179feba5.digest.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -O0 -g -Wall -c /home/bonachea/UPC/upcxx/src/packing.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/40a6a1d562bb74f38144e490b146ca7e393fa6fe.packing.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -O0 -g -Wall -g3 -Wall -Wpointer-arith -Wwrite-strings -Wmissing-format-attribute -Wno-unused -Wno-unused-parameter -Wno-address -c /home/bonachea/UPC/upcxx/src/dist_object.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/6bfeabe72e1ad38514d50d20943dc5ca3d5f6097.dist_object.cpp.o

g++ -std=c++11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -DUPCXX_BACKEND=gasnet1_seq -D_GNU_SOURCE=1 -DGASNET_SEQ -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0 -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/udp-conduit -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356/other/amudp -I/home/bonachea/UPC/upcxx/.nobs/art/85505cc53026d03ee43b04bef3236863e751de24/GASNet-2017.6.0/extended-ref -I/home/bonachea/UPC/upcxx/.nobs/art/4a61e463b27fffebf32c7150c72584ae23644356 -O0 -g -Wall -g3 -Wall -Wpointer-arith -Wwrite-strings -Wmissing-format-attribute -Wno-unused -Wno-unused-parameter -Wno-address -c /home/bonachea/UPC/upcxx/src/backend/gasnet1_seq/backend.cpp -o /home/bonachea/UPC/upcxx/.nobs/art/69f0023d6bb0ed98d1c4e2b84675716653507f24.backend.cpp.o

gcc -std=c11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -MM -MT x /home/bonachea/UPC/upcxx/src/dl_malloc.c

gcc -std=c11 -D_GNU_SOURCE=1 -I/home/bonachea/UPC/upcxx/.nobs/art/b926136af1e701cb104698f164718b020cbfd2ea -O0 -g -Wall -c /home/bonachea/UPC/upcxx/src/dl_malloc.c -o /home/bonachea/UPC/upcxx/.nobs/art/ab80a47c12cbc45c59d227b713e03dbc1d2fb1f4.dl_malloc.c.o

ar rcs /home/bonachea/UPC/upcxx/.nobs/art/70ffa912468f2727afa11d88b59ad2d84d6ec42b/libupcxx.a /home/bonachea/UPC/upcxx/.nobs/art/60aa323b45f7ca958d397e1184b8275d179feba5.digest.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/c1087af8316d318c46ce433eb86efed7736a3ffc.diagnostic.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/63e13ea65cd0cd907a438e490c665d5e0a078e21.core.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/40a6a1d562bb74f38144e490b146ca7e393fa6fe.packing.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/5978a15f84e1431f654eb64c563c8947f661636e.upcxx.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/5978a15f84e1431f654eb64c563c8947f661636e.upcxx.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/6bfeabe72e1ad38514d50d20943dc5ca3d5f6097.dist_object.cpp.o /home/bonachea/UPC/upcxx/.nobs/art/ab80a47c12cbc45c59d227b713e03dbc1d2fb1f4.dl_malloc.c.o /home/bonachea/UPC/upcxx/.nobs/art/69f0023d6bb0ed98d1c4e2b84675716653507f24.backend.cpp.o


bonachea@Helios ~/UPC/upcxx
$ g++ -std=c++11 $(/home/bonachea/UPC/inst-upcxx/bin/upcxx-meta PPFLAGS) test/hello_upcxx.cpp $(/home/bonachea/UPC/inst-upcxx/bin/upcxx-meta LIBFLAGS)
test/hello_upcxx.cpp:1:29: fatal error: upcxx/backend.hpp: No such file or directory
compilation terminated.

bonachea@Helios ~/UPC/upcxx
$ ls -R /home/bonachea/UPC/inst-upcxx
/home/bonachea/UPC/inst-upcxx:
bin  include  lib

/home/bonachea/UPC/inst-upcxx/bin:
upcxx-meta

/home/bonachea/UPC/inst-upcxx/include:
gasnet.h           gasnet_asm.h          gasnet_atomicops.h  gasnet_coll.h              gasnet_help.h    gasnet_timer.h     gasnet_tools.h  gasnet_vis.h
gasnet_ammacros.h  gasnet_atomic_bits.h  gasnet_basic.h      gasnet_handler_internal.h  gasnet_membar.h  gasnet_toolhelp.h  gasnet_trace.h  gasnetex.h

/home/bonachea/UPC/inst-upcxx/lib:
libamudp.a  libgasnet-udp-seq.a  libupcxx.a

[Former user]

Completely agree a user supplied gasnet install tree should be used in place instead of what is done now. Will fix soon.

Using gasnets installer would be less fragile but a little more work and wouldn't change the user experience (modulo bugs).

The upcxx installer works fine on my Mac and Linux so I'll probably need access to this system. It looks like none of the upcxx headers are being copied.

[Dan Bonachea]

Using gasnets installer would be less fragile but a little more work and wouldn't change the user experience (modulo bugs).

I strongly disagree that it doesn't change the user experience.

A proper GASNet install includes MUCH more than headers and libraries for a single conduit, but also:

1. gasnetrun spawners which are absolutely required to use UPCXX programs in a production context. These are collections of shell and perl scripts with dependencies that do not belong hard-coded in a fragile UPCXX installer.

2. Documentation for how to use the system. This includes READMEs with crucial information about how to launch jobs and use the environment variable knobs to tune behavior, known problems, where to report bugs, etc.

3. pkg-config and makefile fragments and all the rest of GASNet, which is required for the GASNet install to be used by other clients. This is likely to be crucial if we ever want UPC++ to interoperate with another GASNet client (eg Legion, who is an ECP project).

UPCXX has no business doing a slap-dash job of installing the GASNet sub-library. We have project members who are paid to maintain a robust GASNet installer -- please use it.

[Dan Bonachea]

Honestly it would be preferable to ONLY support install using an externally-installed GASNet (and properly embed references to it), than to export the ability to generate a broken half-install of GASNet.

[Former user]

Yes, I agree on most counts. Slapdash, sure. But as long as the user has everything needed to build and run a pure upcxx app where they don't want to know of gasnets existence, then the user experience isn't affected. I'm conserving energy here since I'm on vacation and trying to get something for the sep 1st prerelease, so the bar for internal quality is low.

[Dan Bonachea]

But as long as the user has everything needed to build and run a pure upcxx app

There are several conduit configurations where you can build UPCXX apps that simply will not work without the gasnetrun spawner scripts, full stop.
smp-conduit is the only conduit that never needs a spawner and it's essentially a development tool. Real production use requires a spawner and the accompanying system documentation.

I don't see what's difficult about running "gmake install prefix=/upcxx-dir/gasnet" in the gasnet build tree and then referencing that complete install (or just using the user-provided external install). Seems much simpler and more robust than what nobs is currently doing.

[Former user]

Right, lack of spawners is a huge oversight and needs immediate attention. I'm considering that a gaping bug.

I was not aware of "make install prefix=x", (is gmake preferred?) that would make dealing with user provided build trees much easier.

[Dan Bonachea]

(is gmake preferred?)

Agree "make" is the better choice for portability. My fingers are just used to typing gmake for unimportant reasons.

[Former user]

Now I remember. I have the following issue with using gasnet's "make install": It insist on building all the conduits and thread modes as discovered at config time. To workaround this, I was originally configuring with all other conduits and thread modes disabled so that it would only burn time compiling things I need, and then installing. But then Dan was worried that this would add the much higher time sink of a separate configuration (which can be time consuming on HPC filesystems) per conduit,threadmode. So per his advice I switched to configuring for all conduits and threads, but then only issuing make in the conduit specific build directory, and linking directly against the artifacts generated there. But now that I'm using gasnet in this way, I can't invoke "make install" without making compilation time 9x slower than it needs it to be.

So how do I get the best of both: shared configuration across all builds, but build and install only as pertinent to user demand?

Until gasnet gives me finer granularity of what I need installed, I have to peek under the covers. Perhaps there's subrules called by the "install" rule which target conduit+threadmode?

[Dan Bonachea]

But now that I'm using gasnet in this way, I can't invoke "make install" without making compilation time 9x slower than it needs it to be.

I think you're trying to solve a performance issue that has no practical importance. The performance of the install step should not be a consideration.

If we do our job right, "make install" should be a very infrequent operation - end users ideally execute it once per UPCXX release (ie once every 6 months for any machine they are deploying on). Our internal developers will probably never use install for day-to-day work, since it adds an unnecessary step and they can just work in the build directory.

Installation is publication for shared use - the primary reason to install is to deposit the final product (often for use by multiple users) and allow destruction of intermediate build artifacts. First and foremost install needs to be fully comprehensive, including anything all clients will ever need (or at least everything that configure detected). It's unacceptable for install to cut corners in an attempt to shave a few seconds off install time and leave the user with an install tree that's missing what she or her co-workers will need tomorrow when her build directory is gone. This philosophy is directly encoded in the automake infrastructure used by GASNet, it's the correct philosophy, and is not something we are going to erode.

Ideally the "final" installer for UPCXX installs a complete GASNet exactly twice (debug + opt, includes all conduits and thread modes), exactly one copy of the UPCXX headers and documentation, and one libupcxx.a for each of the combinations of {debug,opt}x{conduit}x{seq,par}. If nobs is not capable of doing this then perhaps the outer "install" script should just configure/build/install GASNet twice and then fire up nobs in a loop with external GASNet.

CC: @PHHargrove

[Former user]

Its an optimization with practical importance to me, but I'm special. I very frequently "rm -rf .nobs" because I've modified nobs or the nobsrule.py and I want to see gasnet rebuild itself. Having gasnet go from 30 seconds to several minutes is a pain. This optimization also matters for the no-install usage of upcxx I described in a recent email (below). Its ultimately not detrimental but I insist it matters. But I should be clear that I'm not claiming it matters enough to warrant a frankensteined gasnet install as I have done. That just happens to be what I could muster in a time crunch for Sep 1. I will make sure gasnet gets a full install by Sep 30.

I feel like its more important to address the end-user's experience when doing a local install for their laptop or supercomputer. I think it will be their opinion that matters most for us. And this script falls short there too. For that, I was hoping the power of nobs would be more relevant (but again not directly exposed). The user would keep a upcxx source tree alive and call a script looking just like the installer's "bin/upcxx-meta", except that it sniffs the desired configuration (OPTLEV,DEBUG,CONDUIT,etc) from the current environment and builds on-demand. So the only steps for the user would be:

# user gets our code once
git clone http://.../upcxx.git ~me/upcxx
# create some short-hand
upcxx="~me/upcxx/meta"

# and can immediately build their app
g++ -std=c++11 $(upcxx PPFLAGS) my-app.cpp $(upcxx LIBFLAGS)

# and build again, but in debug mode this time and just on this laptop
export DEBUG=1
export GASNET_CONDUIT="smp"
g++ -std=c++11 $(upcxx PPFLAGS) my-app.cpp $(upcxx LIBFLAGS)

I slightly disagree with your notion of an "ideal" upcxx installation having multiple libraries but a single copy of the header tree. Seperate header trees per configuration allow us to do code-gen into those headers dependent on configuration settings (manually unrolled simd loops maybe?). So I'm not going to advocate we mimic gasnet with one tree, one set of headers, and mangled library names all living together. I'm leaving the mangling up to the outer install script and separate install trees with identical structure. I will provide (by Sep 30) a means for the install script to query the gasnet configure results for a list of all available conduits so that it can craft its loops.

Is this acceptable or should I try harder to put all the artifacts in one installation tree?

[Dan Bonachea]

I'm not arguing that your build process needs to build all of GASNet, just that the install script needs to install all of GASNet (which might mean building it if it's in a partially-built state). I don't see why this affects your day-to-day usage if you never run the install script.

In my day-to-day development I always install packages that I'm not responsible for maintaining (because if something is wrong I'm going to report a bug not go diving through sources to fix it, so there's no point in preserving the source/build directories). On the other hand I rarely install packages I am responsible for maintaining, because in those I frequently edit the source and want to see the result reflected in a build directory I actively use. The only time I ever install my own software is when testing the installer or publishing the package for other users.

Regarding headers (or other artifacts) I don't object to installing multiple copies if there's reason to believe they may eventually differ. GASNet debug and opt headers have exactly this property, which is part of why those are separate install trees. My main point there was I'd rather not end up with a "complete" install of UPCXX installing num_conduits*num_thread_modes completely redundant copies of GASNet (one for each libupcxx.a) where exactly two GASNet install trees would be sufficient for every config of upcxx. However that's a minor optimization (mostly just to save disk space), so it's not critical.

[Former user]

Small gripe, I find it unsatisfactory that gasnet incorporates some configuration parameters (threadmode, conduit) into the name mangling within an install tree, but others (notably debug) are put on the admin to manage as multiple trees. I think it would be better to either put all the mangling on the admin, or none of it. Also, you said ideally an install script should produce every reasonable configuration, yet gasnet's configure leaves out half of them (those with debug=true), forcing the admin to install at least twice. Does gasnet fall just short of your own ideal installer or have I missed something?

Before the looping upcxx install script can exist, a decision will need to be made about how to handle our mangling. Most importantly, how a user can get to the correct mangled name depending on their need. Here's my current thinking: each configuration of upcxx is in a separate install tree under a mangled name (hardlinks amongst trees for identical files to save space). For internally built gasnets, a full copy of the correct gasnet will cohabitate there (again hardlinks for space) except that maybe just maybe I prune out the (threadmode, conduit) lib.a files not relevant so as not to confuse human onlookers. Externally installed gasnets are referenced wherever they are and not pulled into the upcxx tree since the user has taken responsibility for them. The mangled names of each install tree may or may not be human readable (could be format strings like "$debug-$conduit-$etc" or some sha1 hash), either way it won't be documented so users won't assume they can hardcode a way to get at the right mangling. Instead, there will be a "upcxx-meta" script, similar in behavior to the one produce by the single-instance installer, living outside all of the parallel install trees which the user calls to with an environment reflecting their desired configuration. Example:

all-the-upcxx/
  upcxx-meta
  <sha1...#1>/ # pertains to conduit=aires, threads=seq, and more...
    bin/
    include/
      upcxx/{upcxx.hpp, etc...}
      gasnet.h, etc...
    lib/
      libupcxx.a
      aries-conduit/libgasnet-seq.a

  <sha1...#2> # different configuration (conduit=mpi)
    bin/
    include/ # just like sha1...#1
    lib/{libupcxx.a, mpi-conduit/libgasnet-seq.a}

Examples of upcxx-meta:
$(GASNET_CONDUIT=aries THREAD_MODE=seq all-the-upcxx/upcxx-meta PPFLAGS) # retrieve ppflags from sha1...#1
$(GASNET_CONDUIT=mpi THREAD_MODE=seq all-the-upcxx/upcxx-meta PPFLAGS) # retrieve ppflags from sha1...#2

The looping install script's loop will be sweeping across configurations encoded as environment varaibles, installing each one as its own tree, and then it will generate the top-level upcxx-meta script which is called by users and expects to match the pertinent env vars of the user with those swept in the install loop. This will handle external GASNET's very nicely, if the user changes the path GASNET points to (as set by the admin), upcxx-meta will redirect them to the correct upcxx instance. Internal GASNET's might be a little less meaningful in the case of it pointing to local tarballs which existed at install time but no longer. URL's would still make sense though.

So, thoughts?

[Dan Bonachea]

unsatisfactory that gasnet incorporates some configuration parameters (threadmode, conduit) into the name mangling within an install tree, but others (notably debug) .. Does gasnet fall just short of your own ideal installer or have I missed something?

I'm definitely not claiming GASNet's build/install infrastructure is ideal, but it is very robustly portable, stable and well-tested. GASNet is designed to be an embedded compilation target, so the APIs and build infrastructure are not designed for direct use by end users. It's assumed the GASNet client layer (eg UPCXX) will invoke the GASNet build infrastructure however many times makes sense for the high-level task at hand.

Since you asked, there are also many other configuration parameters beyond debug/opt that are not part of a single GASNet install, which include: C compiler family, C compiler version, ABI, GASNet segment mode, PSHM support, and literally several hundred other GASNet configuration parameters that could potentially affect compatibility or runtime operation. Debug/opt just happens to be the most potentially useful toggle in this large configuration space, and one that end users writing new software are likely to want. Bottom-line we can't build libraries for the entire combinatorial space, so had to draw a line somewhere around a set of parameters to encapsulate within a single build/install that we thought represented a minimal useful GASNet install for use in a production-quality client, and we chose that line around (conduit, thread-mode). This decision was influenced by many factors, and after 15 years it's sufficiently baked-in to both GASNet and our existing clients that it's not likely to change (although in EX the meaning of the thread-modes will be redefined somewhat).

Our GASNet clients have dealt with this in different ways. UPCR includes a looping build infrastructure above GASNet (and itself) that performs multiple installs based on debug/opt, UPC translator, and several other parameters as selected by the installing admin, and presents a single unified upcc driver script to select the right one at app build time based on arguments. This has worked relatively well in a PGAS language setting and is the direction I'm trying to steer us towards, but it's not the only valid choice.

Before the looping upcxx install script can exist, a decision will need to be made about how to handle our mangling. .. So, thoughts?

Overall I like the user-visible interface you seem to be proposing for upcxx-meta, although I may have additional feedback once I see more about how it operates. Given UPCXX is a library I think it could be nice to additionally expose the same information via either pkg-config or Makefile fragments, but that could also be a feature upgrade later (potentially generated directly from upcxx-meta).

A few thoughts on your proposed internal design:

• Hard links are not portable, so your scheme will need a way to operate without them. They also don't save much relative to a symbolic link (and unless you plan to write an uninstaller deletion behavior is irrelevant), so it's probably simpler to design for symbolic links.
• Obfuscating paths as sha-1 hashes instead of human-readable names for mangling means you're effectively forcing all users to use upcxx-meta somewhere in their app build infrastructure. This in turn means that script needs to answer every possible valid use case under the sun, including those of users we have not yet recruited. Given this installed product is a static library and not a compiler, I'd lean towards a more conservative and future-proof human-readable naming scheme that is more amenable to "backdooring".
• There's no reason that upcxx and gasnet have to be installed into overlapping trees. It's unlikely we'll have file name collisions between projects, but given we're planning a many-to-one mapping of libupcxx to libgasnet, it might be simpler to just install the 1 or 2 GASNet trees at the top level in their own tree. Ie:

all-the-upcxx/
  upcxx-meta
  gasnet-debug/
    lib/
    include/
    ...
  gasnet-opt/
    lib/
    include/
    ...
  upcxx-debug-smp-par/
    lib/
    include/
    ...  
  upcxx-debug-aries-par/
    lib/
    include/
    ... 
  upcxx-opt-smp-seq/
    lib/
    include/
    ... 
  upcxx-opt-aries-seq/
    lib/
    include/
    ...     
  ...

I think this organization also mostly eliminates the need for linkage within the tree (ie upcxx-meta returns the union of -I and -L options for the selected GASNet and UPCXX). If you optionally wanted to save space for upcxx headers, there could be a top-level upcxx include dir holding files shared between configs.

Finally, it wasn't clear to me if you were proposing to automate anything about the C++ compiler decision (ie $CXX setting). There's actually a lot of subtle ugliness embedded in that particular configuration knob (eg /usr/bin/CC in PrgEnv-gnu/6.0.3 vs PrgEnv-intel/6.0.4), so I'd advocate us NOT trying to solve that problem in this release (ie assume the admin has somehow locked it on one setting for this overall UPCXX install tree, and possibly enforce that somehow).

[Former user]

New install script behavior for multi-config install. Correctly installs gasnet. Old behavior preserved by adding --single to install invocation. I declare this as fixed issue 11.

→ <<cset a0d6e07be029>>