Nobs crash with Python 2.6.9 on Sierra

[Dan Bonachea]

We currently enforce a Python requirement of 2.6 in our scripts, but using Python 2.6.9 on our Sierra system sierra : /System/Library/Frameworks/Python.framework/Versions/2.6/bin/python currently gives a nobs crash:

rm -Rf .nobs/ ; ./install tmp
UPCXX revision: upcxx-2017.9.1-57-gabd7757
System: Darwin sierra.local 16.7.0 Darwin Kernel Version 16.7.0: Wed Oct 4 00:17:00 PDT 2017; root:xnu-3789.71.6~1/RELEASE_X86_64 x86_64
ProductName:    Mac OS X
ProductVersion: 10.12.6
BuildVersion:   16G1036
Xcode 8.3.3
Build version 8E3004b

Date: Thu Jan 25 18:44:14 PST 2018
Current directory: /Users/bonachea/UPC/upcxx
Install directory:
Settings:

/usr/bin/g++
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
/usr/bin/gcc
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Downloading http://mantis.lbl.gov/nightly/unlisted/GASNet-EX-collaborator-snapshot.tar.gz
Finished    http://mantis.lbl.gov/nightly/unlisted/GASNet-EX-collaborator-snapshot.tar.gz
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Uncaught exception

Traceback (most recent call last):
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/tool_main.py", line 86, in main
    printed.wait()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 447, in wait
    return me.wait_futurized().value()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobsrule.py", line 1085, in execute
    kind, dir_path = yield me.get_config()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobsrule.py", line 1034, in execute
    cc, cxx, debug, config, source_dir, user_args = yield me.get_config()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobsrule.py", line 1020, in get_config
    source_dir = yield cxt.gasnet_source()
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/Users/bonachea/UPC/upcxx/nobs/nobs/async.py", line 749, in fire
    arg = gen_send(arg.value())
  File "/Users/bonachea/UPC/upcxx/nobsrule.py", line 886, in execute
    with tarfile.open(tgz) as f:
AttributeError: 'TarFile' object has no attribute '__exit__'
UPC++ Installation failed. Please report the entire log above to: [e-mail]

Before March we need to either fix this (and test for any other problems back to Python 2.6.0), or raise our minimum required Python version to 2.7.something and update our script checks and documentation accordingly.

[Paul Hargrove]

Our current automated testing covers nothing older then 2.7.5 (on Dirac and Kotten). So, we have not (to the best of my knowledge) tested 2.7.[0-4].

[Paul Hargrove]

On Dirac and Kotten, one can now module load python/2.7.0.
This is my first time building Python and so this might be totally broken. However, rm -Rf .nobs; ./install inst works for me with this python build. I would ask that somebody else do a more complete check.

There is also now a python/2.6.0 module if we decide to fix the compatibility rather than raise our minimum.

[Dan Bonachea]

As of 0468581 we now prohibit Python versions < 2.7.5 (see related issue #117)

Leaving this issue open so we remember to return to this after the beta and have further discussion regarding that decision.

[Paul Hargrove]

I have no objection to the resolution of issue #117, which seems to be a safe/sane response to the desire to have a known-good python version documented and enforced in our release.

However, I am concerned by the fact that when I installed 2.7.0 I did not observe the problem Dan reported there. The transcript below is reconstructed from my command history (priot to the [...]) and scroll back buffer (the success message and after) and shows my attempt of ~11 hours ago when I reported "works for me with this python build" in this issue. It happens to be on the same pcp-d-6 node Dan reports using.

I am mentioning this here, not in #117, because when it come time to see about any further adjustments to the required python version, we should try to understand what differs between Dan's and my trials.

I do not think any additional action is required at this time.

{phargrov@pcp-d-6 upcxx}$ rm -rf .nobs/
{phargrov@pcp-d-6 upcxx}$ module load python/2.7.0
{phargrov@pcp-d-6 upcxx}$ ./install inst
[...]
UPC++ successfully installed
{phargrov@pcp-d-6 upcxx}$ python --version
Python 2.7
{phargrov@pcp-d-6 upcxx}$ git describe
upcxx-2017.9.1-35-gd29da42

[Dan Bonachea]

However, I am concerned by the fact that when I installed 2.7.0 I did not observe the problem Dan reported there.

This was caused because Paul was using the develop snapshot before I updated the http: collaborator-snapshot URL to the https: url for the beta release

[Dan Bonachea]

Mass roll-over of unresolved issues to the next milestone.

[Dan Bonachea]

This issue was triaged at the 2018-06-13 Pagoda meeting.

We decided to keep the installer requirement for Python 2.7, so this issue won't be fixed.