BLOCKER: Nobs crash with Python 2.6.0/2.7.0 on Dirac

[Dan Bonachea]

We currently enforce a Python requirement of 2.6 in our scripts, but using Python 2.6.0 OR 2.7.0 on dirac (pcp-d-6 : module load python/2.x.0) currently gives nobs crashes:

$ git describe
upcxx-2017.9.1-61-g016ca07
$ module switch python python/2.6.0
$ python --version
Python 2.6
$ rm -Rf .nobs/ ; ./install poo
UPCXX revision: upcxx-2017.9.1-61-g016ca07
System: Linux pcp-d-6 3.10.0-693.1.1.el7.x86_64 #1 SMP Tue Aug 15 08:36:44 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: Scientific
Description:    Scientific Linux release 7.4 (Nitrogen)
Release:        7.4
Codename:       Nitrogen

Date: Fri Jan 26 08:26:27 PST 2018
Current directory: /home/pcp1/bonachea/UPC/upcxx
Install directory:
Settings: UPCXX_CODEMODE='debug'

/usr/local/pkg/gcc/7.2.0/bin/g++
g++ (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/usr/local/pkg/gcc/7.2.0/bin/gcc
gcc (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Downloading https://gasnet.lbl.gov/EX/GASNet-2017.12.0.tar.gz
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Uncaught exception

Traceback (most recent call last):
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/tool_main.py", line 86, in main
    printed.wait()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 447, in wait
    return me.wait_futurized().value()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1085, in execute
    kind, dir_path = yield me.get_config()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1034, in execute
    cc, cxx, debug, config, source_dir, user_args = yield me.get_config()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1020, in get_config
    source_dir = yield cxt.gasnet_source()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 879, in execute
    yield download()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 604, in fire1
    proxied = lam(*arg_result._val_seq, **arg_result._val_kws)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 876, in download
    urllib.urlretrieve(url, tgz)
  File "/usr/local/pkg/python/2.6.0/lib/python2.6/urllib.py", line 94, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/local/pkg/python/2.6.0/lib/python2.6/urllib.py", line 235, in retrieve
    fp = self.open(url, data)
  File "/usr/local/pkg/python/2.6.0/lib/python2.6/urllib.py", line 200, in open
    return self.open_unknown(fullurl, data)
  File "/usr/local/pkg/python/2.6.0/lib/python2.6/urllib.py", line 212, in open_unknown
    raise IOError, ('url error', 'unknown url type', type)
IOError: [Errno url error] unknown url type: 'https'
UPC++ Installation failed. Please report the entire log above to: [e-mail]

$ module switch python python/2.7.0
$ rm -Rf .nobs/ ; ./install poo
UPCXX revision: upcxx-2017.9.1-61-g016ca07
System: Linux pcp-d-6 3.10.0-693.1.1.el7.x86_64 #1 SMP Tue Aug 15 08:36:44 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: Scientific
Description:    Scientific Linux release 7.4 (Nitrogen)
Release:        7.4
Codename:       Nitrogen

Date: Fri Jan 26 08:26:34 PST 2018
Current directory: /home/pcp1/bonachea/UPC/upcxx
Install directory:
Settings: UPCXX_CODEMODE='debug'

/usr/local/pkg/gcc/7.2.0/bin/g++
g++ (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/usr/local/pkg/gcc/7.2.0/bin/gcc
gcc (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Downloading https://gasnet.lbl.gov/EX/GASNet-2017.12.0.tar.gz
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Uncaught exception

Traceback (most recent call last):
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/tool_main.py", line 86, in main
    printed.wait()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 447, in wait
    return me.wait_futurized().value()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1085, in execute
    kind, dir_path = yield me.get_config()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1034, in execute
    cc, cxx, debug, config, source_dir, user_args = yield me.get_config()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 601, in fire1
    proxied = lam(arg_result)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 549, in result
    return resultoid()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 1020, in get_config
    source_dir = yield cxt.gasnet_source()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/memodb.py", line 467, in memo_execute
    yield result
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 751, in fire
    arg = gen_throw(arg.exception, None, arg.traceback)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 879, in execute
    yield download()
  File "/home/pcp1/bonachea/UPC/upcxx/nobs/nobs/async.py", line 604, in fire1
    proxied = lam(*arg_result._val_seq, **arg_result._val_kws)
  File "/home/pcp1/bonachea/UPC/upcxx/nobsrule.py", line 876, in download
    urllib.urlretrieve(url, tgz)
  File "/usr/local/pkg/python/2.7.0/lib/python2.7/urllib.py", line 91, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/local/pkg/python/2.7.0/lib/python2.7/urllib.py", line 237, in retrieve
    fp = self.open(url, data)
  File "/usr/local/pkg/python/2.7.0/lib/python2.7/urllib.py", line 202, in open
    return self.open_unknown(fullurl, data)
  File "/usr/local/pkg/python/2.7.0/lib/python2.7/urllib.py", line 214, in open_unknown
    raise IOError, ('url error', 'unknown url type', type)
IOError: [Errno url error] unknown url type: 'https'
UPC++ Installation failed. Please report the entire log above to: [e-mail]

$ module switch python python/sys-default 
$ python --version
Python 2.7.5
$ rm -Rf .nobs/ ; ./install poo
UPCXX revision: upcxx-2017.9.1-61-g016ca07
System: Linux pcp-d-6 3.10.0-693.1.1.el7.x86_64 #1 SMP Tue Aug 15 08:36:44 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: Scientific
Description:    Scientific Linux release 7.4 (Nitrogen)
Release:        7.4
Codename:       Nitrogen

Date: Fri Jan 26 08:30:28 PST 2018
Current directory: /home/pcp1/bonachea/UPC/upcxx
Install directory:
Settings: UPCXX_CODEMODE='debug'

/usr/local/pkg/gcc/7.2.0/bin/g++
g++ (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/usr/local/pkg/gcc/7.2.0/bin/gcc
gcc (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Downloading https://gasnet.lbl.gov/EX/GASNet-2017.12.0.tar.gz
Finished    https://gasnet.lbl.gov/EX/GASNet-2017.12.0.tar.gz
Configuring GASNet...
(in /home/pcp1/bonachea/UPC/upcxx/.nobs/art/1a1cfc7cebb7e2113a4184368fd5ca28074c74a0)
/home/pcp1/bonachea/UPC/upcxx/.nobs/art/35f64cafbe12c820f471b72fee18d916c31295c9/GASNet-2017.12.0/configure --enable-debug --disable-psm --disable-mxm --disable-portals4 --disable-ofi --disable-parsync
<works>

For reference, here are some relevant release dates from https://www.python.org/downloads/:

• Python 2.7.9 2014-12-10
• Python 2.7.5 2013-05-12
• Python 2.7.0 2010-07-03
• Python 2.6.0 2008-10-02

IMO this is a show-stopper that needs to be dealt with (using minimal code changes) before release. My vote is that we document 2.7.5 as the minimum version for this beta release (using the patch below) and post-release investigate further.

--- a/README.md
+++ b/README.md
@@ -40,7 +40,7 @@ The current release is known to work on the following configurations:

 Miscellaneous software requirements:

-* Python version 2.6 or newer
+* Python version 2.7.5 or newer

 * Perl version 5.005 or newer

diff --git a/nobs/tool.py b/nobs/tool.py
index c061408..72f13b5 100755
--- a/nobs/tool.py
+++ b/nobs/tool.py
@@ -2,8 +2,8 @@

 import sys

-if sys.version_info < (2,6):
-  sys.stdout.write('ERROR: Python 2.6 required.\n')
+if sys.version_info < (2,7,5):
+  sys.stdout.write('ERROR: Python >= 2.7.5 required.\n')
   exit(1)

 if sys.version_info[0] != 2:
diff --git a/utils/upcxx-run b/utils/upcxx-run
index 78c8a5e..e204757 100755
--- a/utils/upcxx-run
+++ b/utils/upcxx-run
@@ -176,8 +176,8 @@ def set_ssh_servers(ssh_servers, nranks, conduit):


 def main():
-    if sys.version_info < (2, 6):
-        print('Error: Python version >= 2.6 required')
+    if sys.version_info < (2, 7, 5):
+        print('Error: Python version >= 2.7.5 required')
         return 1

     args = load_args()

Note the version checks in both scripts above are NOT printing the expected message to the console for Python 2.6.0, due to other errors:

$ python --version
Python 2.6
$ ./utils/upcxx-run --help
Traceback (most recent call last):
  File "./utils/upcxx-run", line 8, in <module>
    import argparse
ImportError: No module named argparse
$ rm -Rf .nobs/ ; ./install poo
UPCXX revision: upcxx-2017.9.1-61-g016ca07
System: Linux pcp-d-6 3.10.0-693.1.1.el7.x86_64 #1 SMP Tue Aug 15 08:36:44 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: Scientific
Description:    Scientific Linux release 7.4 (Nitrogen)
Release:        7.4
Codename:       Nitrogen

Date: Fri Jan 26 08:40:22 PST 2018
Current directory: /home/pcp1/bonachea/UPC/upcxx
Install directory:
Settings: UPCXX_CODEMODE='debug'

/usr/local/pkg/gcc/7.2.0/bin/g++
g++ (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/usr/local/pkg/gcc/7.2.0/bin/gcc
gcc (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


GASNET must point to local tarball file, tarball url, or gasnet source tree.

[Dan Bonachea]

I was able to make the nobs version check reach the install console by also sending the message to stderr:

diff --git a/nobs/tool.py b/nobs/tool.py
index c061408..6b7f964 100755
--- a/nobs/tool.py
+++ b/nobs/tool.py
@@ -2,8 +2,9 @@

 import sys

-if sys.version_info < (2,6):
-  sys.stdout.write('ERROR: Python 2.6 required.\n')
+if sys.version_info < (2,7,5): # send to stderr to enure install visibility
+  sys.stdout.write('ERROR: Python >= 2.7.5 required.\n')
+  sys.stderr.write('ERROR: Python >= 2.7.5 required.\n')
   exit(1)

 if sys.version_info[0] != 2:

[Dan Bonachea]

Proposed resolution in pull request #10

After the beta we can explore lowering the minimum version provided we fix the problems preventing older versions from working.

[Dan Bonachea]

Merge pull request #10 into develop

Resolves issue #117

• issue-117-python275: issue 117: Raise minimum required Python to v2.7.5

→ <<cset 957e98a2a2de>>