Is the pip3 install broken on macOS?

[Rian Stockbower]

I feel like I'm missing something simple and obvious, but I can't get the Jiang2013 example to work. I'm not a Python developer, but I do have about a decade of professional experience as a software developer in other ecosystems, so I'm not totally ignorant.

This is what I did, starting from a fresh install of macOS:

1. Install the xcode CLI tools (which gets you gcc without needing all 10GB of the xcode IDE): xcode-select --install
2. Downloaded python 3.9.4 from https://www.python.org/downloads/mac-osx/ and installed it (This gets you pip3, too)
3. I then installed presto: pip3 install presto --user which also grabbed the associated dependencies -- everything installed without errors.

I launched the python3 REPL, and quickly tested biopython, and had no problems. My test was fairly simple:

from Bio.Seq import Seq
my_seq = Seq("AGTACACTGGT")
print(my_seq)

1. After that, I imported presto, just to see if python3 could find it: import presto -- no problems. (Wasn't sure how to test it, though.)
2. From there, I downloaded the Jiang sample, and tried to run the bash script, which errored out:

% ./ExamplePipeline_Jiang2013.sh
readlink: illegal option -- f
usage: readlink [-n] [file ...]
readlink: illegal option -- f
usage: readlink [-n] [file ...]
readlink: illegal option -- f
usage: readlink [-n] [file ...]
readlink: illegal option -- f
usage: readlink [-n] [file ...]
DIRECTORY: output
START
   1: FilterSeq length         11:21 04/19/21
./ExamplePipeline_Jiang2013.sh: line 30: FilterSeq.py: command not found
   2: FilterSeq quality        11:21 04/19/21
./ExamplePipeline_Jiang2013.sh: line 35: FilterSeq.py: command not found
   3: MaskPrimers score        11:21 04/19/21
...etc...

Weird. MacOS did recently switch to zsh as the default shell, but it still has bash, not that it should matter given the script header. Nonethless, I launched bash, and tried running it again: same result.

So I went looking for FilterSeq.py, and the other files that are associated with pRESTO:

% ls /Users/rianjs/Library/Python/3.9/lib/python/site-packages/presto
total 344
-rw-r--r--@  1 rianjs  staff  24189 Jun 22  2020 Annotation.py
-rw-r--r--@  1 rianjs  staff  16014 Jul 15  2020 Applications.py
-rw-r--r--@  1 rianjs  staff  14310 Jun 22  2020 Commandline.py
-rw-r--r--@  1 rianjs  staff   1814 Jun 22  2020 Defaults.py
-rw-r--r--@  1 rianjs  staff  12264 Sep 27  2020 IO.py
-rw-r--r--@  1 rianjs  staff  23864 Jun 22  2020 Multiprocessing.py
-rw-r--r--@  1 rianjs  staff  66028 Sep 27  2020 Sequence.py
-rw-r--r--@  1 rianjs  staff    293 Oct 13  2020 Version.py
-rw-r--r--   1 rianjs  staff    259 Jun 22  2020 __init__.py
drwxr-xr-x  11 rianjs  staff    352 Apr 19 10:21 __pycache__

I explored some of the subdirectories, and there was nothing that looked promising. (ls -la didn't show anything else.)

So I downloaded the raw tarball from pypi, and decompressed it. Poking around, I DO see FilterSeq.py, et al in the bin folder, which looks like the meat of pRESTO:

% ls ~/Downloads/presto-0.6.2/bin 
total 600
-rwxr-xr-x@ 1 rianjs  staff  20725 Sep 27  2020 AlignSets.py
-rwxr-xr-x@ 1 rianjs  staff  24308 Jun 22  2020 AssemblePairs.py
-rwxr-xr-x@ 1 rianjs  staff  24122 Jun 22  2020 BuildConsensus.py
-rwxr-xr-x@ 1 rianjs  staff  28429 Jun 22  2020 ClusterSets.py
-rwxr-xr-x@ 1 rianjs  staff  20407 Jun 22  2020 CollapseSeq.py
-rwxr-xr-x@ 1 rianjs  staff  11328 Jun 22  2020 ConvertHeaders.py
-rwxr-xr-x@ 1 rianjs  staff  36792 Jun 22  2020 EstimateError.py
-rwxr-xr-x@ 1 rianjs  staff  12126 Jun 22  2020 FilterSeq.py
-rwxr-xr-x@ 1 rianjs  staff  28060 Jun 22  2020 MaskPrimers.py
-rwxr-xr-x@ 1 rianjs  staff  12157 Jun 22  2020 PairSeq.py
-rwxr-xr-x@ 1 rianjs  staff  21913 Jun 22  2020 ParseHeaders.py
-rwxr-xr-x@ 1 rianjs  staff   5419 Jun 22  2020 ParseLog.py
-rwxr-xr-x@ 1 rianjs  staff  38028 Jun 22  2020 SplitSeq.py
-rwxr-xr-x@ 1 rianjs  staff   7129 Jun 22  2020 UnifyHeaders.py

I went back to /Users/rianjs/Library/Python/3.9/lib/python/site-packages/presto, and there is NOTHING like that in there. I did a find, just to see, and nothing showed up:

rianjs@Galadriel: ~/Library/Python/3.9/lib/python
% find . -name "Filter*"
rianjs@Galadriel: ~/Library/Python/3.9/lib/python
% find . -name "Command*"
./site-packages/presto/Commandline.py
./site-packages/presto/__pycache__/Commandline.cpython-39.pyc

Is the macOS package broken in some way?

[Rian Stockbower]

Figures, as soon as I submit the issue, I discover that actually the files ARE on my machine, just in a place I didn’t expect:

~/Library/Python/3.9/bin/

So I guess the question now is “Why doesn’t the example script work?”

[Jason Vander Heiden]

Greetings @Rian Stockbower , there’s two things going on here:

-1. The package contains both API and executable components. The typical behavior of a python package is to install the API bits into $PYTHONUSERBASE/lib and the scripts into $PYTHONUSERBASE/bin. The location of $PYTHONUSERBASE varies with the OS and python install. It looks like you figured this out as I was writing this, so… it’s just a matter of putting the bin directory into your $PATH

-2. The issue with the example script is because the default installation of readlink on Mac OS is not the GNU version and, hence, does not have the -f argument. When we wrote this script we didn’t check it outside linux, so that’s our mistake.

You can either (1) remove the readlink commands from the beginning of the script and just put in the absolute (or correct relative) paths to the input files or (2) install greadlink (by installing coreutils via homebrew) and alias it to readlink or replace readlink with greadlink in the script.

Either way, we should fix the example script so it’s cross-platform.

[Rian Stockbower]

Yes, sir, those two changes did it, and the example is working great in both zsh and bash. Thank you for your help.

(Random aside: it's been many years since I've felt so out of my depth, at something I should ostensibly be good at. Each platform has its own conventions and idiosyncracies. To me, ASP.NET Core and C# are as easy as breathing, but a few hours in Python, and I feel like I'm stumbling around in the dark! Ah well, at least the underpinning CS concepts are fundamental, even if the platforms are not.)

[Rian Stockbower]

User error.

[Jason Vander Heiden]

Sure thing.

Yeah, python is a bit weird on Mac. Python 2 is EOL, but it’s still the base install in Mac OS, so you have to go fetch a third party install of Python 3. The directory structure isn’t the same as on *nix at all and depends on the install source. I personally find homebrew the easiest way to manage it (over anaconda or the official packages), because it gives you general purpose apt/dnf/rpm like inferface.