shebang of script installed in system site packages venv points to system python in Mac OS Homebrew

[thomasgilgenast]

Environment

• pip version: 20.1.1
• Python version: 3.8.4
• OS: Mac OS Catalina 10.15.5 (Homebrew Python)

Description

Packages installed using pip into a venv created with --system-site-packages and with a Homebrew base interpreter have shebangs written in their scripts that point to the system Python, causing these scripts to crash with ModuleNotFoundError.

To clarify, this might not be a bug in pip - I am opening this issue to get feedback on my workaround and learn about where this issue should actually be opened, as well as to provide information for anyone who encounters this in the future (at first glance this seemed to me to be a bug in pip). If this is indeed not a bug in pip and belongs elsewhere, this issue can be closed and I can re-open it where it "belongs".

Expected behavior

When installing a package into a venv, I expect pip to write the shebang of the scripts included by that package so that they point to the venv's interpreter, not the system Python.

MInimal reproducible example

% python3 -m venv --system-site-packages venv
% source venv/bin/activate
(venv)% pip install tox
(venv)% tox --version
Traceback (most recent call last):
  File ".../venv/bin/tox", line 5, in <module>
    from tox import cmdline
ModuleNotFoundError: No module named 'tox'

I use tox as an example package here, but I have seen this same behavior with every other Python package I have tested.

My understanding/workaround

This happens because the shebang in venv/bin/tox incorrectly points to the system Python, which in turn seems to be because sys.executable is set to the system Python, even though python is executed from inside the venv:

(venv)% python
Python 3.8.4 (default, Jul 14 2020, 02:58:48)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/[email protected]/bin/python3.8'

Passing the absolute path to the venv's Python interpreter does not change this.

This is not the case on Linux (python:3.7 Docker image)

$ python3 -m venv --system-site-packages venv
$ source venv/bin/activate
(venv)$ python
Python 3.7.8 (default, Jun 30 2020, 18:27:23)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/venv/bin/python'

or if --system-site-packages is turned off

% python3 -m venv venv-nossp
% source venv-nossp/bin/activate
(venv-nossp)% python
Python 3.8.4 (default, Jul 14 2020, 02:58:48)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'.../venv-nossp/bin/python'

This github comment by @chbrown (thank you!) pointed me in the direction of /usr/local/lib/python3.8/site-packages/sitecustomize.py, a file generated by Homebrew that appears to be responsible for setting sys.executable to point to /usr/local/opt/[email protected]/bin/python3.8 if PYTHONEXECUTABLE is not set.

If my venv's all include "venv" in their folder name, I can write something like

for p in sys.path:
    if 'venv' in p:
        sys.executable = p.split('lib')[0] + 'bin/python'

at the end of this file, which re-overwrites sys.executable to the venv's interpreter whenever sys.path appears to contain a venv. (There is probably a better way to identify the correct value of sys.executable when running Python from inside a venv.)

After making this change, everything works as expected. However, this feels like a quite fragile/"wrong" way to address the problem.

There are many related github issues (especially pypa/virtualenv#845) that lead me to think that a change in how __PYVENV_LAUNCHER__ is interpreted (or not interpreted) by pip's vendored distlib here is somehow related to this, but in the end I am not sure I understand the problem well enough to say that this is an issue in pip, virtualenv, or Homebrew, or if there's just a "better" way for me to apply my workaround, or if I just have something configured incorrectly on my end (I am quite new to Mac OS).

[uranusjr]

This should be categorised as a Homebrew issue to me. pip does not know about Homebrew’s symlinks and can’t really do anything here.

Edit: Regarding __PYVENV_LAUNCHER__, the environment variable has been removed (python/cpython#9516). I believe 3.8.4 should contain this fix.

[thomasgilgenast]

Got it, I will move this over to Homebrew then. Thanks for the input!

Related to __PYVENV_LAUNCHER__ being removed, I see this behavior with Python 3.8.4 as well as 3.7.7 which I believe should include the backport of the PR you linked. I guess this is further evidence that this is a Homebrew question, because the Homebrew-generated sitecustomize.py interacts with PYTHONEXECUTABLE, not __PYENV_LAUNCHER__ (which as you've pointed out doesn't exist anymore anyway).