Install executables in virtual environments with flit install --python

[madsmtm]

So this is a bit of a wierd one: When installing into a virtual environment via. flit install --python, installed executables link to the wrong python executable in the shebang header.

How to reproduce

Put a requirement like black (or any other package tool that creates an executable) into the requires key, and install your package under a virtual environment.

$ cat flittest/__init__.py
"""Test"""
__version__ = "1.2.3"

$ cat pyproject.toml
[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"
[tool.flit.metadata]
module = "flittest"
author = ""
author-email = ""
requires = ["black"]  # Or any other tool that creates an executable

$ python -m virtualenv test-venv
New python executable in ./test-venv/bin/python
Installing setuptools, pip, wheel...done.

$ flit install --python ./test-venv/bin/python
Copying package file(s) from flittest
------ Download and installation output omitted ------
Installing collected packages: click, toml, appdirs, attrs, black, flittest
  WARNING: The scripts black and blackd are installed in './test-venv/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed appdirs-1.4.3 attrs-19.1.0 black-19.3b0 click-7.0 flittest-1.2.3 toml-0.10.0

$ head -n 1 ./test-venv/bin/black
#!/usr/local/Cellar/python/3.7.3/bin/python3.7

Notice the last line - The shebang points the python executable that was used to run flit, not the virtual environment's python!

If I install the package manually, then I get the expected result:

$ ./test-venv/bin/python -m pip install black
------ Download output omitted ------
Installing collected packages: click, toml, appdirs, attrs, black, flittest
Successfully installed appdirs-1.4.3 attrs-19.1.0 black-19.3b0 click-7.0 flittest-1.2.3 toml-0.10.0

$ head -n 1 ./test-venv/bin/black
#![path-to-current-folder]/test-venv/bin/python

Environment

• Flit 1.3
• MacOS Mojave
• Python 3.7

Please guide me if I'm doing something wrong, or if it's a bug in flit, how I could help fix it 😉

[takluyver]

Odd. It should be creating a wheel and then asking pip to install it, with this line:

https://github.com/takluyver/flit/blob/a2e79f34d3a9edbd8bedb45c1376a3a3cb0344dd/flit/install.py#L338

self.python there should be the path you gave it with --python.

Maybe experiment with making a wheel and then installing it with pip manually.

[madsmtm]

I tried your suggestion, and the shebang was still wrong, so after instrumenting the code a bit I ended up determining the cause:

The environment variable __PYVENV_LAUNCHER__ is set to the python executable running flit, for internal reasons relating to how MacOS (or perhaps just HomeBrew?) launches python. That ends up overwriting the actually launched python executable, see bpo-22490 and python/cpython#9516 for further details.

Basically, because of the above, you can't launch a separate python executable from the currently running one.

If you want, you could handle it with the following code:

import os
if "__PYVENV_LAUNCHER__" in os.environ:
    del os.environ["__PYVENV_LAUNCHER__"]

But I'm not sure if flit should handle this, or if we should expect it being fixed upstream. You're free to close the issue if you don't want to handle it 👍

[takluyver]

Have I understood correctly that python/cpython#9516 would fix this if it were adopted?

If so, I'd be inclined to wait for that rather than trying to work around it in Flit. This looks like we could be getting into an escalating battle of tools trying to override other tools' behaviours which they don't like. Maybe that's not the case, but as a non Mac user, it's not very interesting to spend time trying to figure out an issue that sounds like it's Mac specific.

[madsmtm]

As far as I understand, the PR would fix the issue.
One could make an argument that flit should handle this since __PYVENV_LAUNCHER__ is a bug / a leftover from the other tools (opposed to a feature, where flit definitely shouldn't change the behaviour), and that bug specifically impacts flit.

But on the other hand, it's something really specific, that I can't really expect anyone to maintain, so I'll agree with you, push for python/cpython#9516 or similar, and just close the issue, thanks again 👍

[takluyver]

Thanks. If it doesn't go into Python, I might be persuadable to fix it in flit.

…

On Mon, 22 Jul 2019, 23:07 Mads Marquart, ***@***.***> wrote: Closed #274 <#274>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#274>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACQB5PIFASTKPZK2DRBKXDQAYVR5ANCNFSM4H5FEEGA> .