sys.executable seems incorrect on py3.7 from subprocess

[jamesls]

I'm noticing a difference in python3.7 when trying to reference sys.executable. It doesn't match the first argument (the path to the virtualenv python) and instead points to the system wide python. I noticed this when debugging pip build failures that spawn processes when utilizing pep517 builds, but the issue seems to be more general.

Behavior I'm expecting

So in python versions except py37, I see the behavior I expect. For example in python 2.7:

/tmp/repro $ virtualenv --version
16.7.8
/tmp/repro $ virtualenv --python python2 venv27
Running virtualenv with interpreter /usr/local/bin/python2
Already using interpreter /usr/local/opt/python@2/bin/python2.7
New python executable in /private/tmp/repro/venv27/bin/python2.7
Also creating executable in /private/tmp/repro/venv27/bin/python
Installing setuptools, pip, wheel...
done.

/tmp/repro $ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, subprocess
>>> sys.executable
'/Library/Frameworks/Python.framework/Versions/3.7/bin/python3'
>>> subprocess.Popen(['./venv27/bin/python', '-c', 'import sys; print(sys.executable)']).wait()
/private/tmp/repro/venv27/bin/python
0
>>> ^D

So in the example above, sys.executable points to ./venv27/bin/python which is what I'd expect. I've also confirmed the same expected behavior in python3.6.

However if I try the same thing using python3.7, I get different behavior.

/tmp/repro $ virtualenv --python python3 venv37
Running virtualenv with interpreter /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Already using interpreter /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.7'
New python executable in /private/tmp/repro/venv37/bin/python3
Also creating executable in /private/tmp/repro/venv37/bin/python
Installing setuptools, pip, wheel...
done.

/tmp/repro $ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, subprocess
>>> sys.executable
'/Library/Frameworks/Python.framework/Versions/3.7/bin/python3'
>>> subprocess.Popen(['./venv37/bin/python', '-c', 'import sys; print(sys.executable)']).wait()
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3
0
>>>

In this example, sys.executable is not pointing to ./venv37/bin/python and instead points to the system wide python.

If I use the built in venv module in python3.7 I don't see the issue:

/tmp/repro $ python3 -m venv venv37-builtin
/tmp/repro $ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen(['./venv37-builtin/bin/python', '-c', 'import sys; print(sys.executable)']).wait()
/private/tmp/repro/venv37-builtin/bin/python

UPDATE: Looking into this more, it appears to be due to differences in __PYVENV_LAUNCHER__ being set. In my case, /usr/local/bin/python3 is a framework build of python. Here's the main difference I can see:

$ virtualenv --python /usr/local/bin/python3 venv37-external
$ /usr/local/bin/python3 -m venv venv37-internal

$ ./venv37-builtin/bin/python -c "import sys, os; print(f'\nsys.executable: {sys.executable}\n__PYVENV_LAUNCHER__: {os.environ.get(\"__PYVENV_LAUNCHER__\")}')"

sys.executable: /private/tmp/asdf/venv37-builtin/bin/python
__PYVENV_LAUNCHER__: /private/tmp/asdf/venv37-builtin/bin/python


$ ./venv37-external/bin/python -c "import sys, os; print(f'\nsys.executable: {sys.executable}\n__PYVENV_LAUNCHER__: {os.environ.get(\"__PYVENV_LAUNCHER__\")}')"

sys.executable: /private/tmp/asdf/venv37-external/bin/python
__PYVENV_LAUNCHER__: None

[gaborbernat]

This is fixed by the rewrite branch I believe.