Remove bundled setuptools #95299

sbidoul · 2022-07-26T20:37:59Z

Feature or enhancement

Remove the bundled setuptools so that ensurepip and python -m venv only installs pip.

Context

The setup.py install command of setuptools is deprecated.

However, in an environment where setuptools is installed but wheel is not (such as one created with python -m venv), pip falls back on running the deprecated and non-standard setup.py install.

Since version 22.1 pip works correctly by default in environments where setuptools is not installed, by enabling its PEP 517 mode automatically, leading to unsurprising installations in most cases.

So, in order to progressively expose more users to standard-based installation workflows, we (the pip team) would like that virtual environments are created without setuptools by default.

Users faced with failing installations following this change (likely due to packages with customized setup.py that do not support building a wheel) can easily pip install setuptools to solve the issue.

Previous discussion

pypa/pip#8102 (comment) and following comments has some more context.

Linked PRs

The text was updated successfully, but these errors were encountered:

sbidoul · 2022-07-26T20:40:06Z

@pablogsal do you think it is still acceptable to do this in 3.11? If positive we can likely land a PR on time for RC1.

pablogsal · 2022-07-26T20:41:56Z

Unfortunately I don't feel comfortable landing this so late in the release cycle, specially so close to RC1 and given the current stability of 3.11 :(

This will need to wait until 3.12.

rafrafek · 2022-11-07T15:09:39Z

It feels strange that setuptools is included when using -m venv, but wheel is not included.

pradyunsg · 2023-01-14T22:44:15Z

I've filed #101039 for this (almost had a number with three 0s), targetting 3.12.

edmorley · 2023-02-23T12:49:23Z

Since version 22.1 pip works correctly by default in environments where setuptools is not installed, by enabling its PEP 517 mode automatically, leading to unsurprising installations in most cases.

Do you have some examples of cases where the PEP 517 mode doesn't work for legacy packages? :-)

FFY00 · 2023-03-04T08:05:56Z

PEP 517 is backwards compatible, it should work on legacy packages, the opposite is the issue, pip trying to use the legacy path for packages that need PEP 517.

…directly" This reverts commit 41c5a66.

# This is the 1st commit message: pythongh-95299: Rework test_cppext.py to not invoke setup.py directly # This is the commit message #2: Add tests/cppextdata data to `TESTSUBDIRS` # This is the commit message #3: Revert "Add tests/cppextdata data to `TESTSUBDIRS`" This reverts commit 635492e.

) * gh-95299: Rework test_cppext.py to not invoke setup.py directly * Add tests/cppextdata data to `TESTSUBDIRS` * Revert "Add tests/cppextdata data to `TESTSUBDIRS`" This reverts commit 635492e. * Revert "gh-95299: Rework test_cppext.py to not invoke setup.py directly" This reverts commit 41c5a66. * Build and install the extension in a temporary directory instead * Pull in wheels for setuptools and wheel for testing extension builds

* main: pythongh-103479: [Enum] require __new__ to be considered a data type (pythonGH-103495) pythongh-103365: [Enum] STRICT boundary corrections (pythonGH-103494) pythonGH-103488: Use return-offset, not yield-offset. (pythonGH-103502) pythongh-103088: Fix test_venv error message to avoid bytes/str warning (pythonGH-103500) pythonGH-103082: Turn on branch events for FOR_ITER instructions. (python#103507) pythongh-102978: Fix mock.patch function signatures for class and staticmethod decorators (python#103228) pythongh-103462: Ensure SelectorSocketTransport.writelines registers a writer when data is still pending (python#103463) pythongh-95299: Rework test_cppext.py to not invoke setup.py directly (python#103316)

…101039) Remove the bundled setuptools wheel from ensurepip, and stop installing setuptools in environments created by venv. Co-Authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: C.A.M. Gerlach <[email protected]> Co-authored-by: Oleg Iarygin <[email protected]>

pradyunsg · 2023-04-18T04:45:38Z

https://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder is the final piece of work here -- that example needs to be updated to use something more modern or otherwise behave differently to showcase extending behaviours of the class.

…python#103316) * pythongh-95299: Rework test_cppext.py to not invoke setup.py directly * Add tests/cppextdata data to `TESTSUBDIRS` * Revert "Add tests/cppextdata data to `TESTSUBDIRS`" This reverts commit 635492e. * Revert "pythongh-95299: Rework test_cppext.py to not invoke setup.py directly" This reverts commit 41c5a66. * Build and install the extension in a temporary directory instead * Pull in wheels for setuptools and wheel for testing extension builds

…ripts

…H-103613)

* main: (24 commits) pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561) pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545) pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634) pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633) pythongh-102856: Initial implementation of PEP 701 (python#102855) pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589) pythongh-83004: Harden msvcrt further (python#103420) pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626) pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314) pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586) pythongh-103583: Always pass multibyte codec structs as const (python#103588) pythongh-103617: Fix compiler warning in _iomodule.c (python#103618) pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600) pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576) pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613) [Doc] Fix a typo in optparse.rst (python#103504) pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531) pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039) pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569) pythongh-67230: update whatsnew note for csv changes (python#103598) ...

This PR fixes GitHub actions for unit tests by adding setuptools as they are not pre-installed anymore. python/cpython#95299

* Update swift from branch 'master' to 9c38d756e0233eaa5f9e26066756eea9ce6a11ad - Merge "Use entry_points for server executables" - Use entry_points for server executables The old [files]scripts method of specifying executable Python scripts triggers some legacy easy-install-like mode for editable installs, which relies on pkg_resources. Recent versions of setuptools (67.5.0+) have started emitting warnings when importing pkg_resources, which in turn cause quite noticeable slowdowns in process startup. This is particularly prominant on py312, which stopped pre-installing (an often older version of) setuptools in new venvs. See also: - python/cpython#95299 - pypa/setuptools#3843 - pypa/setuptools#3966 Now, use [entry_points]console_scripts to specify these executables, which does not use pkg_resources in the generated script files. Change-Id: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83

The old [files]scripts method of specifying executable Python scripts triggers some legacy easy-install-like mode for editable installs, which relies on pkg_resources. Recent versions of setuptools (67.5.0+) have started emitting warnings when importing pkg_resources, which in turn cause quite noticeable slowdowns in process startup. This is particularly prominant on py312, which stopped pre-installing (an often older version of) setuptools in new venvs. See also: - python/cpython#95299 - pypa/setuptools#3843 - pypa/setuptools#3966 Now, use [entry_points]console_scripts to specify these executables, which does not use pkg_resources in the generated script files. Change-Id: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83

This does: 1. update python-3.12 to 3.12.5 which released yesterday 2. Replace the bundled wheels of pip and setuptools that are found in python with dependencies on * py3-pip-wheel for python 3.10, 3.11, 3.12 * py3-setuptools-wheel for python 3.10, 3.11 3. Add tests running 'python -m venv' and 'pip check' to make sure that the included versions are supported with the used python. 4. compile python with `--with-wheel-pkg-dir=/usr/share/python-wheels`. This enables ensurepip to find the wheels added in 2. Things to note: * python3.12 no longer has setuptools included python/cpython#95299 --------- Co-authored-by: Dimitri John Ledkov <[email protected]>

# What this PR does _tldr;_ I think we should install `setuptools` into our engine `Dockerfile` + in our CI env because Python 3.12 no longer installs `distutils` by default. This should unblock us from being able to merge #4656 and #4555. **More details** I would like to be able to merge #4656 and #4555. _However_, in both of these PRs `setuptools` is being removed from `requirements-dev.txt` ([here](https://github.com/grafana/oncall/pull/4555/files#diff-d8146d0816a943b0fa69a20399d7bbdb58e1c84c8b7933b2ba6dea7c10c410f5L113-L116) and [here](https://github.com/grafana/oncall/pull/4656/files#diff-d8146d0816a943b0fa69a20399d7bbdb58e1c84c8b7933b2ba6dea7c10c410f5L113-L116)). This leads to things breaking because of: ```bash File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/polymorphic/__init__.py", line 9, in <module> import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' ``` - https://github.com/grafana/oncall/actions/runs/9865348392/job/27242117474?pr=4555#step:5:98 - https://github.com/grafana/oncall/actions/runs/10078898966/job/27864920455?pr=4656#step:5:100 Python 3.12 made a change to no longer pre-install `distutils` ([relevant release notes](https://docs.python.org/3/whatsnew/3.12.html#:~:text=The%20third%2Dparty%20Setuptools%20package%20continues%20to%20provide%20distutils%2C%20if%20you%20still%20require%20it%20in%20Python%203.12%20and%20beyond)): > [PEP 632](https://peps.python.org/pep-0632/): Remove the distutils package. See [the migration guide](https://peps.python.org/pep-0632/#migration-advice) for advice replacing the APIs it provided. The third-party [Setuptools](https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html) package continues to provide distutils, if you still require it in Python 3.12 and beyond. > > [gh-95299](python/cpython#95299): Do not pre-install setuptools in virtual environments created with [venv](https://docs.python.org/3/library/venv.html#module-venv). This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the [activated](https://docs.python.org/3/library/venv.html#venv-explanation) virtual environment. Additionally, `setuptools` is in `pip-tools` `UNSAFE_PACKAGES` list ([related GitHub issue](pypa/pipenv#1417 (comment))), hence why I think Dependabot is removing it in #4656 and #4555. ## Checklist - [x] Unit, integration, and e2e (if applicable) tests updated - [x] Documentation added (or `pr:no public docs` PR label added if not required) - [x] Added the relevant release notes label (see labels prefixed w/ `release:`). These labels dictate how your PR will show up in the autogenerated release notes.

For parity with the `ensurepip` and `venv` modules, which no longer install setuptools as of Python 3.12: - python/cpython#95299 - python/cpython@ece20db Fixes indygreg#300.

For parity with the `ensurepip` and `venv` modules, which no longer install setuptools as of Python 3.12: - python/cpython#95299 - python/cpython@ece20db Fixes #300.

sbidoul added the type-feature A feature request or enhancement label Jul 26, 2022

sbidoul mentioned this issue Jul 26, 2022

Raise a warning when pip falls back to the "legacy" direct install code pypa/pip#8102

Closed

hugovk added the 3.12 bugs and security fixes label Jul 26, 2022

cjolowicz mentioned this issue Aug 28, 2022

Consider preparing venv and virtualenv sessions without setuptools and wheel wntrblm/nox#557

Open

pradyunsg mentioned this issue Sep 23, 2022

[POC] gh-85454: Extract and inject distutils from setuptools whl #95254

Closed

bedevere-bot mentioned this issue Jan 14, 2023

gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

Merged

pradyunsg added the topic-ensurepip label Feb 21, 2023

hugovk mentioned this issue Feb 24, 2023

gh-102202: Update bundled setuptools to 67.4.0 #102208

Closed

pradyunsg added a commit to pradyunsg/cpython that referenced this issue Apr 6, 2023

pythongh-95299: Rework test_cppext.py to not invoke setup.py directly

41c5a66

bedevere-bot mentioned this issue Apr 6, 2023

gh-95299: Rework test_cppext.py to not invoke setup.py directly #103316

Merged

pradyunsg added a commit to pradyunsg/cpython that referenced this issue Apr 6, 2023

Revert "pythongh-95299: Rework test_cppext.py to not invoke setup.py …

d79d6a6

…directly" This reverts commit 41c5a66.

pradyunsg added a commit to pradyunsg/cpython that referenced this issue Apr 9, 2023

pythongh-95299: Rework test_cppext.py to use explicitly in a tempdir

02d8a63

pradyunsg added a commit to pradyunsg/cpython that referenced this issue Apr 9, 2023

pythongh-95299: Rework test_cppext.py to use explicitly in a tempdir

846a148

bedevere-bot mentioned this issue Apr 18, 2023

gh-95299: Remove lingering setuptools reference in installer scripts #103613

Merged

zooba added a commit to zooba/cpython that referenced this issue Apr 18, 2023

pythongh-95299: Remove lingering setuptools reference in installer sc…

c909a07

…ripts

zooba added a commit that referenced this issue Apr 18, 2023

gh-95299: Remove lingering setuptools reference in installer scripts (G…

78cac52

…H-103613)

pfmoore mentioned this issue Apr 22, 2023

gh-103559: Update bundled pip version to 23.1.1 #103560

Merged

jirimoravcik mentioned this issue Jul 4, 2024

chore: add setuptools to opt deps apify/apify-shared-python#22

Merged

jirimoravcik added a commit to apify/apify-shared-python that referenced this issue Jul 4, 2024

chore: add setuptools to opt deps (#22)

a9a0bdc

This PR fixes GitHub actions for unit tests by adding setuptools as they are not pre-installed anymore. python/cpython#95299

Morikko mentioned this issue Jul 8, 2024

No module named 'pkg_resources' when using pipx owdevel/ytdl-nfo#30

Closed

jonathan343 mentioned this issue Jul 19, 2024

Fix CI for Python >= 3.12 boto/botocore#3223

Merged

Ch00k mentioned this issue Jul 24, 2024

Cant install on python 3.12 Ch00k/ffmpy#73

Closed

dac514 mentioned this issue Jul 25, 2024

Error trying to install toolkit with pip3 on Linux Scony/godot-gdscript-toolkit#301

Closed

jjhelmus mentioned this issue Jul 31, 2024

Pip 24.1.2 does not include wheel as a dependency. conda-forge/pip-feedstock#124

Closed

1 task

j4ger mentioned this issue Aug 6, 2024

'distutils' is no longer provided with python3 package nix-community/NUR#747

Closed

smoser mentioned this issue Aug 7, 2024

Fix/python ensurepip wheels wolfi-dev/os#25818

Merged

This was referenced Aug 8, 2024

Add Valkey support jd/pifpaf#175

Merged

pifpaf has a dependency on distutils (and current releases on pkg_resources as well) that are not installed by default on python 3.12 and above jd/pifpaf#176

Closed

joeyorlando mentioned this issue Aug 9, 2024

Fix missing setuptools dep grafana/oncall#4799

Merged

3 tasks

jrmfg mentioned this issue Aug 19, 2024

chore(deps): update dependency cloudevents to v1.11.0 GoogleCloudPlatform/python-docs-samples#12093

Merged

1 task

This was referenced Aug 21, 2024

Stop including setuptools and wheel in Python 3.12+ images docker-library/python#952

Closed

Stop bundling setuptools for Python 3.12+ indygreg/python-build-standalone#300

Closed

edmorley mentioned this issue Aug 24, 2024

Stop bundling setuptools for Python 3.12+ indygreg/python-build-standalone#301

Merged

sivanantha321 mentioned this issue Aug 29, 2024

ModuleNotFoundError: No module named 'pkg_resources' with Python 3.12 jpmml/jpmml-evaluator-python#27

Open

benoit74 mentioned this issue Sep 2, 2024

Library is not compatible with Python 3.12 noumar/iso639#16

Open

aodj mentioned this issue Sep 17, 2024

3.12 compat nickstenning/honcho#247

Closed

brokensound77 mentioned this issue Sep 17, 2024

[Bug] Dependency using a deprecated and removed module (pkg_resources) elastic/detection-rules#4083

Open

simonmysun mentioned this issue Sep 25, 2024

fix: Add missing dev dependencies: prebuildify and python package distutils in CI kelektiv/node.bcrypt.js#1050

Open

CertainLach mentioned this issue Sep 28, 2024

python3Packages.xen: init at 4.19.0 NixOS/nixpkgs#345171

Merged

13 tasks

levitsky mentioned this issue Oct 8, 2024

ImprtError on Python 3.12 due to missing pkg_resouces bigbio/sdrf-pipelines#175

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove bundled setuptools #95299

Remove bundled setuptools #95299

sbidoul commented Jul 26, 2022 •

edited by bedevere-bot

Loading

sbidoul commented Jul 26, 2022

pablogsal commented Jul 26, 2022

rafrafek commented Nov 7, 2022

pradyunsg commented Jan 14, 2023

edmorley commented Feb 23, 2023

FFY00 commented Mar 4, 2023

pradyunsg commented Apr 18, 2023

Remove bundled setuptools #95299

Remove bundled setuptools #95299

Comments

sbidoul commented Jul 26, 2022 • edited by bedevere-bot Loading

Feature or enhancement

Context

Previous discussion

Linked PRs

sbidoul commented Jul 26, 2022

pablogsal commented Jul 26, 2022

rafrafek commented Nov 7, 2022

pradyunsg commented Jan 14, 2023

edmorley commented Feb 23, 2023

FFY00 commented Mar 4, 2023

pradyunsg commented Apr 18, 2023

sbidoul commented Jul 26, 2022 •

edited by bedevere-bot

Loading