Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip 20.3b1 bug with --pre for already installed package #9083

Closed
rgommers opened this issue Nov 1, 2020 · 6 comments · Fixed by #9085
Closed

pip 20.3b1 bug with --pre for already installed package #9083

rgommers opened this issue Nov 1, 2020 · 6 comments · Fixed by #9085
Labels
type: bug A confirmed bug or unintended behavior
Milestone
20.3

Comments

@rgommers
Copy link

rgommers commented Nov 1, 2020

Environment

  • pip version: 20.3b1
  • Python version: 3.8
  • OS: Linux

Description

After the release of pip 20.3b1 yesterday we started seeing issues on SciPy CI jobs that use pip install --pre --upgrade numpy. Here is a full build log: scipy/scipy#12729 (comment)

There's two exceptions actually, the version normalization seems also to be missing somewhere. But it's the first exception that's the more problematic one.

Expected behavior

Package should be upgraded without any errors.

How to Reproduce

Here is a standalone reproducer:

conda create -n piptest python=3.8 pip=19.3  # start from 19.3 (TravisCI default)
conda activate piptest
pip install -U pip --pre  # installs 20.3b1
pip install numpy==1.17.3  # install an older numpy version, so we can upgrade
pip install --pre --upgrade numpy  -i https://pypi.anaconda.org/scipy-wheels-nightly/simple

Output

Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Requirement already satisfied: numpy in /home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages (1.17.3)
ERROR: Exception:
Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 186, in _merge_into_criterion                                                                                                          
    crit = self.state.criteria[name]                                                                                          
KeyError: 'numpy'                                                                                                             

During handling of the above exception, another exception occurred:                                                           

Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 217, in _main                                                                                                                            
    status = self.run(options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper                                                                                                                           
    return func(self, options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(                                                                                       
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 119, in resolve                                                                                                            
    self._result = resolver.resolve(                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve                                                                                                                        
    state = resolution.resolve(requirements, max_rounds=max_rounds)                                                           
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 294, in resolve                                                                                                                        
    name, crit = self._merge_into_criterion(r, parent=None)                                                                   
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 188, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 89, in from_requirement
    if not candidates:
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 96, in __bool__
    return any(self)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 81, in __iter__
    candidates = _insert_installed(self._installed, self._get_others())
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 41, in _insert_installed
    candidates = sorted(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 198, in iter_index_candidates
    yield self._make_candidate_from_link(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 144, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 286, in __init__
    assert str(version) == wheel.version, (
AssertionError: <Version('1.20.0.dev0+20201101041405.5c37621')> != '1.20.0.dev0+20201101041405-5c37621' for wheel numpy
@rgommers rgommers mentioned this issue Nov 1, 2020
Merged
@uranusjr
Copy link
Member

uranusjr commented Nov 1, 2020

Thanks for catching this! According to PEP 440’s Local Version Segments section:

The normal form is using the . character. This allows versions such as 1.0+ubuntu-1 to be normalized to 1.0+ubuntu.1.

So we’re not doing the comparison correctly.

@uranusjr uranusjr added C: new resolver type: bug A confirmed bug or unintended behavior labels Nov 1, 2020
@uranusjr uranusjr added this to the 20.3 milestone Nov 1, 2020
@uranusjr uranusjr mentioned this issue Nov 1, 2020
Merged
@rgommers rgommers changed the title pip 20.3b1 bug with --pre for already install package pip 20.3b1 bug with --pre for already installed package Nov 1, 2020
@uranusjr
Copy link
Member

uranusjr commented Nov 1, 2020

For people following packaging discussions, this problem was actually discussed (theoretically at the time) in a recent thread: Escaping versions for wheel, sdist, and .dist-info names. The numpy wheel in question is escaping dot with an underscore, and the spec is vague on whether this should be done or not.

@rgommers
Copy link
Author

rgommers commented Nov 1, 2020

The numpy wheel in question is escaping dot with an underscore, and the spec is vague on whether this should be done or not.

It seems allowed, but either way it's not a numpy issue. The _ gets normalized in two different ways, once to . and once to -. Which then makes the != check fail.

@uranusjr it's not clear to me that that also explains the first exception - does it, or is that an unrelated resolver issue?

@uranusjr
Copy link
Member

uranusjr commented Nov 1, 2020

By the first exception I assume you mean KeyError: 'numpy'? That’s not a bug, but an expected exception. But the exception is handled like this (very simplified):

try:
    crit = self.state.criteria[name]
except KeyError:
    crit = _calculate_criteria_differently(...) # This eventually calls the above assertion.

And that handler blew up when it shouldn’t.

rgommers added a commit to scipy/scipy that referenced this issue Nov 1, 2020
@kousu @rgommers
DEP: remove setup_requires (#12729)
988d697 
* DEP: remove setup_requires.

This doesn't seem to work anymore.
Most people will be installing with pip, which will pick up
pyproject.toml and install the needed deps. For people who
want to keep using "python setup.py install", replace setup_requires
with a printed error.

Addresses #12727.

* CI: break up travis_retry pip install commands, pin pip version

Otherwise this installs pip 20.3b1 (unclear why), and that's broken in
combination with the numpy nightly wheels with an underscore in the name
(see pypa/pip#9083).

Co-authored-by: Ralf Gommers <[email protected]>
@xavfernandez xavfernandez mentioned this issue Nov 3, 2020
Closed
This was referenced Dec 1, 2020
Closed
Closed
@nagesh-chowdaiah
Copy link 

ERROR: Exception: 

Traceback (most recent call last): 

 File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion 

 crit = self.state.criteria[name] 

KeyError: 'cython'

still getting the error

@pradyunsg
Copy link
Member

@nagesh-chowdaiah what version of pip are you using?

@pypa pypa locked as resolved and limited conversation to collaborators Apr 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type: bug A confirmed bug or unintended behavior
Projects
None yet
Milestone
20.3
Development

Successfully merging a pull request may close this issue.

Use packaging.version to check version equality uranusjr/pip
4 participants
@rgommers @uranusjr @pradyunsg @nagesh-chowdaiah