pip takes too long to resolve conflicting dependencies #9517
Comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Yes. All I described happens in Python 3.7.x |
|
This seems to be a case where the resolver is trying too hard to find a solution that doesn't exist. I think that's the case because of the warning at the end of the pip deptree that's been posted. I think this can be bundled into #9187, since nothing about this is specific to 21.0 AFAICT. |
pradyunsg
changed the title
|
It used to work with the old resolver. The current behavior is clearly a pip==21.0 issue. It does not happen with pip 20.x, for example, or older versions. If pip tries to find a solution and it does not exist, it should print a warning and solves it like before or by doing an educated guess, or something. There are several projects like ours. They use/depend on packages with a lack of information. We still must deploy to production. |
|
The old resolver “works” in the same sense that a random person assess situations faster than an expert. The rando guesses randomly, while the expert needs to actually analyse the situation. If the old resolver gets things right, you’re lucky. But that’s not a very useful baseline to evaluate performance. |
|
@uranusjr I am not saying pip must use the old resolver. I recognize the new resolver is predictable and better. Still, there are scenarios like the one I reported where this resolver does not work. It was running for 3 hours and could not install all the dependencies! Is it unreasonable to pick the best candidate package on those cases? The new resolver does not event report which package is unsolvable. It has to be a better solution for this. And we are not the only ones having problems. If picking one version of a package is not acceptable when the resolver cannot resolve, please suggest some alternative: print the package having issues? Restarting the installation using the old resolver? Being unable to install a project with new versions of pip does not seem a solution. |
|
I was trying to reproduce this example to find ways that pip resolver could be improved but it seems that is is no longer reproducible? I used Python 3.7.3 and Pip 21.0.1 with this from setuptools import setup
setup(
name="foo",
version="1.0",
install_requires=[
'spacy==2.2.3',
'sentence-transformers',
'sentencepiece==0.1.85',
'scispacy>=0.2.0',
'nltk',
'python-docx',
'langdetect',
'fuzzywuzzy==0.15.0',
'python-Levenshtein==0.12.0',
'annoy==1.16.3',
'pyDatalog',
'srsly==1.0.1',
'nest_asyncio==1.4.2']
)And it successfully built: Actually it's interesting that it installs I guess if the requirements explicitly state that version it gets installed anyway. |
|
Looks like langdetect 1.0.8 has a better dependency tree. Maybe changing it to |
Indeed, tried adding that but still installed fine. Rereading the initial report it looks like they didn't fully list their requirements as the first item in the log states It still installed fine, in fact reading the original post further I think there are many more packages that haven't listed, but reverse engineering their requirements from their logs is too laborious for me to try further. I thought I had a good idea for speeding up these kind of situations, but finding a good test case sure is hard! |
|
I'm encountering this issue on a very small From inspecting their dependencies manually, I can see very clearly that there is an incompatible dependency on
It looks like pip gets stuck backtracking on a whole bunch of different things. If there is no valid solution and if we don't think pip should install an invalid solution, then it should at least fail faster so I can move on. There are only three versions of
Expand for logs. I canceled it after 6 minutes. |
|
@uranusjr Maybe I'm being naive but it seems looking at the above example pip should be able to fail / resolve much faster. Here is the test case: from setuptools import setup
setup(
name="foo",
version="1.0",
install_requires=[
'spacy==2.3.5',
'spacy-pkuseg'
]
)Currently as explained above pip gets stuck here. By hand I can resolve this as impossible by doing the following:
In this example it would look like this:
Would this breadth first search through the requirements decency tree not be a much faster resolution in lots of failure cases? Or am I being naive in the scope of the problem or the complexity to implement? |
|
Breadth-first search is definitely an interesting proposal (and IIRC was raised before), but the idea of a “depth” is a bit more complicated past the most simplistic examples because the dependencies usually form a graph instead of a tree. Nowhere technically infeasible (not even that difficult) though, so I’d definitely encourage you to look into the possibility. The biggest issue to the approach in my mind is not about the implementation, but to somehow know whether it is an improvement. It obviously improves the example here, and likely many others showing a similar failure pattern, but would it worsen cases that run just fine right now? If it does, is the improvement enough to justify the degradation? How are we even supposed to put cases on a scale? Those are the more important questions to answer than the implementation itself. |
|
Yeah, I'm not sure that the breadth first search would be better in all/most cases. I think the main thing is collecting all packages where there can only be 1 possible version installed and then trying not to be "dumb" in the search methodology through the remaining packages (breadth first was my first idea but I have other ideas). Understood on the problems you've outlined, it's hard to know if you're making a net positive or a net negative or if you've made certain edge cases 1000x worse. But I have some ideas on testing this as well. I'll start taking a look at the |
|
Last I'd tried breadth first search, I'd found the results to be a bit surprising and mote difficult to reason about than depth first search. I also didn't find any package manager that approached the problem that way. That was 2 years ago tho. |
|
FYI I'm no longer able to reproduce the issue with: I've got a new optimization I am testing, so if anyone has a reproducible example let me know. |
|
@notatallshaw which version of pip are you using? I am encountering this issue :/ |
|
@daniellepintz They're working on making improvements to pip's behaviour in this area, and I believe that was a comment about the improved behaviour they're seeing on the local in-development copy of pip that they have. |
Do you have a reproducible set of requirements? As @pradyunsg mentions I am working on what should be a performance improvement when people encounter this issue, but I am always looking for more reproducible examples to confirm it actually works. |
|
Someone just posted an example here of azure-cli with Python 3.9 on Linux but then deleted the comment. I tried in a fresh virtual environment but wasn't albe to reproduce. In general I think this issue should be closed, all reported and reproducible examples were fixed by #10479 New examples should be new issues, and feel free to CC me and I'll check if I can reproduce and see what the issue is. |
What did you want to do?
We run the command:
on a CI node with no previous cache using a virtual environment.
There was NO cache and there were NO previous packages. We are using a new virtual environment
Python 3.7.3
setup.py
Output
Additional information
The text was updated successfully, but these errors were encountered: