-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Please provide a way to install pipenv without using vendored dependencies #1887
Comments
Reference for the way Most of the constraints and rationale there hold for |
Trying to replace some vendored packages with globally installed ones revealed interesting errors. E.g. when replacing Any pointers what to watch out for when tackling this issue? |
Note: I closed #1889 again, as it turns out I had misremembered how the |
@venthur |
@techalchemy Wait, I've removed the I suspect that something happens in |
@venthur I believe
When I test locally I usually test on a ramdisk and usually make a virtualenv to use for this. |
This is exactly what I'm doing. When I said I Maybe this diff makes a bit more clear what I'm doing: master...venthur:fix/unbundle (I've basically removed jinja2 and semver -- no specific reasons, gotta start somewhere) and added them to Then I followed the advice in $ cd pipenv
$ virtualenv ~/pipenv-venv # You can use a different path if you like.
$ source ~/pipenv-venv/bin/activate
$ python setup.py develop
$ pipenv install --dev
$ # To run the test suite locally::
$ pipenv run pytest tests and get the import error for semver. Calling $ pipenv run python -m semver works, however, which is the confusing part. |
@venthur I think I understand what's happening... Is this fixed if you delete |
@techalchemy unfortunately not. Same error as before. |
@venthur ok, this relates specifically to how pytest handles paths -- unless invoked with |
I'm fairly certain it has something to do how the new process is created in Here's a quick way to reproduce the problem:
|
@venthur as I mentioned, this is because |
Generally speaking,
You can see the last step is in the context of a new virtualenv, which doesn't have the originating environment's python path anymore which is why debundling things loses them. You'll find in |
Are there anything left to do on this one? |
Given there was no progress reported on this issue, but I haven't been looking at commit logs to see if there was something that just went unmentioned... my question would be if there's anything that got done. |
@eli-schwartz There had been a lot of effort to clean up patch generation and licensing issues around vendored libraries. I read like ten of them all related to this issue. I asked if there are anything left because I know there already are things done. |
Patch generation is related inasmuch as it's a prerequisite to effectively getting changes to stick, but I'm not seeing how it itself results in pipenv being able to function with devendored dependencies. I'd see it as a "dependent issue needing to be resolved before this one can be worked on", not "part of this issue". All it really says is "great, now we can finally get started on trying to make devendoring work". I'm not seeing how licensing is related at all. It's a completely separate issue that simply happened to be discovered during the course of someone's attempts to devendor pipenv. In fact, it's only when pip is vendored that there's licensing issues at all -- removing them (by devendoring them) automatically resolves the licensing issues, because the only thing that needs to be licensed is pipenv itself, as the devendored dependencies coming from the system already come with licenses. |
Hm not much to discuss here honestly. Licensing was the issue that had the domino effect of bringing in vendoring and patching tooling because of other distro maintainers needs. As for tooling to de-vendor our vendored dependencies, someone could port the pip tooling over if they were so inclined but it’s not really high priority for us to handle this ourselves right now |
Basically I don’t have any issue with having this, but I doubt we are going to build it so if distro maintainers want to have this the fastest route is going to be to PR in the tooling from pip |
If I understand @venthur's previous comments, pipenv doesn't use custom import paths at all, but relies on running some bits of its own code inside the pipenv-created virtualenv, so site-packages cannot be used, then inserting its own "vendor" and "patched" directories onto the path. I'm not sure how pip's tooling would help there. However, it looks like #1758 would fix this for the case mentioned earlier here, so I guess that would be something that gets us closer. |
@eli-schwartz that is just not correct, we have custom import paths all over the codebase. |
I've gone through the process of complete de-vendoring of IMO the first step should be more clarity between And a concerted effort needs to be put into upstreaming the functional-changes patches. I've found several patches which havent been provided to upstream, and sometimes I couldnt find an issue raised upstream. Probably there are good reasons for this, but it work to be done. Also there are a few functional changes which obviously are not acceptable upstream, such as the Anyone interested can find all The following are my notes wrt the patches in the 2018.11.26 release - some of these are now irrelevant, but packagers of 2018.11.26 release will still benefit from them. Most important is be3c771 , not a listed patch, replacing cursor with vistir due to recently uncovered legal problems with using GPL
There are also a few other patches possibly needed by de-vendoring |
Regarding vendored/patched, we switched to a more programatically patching workflow a while ago, so there’s functionally no differences between |
@uranusjr, but there are two distinct types of patches -- patches only needed for the vendoring to work, and patches which change the vendored package. |
Yes, I don’t object to your proposal of repurposing them to distinguish between different kinds of patching, my comment was merely exaplaining why they seem to lack clarity. |
I've mostly finished a second pass of de-vendoring, and anyone attempting it should be aware that there are bugs caused by the de-vendoring, which are not fully diagnosed, but do appear to be quite are serious, and they do not appear to be easily fixed. The only counter-point is RedHat has a partially de-vendored pipenv package, and I havent seen many complaints about it. You can see some of the problems at PRs #3645 and #3670 , and issue #3644 . In short, any Pipfile containing a reference to a package which is also a dependency of pipenv may be buggy when pipenv is de-vendored. Expected behaviour of pipenv is broken in many, many ways. This has so far only been verified to occur with As More analysis needed. Probably a good place to start is creating Pipfile tests which include If the goal of de-vendoring is only to avoid duplicates and licensing issues, IMO a better approach is to have the pipenv package depend on all of the vendored dependencies (so any change in dependencies triggers a rebuild), and then re-vendor the dependencies at build time so that the pipenv has a copy of the dependencies, but those copies are identical to the real package provided by the distro, and any fixes in the dependencies will be automatically replicated into pipenv. I've done this, and it avoids the problems mentioned above. The downside is that the pipenv package contents will be modified quite frequently, for any minor modification in any dependency, so the package version will be bumped up regularly, and users will be downloading new versions of (quite large) pipenv frequently. Another word of caution, any very minor bugs in packaging of dependencies can cause test failures, such as bad constraints in the egg-info |
Latest spec and patches at https://build.opensuse.org/package/show/home:jayvdb:coala:python3-bears/python-pipenv , with specs and patches for de-vendored dependencies all in https://build.opensuse.org/project/show/devel:languages:python . It is passing all unit and integration tests except for about eight that are explicitly de-selected with reasons given, and a few test scenarios at sarugaku/requirementslib#152 which are not fully understood yet. |
At this time, all |
As discussed in #1885 it is very hard for downstream (i.e. Debian) maintainers to package
pipenv
when it provides almost all its dependencies in thevendor
andpatched
directory.Quoting myself from #1885 (comment)
So in order to make life for downstream package maintainers easier, please provide a way -- similar to what
pip
does -- to installpipenv
"unbundeled" and thus using the packages installed on the target system.The text was updated successfully, but these errors were encountered: