Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip freeze fails to properly return hg URL if virtualenv is under Git controlled project #3988

Closed
ruiwen opened this issue Sep 26, 2016 · 7 comments · Fixed by #7593
Closed
Labels
auto-locked Outdated issues that have been locked by automation C: freeze 'pip freeze' related C: vcs pip's interaction with version control systems like git, svn and bzr state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior

Comments

@ruiwen
Copy link

ruiwen commented Sep 26, 2016

  • Pip version: 8.1.2
  • Python version: 2.7.12
  • Operating System: Ubuntu 16.04.1

Description:

I've got my virtulenv directory (venv) under my project folder

acme-server/
├── bin
├── acme
├── manage.py
├── Procfile
├── requirements
├── requirements.txt
├── venv
└── web

One of my requirements is installed via -e hg+https://, which ends up getting cloned to venv/src.

I'd like to automatically update my requirements.txt via pip freeze, but pip keeps returning my hg repo as a git URL instead.

I'd expect

-e hg+https://bitbucket.org/somedev/some-project@<some_projects_hash>#egg=some_project

but I keep getting this instead

-e [email protected]:Acme/acme-server.git@<my_projects_hash>#egg=some_project&subdirectory=venv/src/some_project

which isn't very helpful for my requirements.txt.

What I've run:

I dug into pip's source, and narrowed it down to controls_location() in pip/vcs/git.py.


   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
           r = cls().run_command(['rev-parse'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False

Turns out, running git rev-parse in the hg-cloned copy of some_project does not raise an error, because git traverses the tree upwards, eventually reaching the top level of my own project and finding the .git there instead. That would seem why <my_projects_hash> appears in the pip freeze output.

In pip/vcs::VcsSupport.get_backend_name(), it would appear the the git module is running first, and because git rev-parse is not raising an error from within the hg-cloned directory under venv/src, is claiming the hg-cloned project as a git repo instead (even before the hg module has a chance to run).

@ruiwen
Copy link
Author

ruiwen commented Sep 27, 2016

Using the --show-cdup option to git rev-parse in controls_location() seems to avoid my particular problem.

   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
-          r = cls().run_command(['rev-parse'],
+          r = cls().run_command(['rev-parse', '--show-cdup'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False

In the hg-cloned project, with git identifying the top-level project .git/,

$ pwd
/home/user/acme-server/venv/src/some-project
$ git rev-parse --show-cdup
../../..

In a git-cloned project under venv/src/

$ pwd
/home/user/acme-server/venv/src/some-git-project
$ git rev-parse --show-cdup

$

@xavfernandez xavfernandez added the C: vcs pip's interaction with version control systems like git, svn and bzr label Sep 29, 2016
@xavfernandez
Copy link
Member

Unfortunately, I don't see why --show-cdup would be of any help: git rev-parse --show-cdup still returns a valid path so pip will believe it found a git repo.

Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?
But even this heuristic would not always provide the right answer so...
We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...
cc @dstufft @pfmoore

@pfmoore
Copy link
Member

pfmoore commented Sep 29, 2016

Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

@ruiwen
Copy link
Author

ruiwen commented Sep 30, 2016

Unfortunately, I don't see why --show-cdup would be of any help: git rev-parse --show-cdup still returns a valid path so pip will believe it found a git repo.

Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?

This is what worked for me though. rev-parse --show-cdup returned a non zero-length path in the hg-cloned repo (vs a zero-length path in another git-cloned repo) under venv/src).

The return not r line after calling rev-parse --show-cdup then returned False for the git module's controls_location() in that scenario, which was correct.

Since the command is being run in the cloned project's directory, cwd=location, wouldn't a non zero-length return indicate that this is not a git repo (while potentially being in a git repo)?

We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...

Given that pip officially supports doing a pip install -e hg+https://..., shouldn't sanely returning the repo during pip freeze be a non-edge case?

Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

It might be that the "edge case" here is the location of the venv. There are other sites that do show examples of having the virtuelenv directory under the project folder though

@xavfernandez
Copy link
Member

This is what worked for me though. rev-parse --show-cdup returned a non zero-length path in the hg-cloned repo (vs a zero-length path in another git-cloned repo) under venv/src).

The return not r line after calling rev-parse --show-cdup then returned False for the git module's controls_location() in that scenario, which was correct.

Since the command is being run in the cloned project's directory, cwd=location, wouldn't a non zero-length return indicate that this is not a git repo (while potentially being in a git repo)?

My bad, I scanned the code too quickly.
git rev-parse should return nothing in a git repo, not the path to root dir.
Adding --show-cdup would break pip freeze for project installed via subdirectory of a git repo (as explained in https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support).

It might be that the "edge case" here is the location of the venv. There are other sites that do show examples of having the virtuelenv directory under the project folder though

I agree that having a venv in your project folder (and hence in a vcs repository) is a common usecase.

So maybe changing get_backend_name to test all VCS and return the one with the smallest subdirectory would be appropriate ? PR welcome :)

@ruiwen
Copy link
Author

ruiwen commented Oct 1, 2016

So maybe changing get_backend_name to test all VCS and return the one with the smallest subdirectory would be appropriate ?

I could take a stab at it when I have some free time from my primary project =) Would this be the preferred way about it?

@xavfernandez
Copy link
Member

xavfernandez commented Oct 4, 2016

Would this be the preferred way about it?

I don't think of any other way (as --show-cdup would break the subdirectory use case), so yes.

@xavfernandez xavfernandez added the C: freeze 'pip freeze' related label Oct 15, 2016
@pradyunsg pradyunsg added the type: bug A confirmed bug or unintended behavior label Jul 20, 2017
@chrahunt chrahunt added the state: awaiting PR Feature discussed, PR is needed label Aug 31, 2019
@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Mar 10, 2020
@lock lock bot locked as resolved and limited conversation to collaborators Mar 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation C: freeze 'pip freeze' related C: vcs pip's interaction with version control systems like git, svn and bzr state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants