Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support namespace packages from command line #5759

Closed
gvanrossum opened this issue Oct 9, 2018 · 8 comments · Fixed by #9742
Closed

Support namespace packages from command line #5759

gvanrossum opened this issue Oct 9, 2018 · 8 comments · Fixed by #9742
Labels

Comments

@gvanrossum
Copy link
Member

gvanrossum commented Oct 9, 2018

Currently the --namespace-packages option only applies to imported modules. Files, modules and packages specified on the command line (as files or with -p or -m) cannot be in namespace packages. We should fix this.

Follow-up to #5591.

@gvanrossum
Copy link
Member Author

Update: -m honors --namespace-packages just fine. It's just -p and specifying files on the command line that don't. With -p I get Can't find package 'foo.bar' and when specifying the full path mypy assumes the top-level namespace (foo) is not a package at all.

@asottile
Copy link
Contributor

asottile commented Dec 2, 2018

I have a similar problem. Here's a minimal reproduction:

$ mypy --version
mypy 0.641
$ mkdir {a,b} && touch {a,b}/a.py
$ mypy a/a.py b/a.py --namespace-packages
b/a.py: error: Duplicate module named 'a'

I'm working around this by adding __init__.py files. Note that I'm invoking in this way as I'm using mypy with pre-commit via mirrors-mypy

@tsibley
Copy link

tsibley commented Jan 22, 2019

I'm running into this issue. While -m works with --namespace-packages, that means I have to specify every module in my package individually on the command-line; this is not nearly as nice as just running mypy foo.

tsibley added a commit to nextstrain/cli that referenced this issue Jan 22, 2019
nextstrain is a namespace package (no __init__.py), which mypy doesn't
yet support in imports if also named on the command-line.¹  While the
previous version of this code was correct and ran just fine, mypy threw
an error.

The namespace package is a) more modern Python 3 style and b) better for
future co-operation in the nextstrain.* namespace, so I wanted to keep
it instead of caving and adding an __init__.py just for mypy.  Instead,
import a little more messily from nextstrain.cli.__version__.

¹ See <python/mypy#5759>.
@gvanrossum
Copy link
Member Author

It's on our list of issue, but that list is rather long. Perhaps one of you is interested in submitting a PR to help this along more quickly?

@tsibley
Copy link

tsibley commented Jan 25, 2019

It seems as if @rafales started on addressing this: master...rafales:namespace-packages

I might be able to find the time to make a PR, but would appreciate some guidance on where to start in order to save myself a lot of bootstrapping time understanding how mypy is put together.

@russellwinstead
Copy link

It looks like it comes down to the options not being passed to the FindModuleCache object here:

mypy/mypy/main.py

Lines 783 to 784 in b027308

# TODO: use the same cache that the BuildManager will
cache = FindModuleCache(search_paths, fscache)

@rafales
Copy link

rafales commented Feb 3, 2019 via email

pcattori added a commit to pcattori/tamr-client that referenced this issue Mar 21, 2020
Packages should be typechecked with `--package` flag. Previously, a
typecheck error in attribute.py went unnoticed by mypy.

Additionally, for `--package` to work, __init__.py files are required
(at least for now, see python/mypy#5759 ).
pcattori added a commit to pcattori/tamr-client that referenced this issue Apr 4, 2020
Packages should be typechecked with `--package` flag. Previously, a
typecheck error in attribute.py went unnoticed by mypy.

Additionally, for `--package` to work, __init__.py files are required
(at least for now, see python/mypy#5759 ).
hauntsaninja pushed a commit to hauntsaninja/mypy that referenced this issue Oct 19, 2020
Fixes part of python#5759
The other part is passing files arguments, which we make steps towards
in python#9614
hauntsaninja added a commit that referenced this issue Oct 24, 2020
Fixes part of #5759
The other part is passing files arguments, which we make steps towards
in #9614

Co-authored-by: hauntsaninja <>
hauntsaninja added a commit that referenced this issue Dec 12, 2020
This is the successor to #9632. Things should basically be as discussed in that PR. Since #9616 is merged, this should now resolve #5759.

We leave the Bazel integration with `--package-root` almost entirely untouched, save for a) one change that's a bugfix / doesn't affect the core of what `--package-root` is doing, b) another drive by bugfix that's not related to this PR.
Change a) fixes the package root `__init__.py` hackery when passed absolute paths. Change b) fixes the validation logic for package roots above the current directory; it was broken if you passed `..` as a package root

Since we're leaving `--package-root` alone for now, I named the new flag `--explicit-package-base` to try and avoid confusion. Doing so also matches the language used by BuildSource a little better.

The new logic is summarised in the docstring of `SourceFinder.crawl_up`.

Some commentary:
- I change `find_sources_in_dir ` to call `crawl_up` directly to construct the BuildSource. This helps codify the fact that files discovered will use the same module names as if you passed them directly.
- Doing so keeps things DRY with the more complicated logic and means, for instance, that we now do more sensible things in some cases when we recursively explore directories that have invalid package names.
- Speaking of invalid package names, if we encounter a directory name with an invalid package name, we stop crawling. This is necessary because with namespace packages there's no guarantee that what we're crawling was meant to be a Python package. I add back in a check in the presence of `__init__.py` to preserve current unit tests where we raise InvalidSourceList.
- The changes to modulefinder are purely cosmetic and can be ignored (there's some similar logic between the two files and this just makes sure they mirror each other closely)
- One notable change is we now always use absolute paths to crawl. This makes the behaviour more predictable and addresses a common complaint: fixes #9677, fixes #8726 and others.
- I figured this could use more extensive testing than a couple slow cmdline tests. Hopefully this test setup also helps clarify the behaviour :-)

Co-authored-by: hauntsaninja <>
@gvanrossum
Copy link
Member Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
7 participants