Include modules in directories without `__init__.py`

[jstriebel]

Python allows importing modules from directories without __init__.py (implicit namespace packages per PEP 420). However, grimp doesn't find these modules, so import-linter (my actual use-case) silently excludes them from analysis.

This means contracts may not cover the entire codebase with no warning.

Note: Grimp already supports namespace packages for the cross-distribution case (e.g., google.cloud.logging). This issue is different: it's about directories within a regular package that lack __init__.py.

Example

mypackage/
├── __init__.py
├── core.py
├── a/               # no __init__.py
│   └── missing.py   # ← Python can import this
└── b/
    ├── __init__.py
    └── found.py

Python can import mypackage.a.missing:

>>> from mypackage.a import missing  # works

But grimp doesn't see it:

>>> import grimp
>>> graph = grimp.build_graph("mypackage")
>>> "mypackage.a.missing" in graph.modules
False

Suggested fix

Add an option (e.g., include_implicit_namespace_packages = true) to scan the filesystem for .py files in directories without __init__.py and include them in the graph.

If the above is too invasive, at least warn when .py files exist on disk but weren't included in analysis.

Reproduction

mkdir -p mypackage/a mypackage/b
touch mypackage/__init__.py mypackage/core.py
touch mypackage/a/missing.py
touch mypackage/b/__init__.py mypackage/b/found.py

import grimp

graph = grimp.build_graph("mypackage")
print(sorted(m for m in graph.modules))
# ['mypackage', 'mypackage.b', 'mypackage.b.found', 'mypackage.core']
# mypackage.a.missing is missing

Environment

• import-linter: 2.9
• Python: 3.11+

This might be directly related to seddonym/import-linter#295

[seddonym]

Thanks for raising this issue.

Grimp already supports PEP 420 namespace packages. From what I understand of the spec, mypackage.a is not technically a namespace package, it's just missing an __init__.py file.

If you were to delete the mypackage/__init__.py, or if you were to add mypackage/a/__init__.py, I think Grimp will work as you expect. Is there any reason not to do that (genuine question)?

[jstriebel]

From what I understand of the spec, mypackage.a is not technically a namespace package, it's just missing an init.py file.

There's some vagueness IMO. The rationale clearly talks about the cross-distribution case, whereas the spec just talks about packages, which also is the correct term for any subpackage, e.g. mypackage.a in this case is considered a namespace-package.

Is there any reason not to do that (genuine question)?

Python silently accepts it, so there's no feedback loop telling developers to fix it.

It's rare, consensus seems to be to add them. I've checked the most popular packages, and they use __init__.py very consistently most of the time. I've seen two use-cases so far:

1. Packages where the __init__.py was simply omitted or forgotten. For typical Python import use-cases this doesn't matter, so the omission often goes unnoticed.
   Example: databricks.sql.common, e.g. imported here. I've found ~13 relevant examples similar to this after checking 800 most downloaded PyPI packages with a script.
2. Packages that are not imported, but contain scripts. In most cases this lives in a top-level folder e.g. scripts, sometimes this is nested inside the library itself, and potential import-linter rules could/should arguably apply for those cases.
   Example: scipy.ndimage.utils containing a single script, where scipy.ndimage is a package that's imported regularly.

---

My main concern is that there's a difference in behavior between Python and Grimp, which can lead to Grimp silently missing relevant files. Do you tend towards automatically discovering these packages, and parsing them fully, warning on such cases, or ignoring them?

PS: A relevant tool to warn about this pattern is the flake8-no-pep420 linter, or the corresponding ruff rule.

[seddonym]

Sorry for delay in replying, I needed to think about it.

My concern with just turning it on is that sometimes packages may have deeply nested Python modules that are nothing to do with the package in question. For example, I work on a project where frontend assets are in folders inside the Python codebase. When they are built, the node_modules are added, and some node modules have Python files in. I just had a look at the codebase and there's a Python file at mypackage/static/node_modules/stylelint/node_modules/flatted/python/flatted.py. I don't think that module should show up as part of the Graph, at least not by default.

Another consideration is performance. It's helpful to be able to stop drilling down inside the file system if no further __init__.py files are found, if we change the behaviour then it could slow down building a graph (though perhaps by not enough to matter). And if we want to warn users about forgotten __init__.py files, we might need to explore the whole of a deep file system (say, of static files) just on the offchance of finding a mistake.

So where I'm leaning is an argument to build_graph to control the behaviour. Maybe something like: orphan_discovery, which could be passed "off", "warn" or "on". What do you think?

If you like that, I'd be happy to review a pull request.