Skip to content

gh-84530: fix namespace package support in modulefinder #29196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

gh-84530: fix namespace package support in modulefinder #29196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7610327
bpo-40350: fix namespace package support in modulefinder
FFY00 Oct 23, 2021
23fb4e0
consider that namespace package specs might have the loader set
FFY00 Oct 25, 2021
03a1384
Merge branch 'main' into bpo-40350
FFY00 Nov 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions Lib/importlib/_bootstrap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -752,17 +752,18 @@ def _init_module_attrs(spec, module, *, override=False):
loader = NamespaceLoader.__new__(NamespaceLoader)
loader._path = spec.submodule_search_locations
spec.loader = loader
# While the docs say that module.__file__ is not set for
# built-in modules, and the code below will avoid setting it if
# spec.has_location is false, this is incorrect for namespace
# packages. Namespace packages have no location, but their
# __spec__.origin is None, and thus their module.__file__
# should also be None for consistency. While a bit of a hack,
# this is the best place to ensure this consistency.
#
# See # https://docs.python.org/3/library/importlib.html#importlib.abc.Loader.load_module
# and bpo-32305
module.__file__ = None
if _bootstrap_external and isinstance(loader, _bootstrap_external.NamespaceLoader):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rescoping this change means it no longer falls under "A backward compatibility hack." Is that intentional?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the if loader is None block above still needed now that spec.loader is set in _bootstrap_external?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did do a deep dive into this and was meaning to write a more in-depth explanation. If I understand correctly, the backwards compatibility hack is instantiating the loader, not setting __file__. This function is meant to fill these missing attributes, but up until now it always expected to be the one instantiating the loader on namespace packages, which now changed with this PR. This should also trip up on 3rd party finders that set the loader for namespace packages, but so far no one has reported it yet.

# While the docs say that module.__file__ is not set for
# built-in modules, and the code below will avoid setting it if
# spec.has_location is false, this is incorrect for namespace
# packages. Namespace packages have no location, but their
# __spec__.origin is None, and thus their module.__file__
# should also be None for consistency. While a bit of a hack,
# this is the best place to ensure this consistency.
#
# See # https://docs.python.org/3/library/importlib.html#importlib.abc.Loader.load_module
# and bpo-32305
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably this comment should now also mention bpo-40350.

module.__file__ = None
try:
module.__loader__ = loader
except AttributeError:
Expand Down
1 change: 1 addition & 0 deletions Lib/importlib/_bootstrap_external.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1296,6 +1296,7 @@ def find_spec(cls, fullname, path=None, target=None):
# can create the namespace package.
spec.origin = None
spec.submodule_search_locations = _NamespacePath(fullname, namespace_path, cls._get_spec)
spec.loader = NamespaceLoader(fullname, namespace_path, cls._get_spec)
return spec
else:
return None
Expand Down
6 changes: 5 additions & 1 deletion Lib/modulefinder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
_PKG_DIRECTORY = 5
_C_BUILTIN = 6
_PY_FROZEN = 7
_NAMESPACE = 8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change feels wrong based on the comment above ("Old imp constants"), because it introduces a "new old constant". Perhaps the comment just needs to be replaced with something more accurate.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I share the same concern, but these constants are being used here to identify the module type, so maybe we can change the comment to "constants initially imported from imp module" or something similar.


# Modulefinder does a good job at simulating Python's, but it can not
# handle __path__ modifications packages make at runtime. Therefore there
Expand Down Expand Up @@ -66,7 +67,10 @@ def _find_module(name, path=None):

file_path = spec.origin

if spec.loader.is_package(name):
if isinstance(spec.loader, importlib.machinery.NamespaceLoader):
return None, spec.submodule_search_locations, ("", "", _NAMESPACE)

if spec.loader.is_package(name): # non-namespace package
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)

if isinstance(spec.loader, importlib.machinery.SourceFileLoader):
Expand Down
15 changes: 15 additions & 0 deletions Lib/test/test_modulefinder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,18 @@
from sys import version_info
"""]

namespace_package_test = [
"module",
["a", "module"],
["a.c", "blahblah"], [],
"""\
module.py
import a
import a.c
import blahblah
a/b.py
"""]

absolute_import_test = [
"a.module",
["a", "a.module",
Expand Down Expand Up @@ -353,6 +365,9 @@ def _do_test(self, info, report=False, debug=0, replace_paths=[], modulefinder_c
def test_package(self):
self._do_test(package_test)

def test_namespace_package(self):
self._do_test(namespace_package_test)

def test_maybe(self):
self._do_test(maybe_test)

Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2021-10-23-22-12-13.bpo-40350.t0dQMY.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix support for namespace packages in :mod:`modulefinder`.
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy