Moving plugin namespaces from pkg_resources to importlib #9340

ThrawnCA · 2026-05-08T00:55:11Z

ThrawnCA
May 8, 2026

We're encountering some perplexing behaviour regarding Python namespaces when moving to CKAN 2.11.5, and I'm hoping that someone else will have insights.

We run two sites, with different plugin sets that both include ckanext-s3filestore, and one works fine on CKAN 2.11.5, while the other crashes when attempting to run any CLI command on 2.11.5, with a stack trace similar to:

07:56:09,183 INFO  [ckan.config.environment] Loading static files from public
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/ckan", line 8, in <module>
    sys.exit(ckan())
             ^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 120, in parse_args
    result = super().parse_args(ctx, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1644, in parse_args
    rest = super().parse_args(ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 130, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 139, in _add_ctx_object
    ctx.obj = CtxObject(path)
              ^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 56, in __init__
    self.app = make_app(raw_config)
               ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/config/middleware/__init__.py", line 29, in make_app
    load_environment(conf)
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/config/environment.py", line 68, in load_environment
    p.load_all()
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 148, in load_all
    load(*plugins)
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 164, in load
    service = _get_service(plugin)
              ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 298, in _get_service
    return ep.load()(name=plugin_name)
           ^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ckanext.s3filestore'

Investigation showed various workarounds that would resolve it, but we're still scratching our heads about exactly what is happening here.

If we undo commit 4557290 and restore the namespace declaration in ckanext/stats/__init__.py, then the error goes away.
Similarly, if we add a namespace declaration to one of our custom plugins, in a submodule of ckanext, or install a plugin that has such a declaration, then the error goes away. A namespace declaration in ckanext/__init__.py does not suffice. This appears to be why one of our two sites is working, as it installs ckanext-ytp-comments, ckanext-archiver, ckanext-harvest, and ckanext-datarequests, all of which have that declaration.
If we revert pysolr to version 3.9.0 (as was used in CKAN 2.11.4), then the error goes away.

Removing __init__.py from ckanext-s3filestore, however, to comply with importlib style, does not help.

Does anyone have more insight on what is happening and what is the proper solution to this?

ThrawnCA · 2026-05-08T12:26:11Z

ThrawnCA
May 8, 2026
Author

This appears to also affect ckanext-xloader: ckan/ckanext-xloader#278

0 replies

wardi · 2026-05-20T13:55:05Z

wardi
May 20, 2026
Maintainer

Which version of setuptools are you using?

After upgrading to setuptools 82 I needed to reinstall ckan and its extensions so that the pkgutil path in ckanext/__init__.py would be used:

ckan/ckanext/__init__.py

Lines 7 to 9 in c7effcf

    
           except ImportError: 
        
               import pkgutil 
        
               __path__ = pkgutil.extend_path(__path__, __name__)

I don't know why having a namespace declaration in a submodule would seem to fix the issue, that's really odd.

1 reply

ThrawnCA May 22, 2026
Author

Tried with setuptools 63 and 81, same symptoms.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moving plugin namespaces from pkg_resources to importlib #9340

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Moving plugin namespaces from pkg_resources to importlib #9340

Uh oh!

Uh oh!

ThrawnCA May 8, 2026

Replies: 2 comments · 1 reply

Uh oh!

ThrawnCA May 8, 2026 Author

Uh oh!

wardi May 20, 2026 Maintainer

Uh oh!

ThrawnCA May 22, 2026 Author

ThrawnCA
May 8, 2026

Replies: 2 comments 1 reply

ThrawnCA
May 8, 2026
Author

wardi
May 20, 2026
Maintainer

ThrawnCA May 22, 2026
Author