Coverage breaks with implicit namespaces

Issue #456 resolved
Arcadiy Ivanov
created an issue

Under PEP-0420 there are implicit namespaces.

  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 119, in do_coverage
    failure = _build_coverage_report(project, logger, execution_name, execution_prefix, coverage, modules)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 183, in _build_coverage_report
    module_report_data = _build_module_report(coverage, module)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 159, in _build_module_report
    return ModuleCoverageReport(coverage._analyze(module))
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 889, in _analyze
    it = self._get_file_reporter(it)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 914, in _get_file_reporter
    file_reporter = PythonFileReporter(morf, self)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/python.py", line 101, in __init__
    if filename.endswith(('.pyc', '.pyo')):
AttributeError: module 'namespace' has no attribute 'endswith'

This happens when coverage instruments "namespace.package.module"

Comments (11)

  1. Ned Batchelder repo owner

    @Arcadiy Ivanov Thanks for this. I have no experience with namespace packages. The test in the GitHub pull request uses a generated module. Can you provide a way to create an actual namespace package to test against? I'd feel better if it were as real as possible.

  2. Arcadiy Ivanov reporter

    @Ned Batchelder I'll just quote the PEP-0420 here if you don't mind:

    During import processing, the import machinery will continue to iterate over each directory in the parent path as it does in Python 3.2. While looking for a module or package named "foo", for each directory in the parent path:
    
    If <directory>/foo/__init__.py is found, a regular package is imported and returned.
    If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform and whether the -O flag is specified. The list here is representative.
    If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent path.
    Otherwise the scan continues with the next directory in the parent path.
    
    If the scan completes without returning a module or package, and at least one directory was recorded, then a namespace package is created. The new namespace package:
    
    Has a __path__ attribute set to an iterable of the path strings that were found and recorded during the scan.
    Does not have a __file__ attribute.
    

    The latter is the root cause of the problem if namespace is submitted as a module to be inspected/analyzed.

  3. Ned Batchelder repo owner

    Thanks for the pointer to the PEP, I hadn't realized Python 3.3 included this change. I tried this scenario with no problem:

    $ tree .
    .
    ├── main.py
    └── namespace
        └── package
            └── module.py
    
    2 directories, 2 files
    $ cat main.py
    import namespace.package.module as m
    
    print("Var is {}".format(m.VAR))
    $ cat namespace/package/module.py
    VAR = 17
    
    $ python3.5 main.py
    Var is 17
    $ coverage-3.5 run main.py
    Var is 17
    $ coverage-3.5 report -m
    Name                          Stmts   Miss  Cover   Missing
    -----------------------------------------------------------
    main.py                           2      0   100%
    namespace/package/module.py       1      0   100%
    -----------------------------------------------------------
    TOTAL                             3      0   100%
    

    If you could provide more details about exactly what you are trying that is failing, I would appreciate it.

  4. Arcadiy Ivanov reporter

    Here's the specific scenario I'm targeting (pybuilder):

    • Enumerate ALL modules in the the source directory
    • For every module enumerated, retrieve module analysis object.

    When you enumerate modules from the file system, unless you implement the above protocol exactly you don't know if a module is a namespace or a package or a module-proper. And if namespace is supplied directly, there is a blowout on coverage side.

    Please see the workaround in PyBuilder: https://github.com/pybuilder/pybuilder/pull/302

  5. Log in to comment