Fails to fetch deps for numba==0.50.1

[abitrolly]

What you were trying to do (and why)

See the dependency branch for numba (which is a dependency it self).

pipgrip -vvv --tree numba==0.50.1

Or in a clean container.

podman run --rm -it python:3 bash -c "pip3 install pipgrip && ls -la && pipgrip -vvv --tree numba==0.50.1"

What happened (including command output

ModuleNotFoundError: No module named 'numpy'.

Command output

$ pipgrip -vvv --tree numba==0.50.1

DEBUG: pip version: [21, 2, 4]
DEBUG: pipgrip version: 0.6.10
INFO: discovering numba==0.50.1
DEBUG: Downloading/building wheel for numba==0.50.1 into cache_dir /root/.cache/pip/wheels/pipgrip
DEBUG: ['/usr/local/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/root/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'numba==0.50.1']
ERROR: Collecting numba==0.50.1
  Downloading numba-0.50.1.tar.gz (2.0 MB)
Building wheels for collected packages: numba
  Building wheel for numba (setup.py): started
  Building wheel for numba (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gj3k3z69
       cwd: /tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py", line 358, in <module>
      metadata['ext_modules'] = get_ext_modules()
    File "/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py", line 94, in get_ext_modules
      import numpy.distutils.misc_util as np_misc
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for numba
  Running setup.py clean for numba
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py", line 358, in <module>
      metadata['ext_modules'] = get_ext_modules()
    File "/tmp/pip-wheel-d69jbhr7/numba_85cab453a57940c8bc4abf92d91bb9f1/setup.py", line 94, in get_ext_modules
      import numpy.distutils.misc_util as np_misc
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed cleaning build dir for numba
Failed to build numba
ERROR: Failed to build one or more wheels

Error: Command '['/usr/local/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/root/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'numba==0.50.1']' returned non-zero exit status 1.

What you expected to happen

Fetch numpy with other dependencies and show the tree.

[ddelange]

Hi @abitrolly,

Will close this issue as duplicate of #40 (see #52 for another example of failing source build).

You can reproduce this issue (installing numba from tar.gz aka sdist) in a vanilla python environment by running pip install --no-binary :all: numba. The fact is that numba has hidden prerequisites for installing from source - namely numpy installed in the same environment. If you do so, the pipgrip command should start passing (it just runs pip in a subshell).

Hope that helps!

[abitrolly]

Hi @ddelange.

I am not sure it is a duplicate. There are no hidden prerequisites in numba - numpy is explicitly listed in install_requires

https://github.com/numba/numba/blob/e031b70c31c8553521ee89b02db8d2d6e995f9eb/setup.py#L360

Looking at the log, pip wheel is executed with --no-deps command. What is the purpose for that?

DEBUG: ['/usr/local/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/root/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'numba==0.50.1']

[ddelange]

I am not sure it is a duplicate. There are no hidden prerequisites in numba - numpy is explicitly listed in install_requires

install_requires specifies the additional packages required by the package during runtime, so after the package has been built. There is also a setup_requires key which will be temporarily fetched at an earlier build stage. Despite numba requirinng numpy for runtime, there will also have to be a version of numpy for build time. Your error most likely comes from a (late) build stage that tries to import numpy: https://github.com/numba/numba/blob/0.54.0/setup.py#L141

Looking at the log, pip wheel is executed with --no-deps command. What is the purpose for that?

DEBUG: ['/usr/local/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/root/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'numba==0.50.1']

The option ensures that pip ignores install_requires, and the command stops once the requested wheel has been built. We dont want pip to do the dependency resolving, but rather we just want the metadata for the package version requested at that specific stage during pipgrip's version solving:
https://github.com/ddelange/pipgrip/blob/0.6.10/src/pipgrip/pipper.py#L380

[ddelange]

Your error will not occur when using bdist_wheel (there are many wheels available for different processors etc, maybe just not for yours: https://pypi.org/project/numba/0.50.1/#files) instead of sdist, because source distributions often have extra requirements like this. See also source installation instructions for numba (they mention numpy needs to be installed to build): https://numba.readthedocs.io/en/stable/user/installing.html#installing-from-source

If there is a compatible wheel available for your system, pip (and therefore pipgrip) will always prefer it over source distributions unless explicitly specified like in my first comment above, and you won't have issues like this!

[abitrolly]

Ok. There is also setup_requires https://github.com/numba/numba/blob/e031b70c31c8553521ee89b02db8d2d6e995f9eb/setup.py#L359 that also specifies that numpy is needed for building the package.

The option ensures that pip ignores install_requires, and the command stops once the requested wheel has been built.

Are you sure that pip wheel --no-deps ignores install_requires and not setup_requires? What will happen if it is omitted?

We dont want pip to do the dependency resolving, but rather we just want the metadata for the package version requested at that specific stage during pipgrip's version solving:

Why not let pip do its dependency resolving if that is needed for it to build the wheel? The metadata for the wheel won't change, but there is no metadata if the wheel fails to build.

If there is a compatible wheel available for your system, pip (and therefore pipgrip) will always prefer it over source distributions unless explicitly specified like in my first comment above, and you won't have issues like this!

There is no compatible wheel, so here is the issue.

[ddelange]

Regarding --no-deps: I'm not at my machine right now, so I can't confirm whether pip install --no-binary :all: numba fails in a fresh env without numpy manually installed beforehand, but in any case --no-deps only controls what happens once the wheel has been successfully built. If it's not present, you'll see additional output once the numba wheel has successfully finished. If it's present, its an early exit.

Regarding setup_requires: if you can confirm the above fails with a numpy ModuleNotFoundError, it is probably an pip sdist issue (conda always has numpy pre-added to fresh envs) to open with numba: to only attempt the numpy import at a later build stage (after setup_requires related build_options have been run).

Are you sure that pip wheel --no-deps ignores install_requires and not setup_requires?

Yes pretty sure

Why not let pip do its dependency resolving if that is needed for it to build the wheel?

The depencency resolving of pip is decoupled from building packages afaik

Hope that helps!