Support --tree rendering from requirements.txt

[abitrolly]

Description

pipgrip --tree -r requirements/base.txt doesn't render a tree. It starts to install packages (and fails, because some dependencies are faulty). It would be nice if it didn't try to install anything unless an explicit --install is given. Or at least show partial dependency tree for specific package before installing it.

Use case / motivation

I am trying to troubleshoot who adds a failing dependency which is not immediately present in requirements.txt.

[ddelange]

Hi @abitrolly,

Could you paste the full output/traceback of your above command, including -vvv for more verbosity?

--install is decoupled: it will only run after dependency resolution succeeded (and the tree has been successfully printed).

Just guessing here: maybe your traceback is not from pipgrip trying to install a package, but rather occurs during dependency resolution? Like a traceback from pip trying to build a package?

pipgrip vendors this process out to pip, by calling something like pip wheel --no-deps some_package. When no wheels (.whl) are available for a package on PyPI.org, pip will resort to building the wheel from the source distribution (.tar.gz), which might need additional (system) dependencies installed on your system to succeed.

[abitrolly]

Without -v for now.

2:51:30 PM: + pipgrip --tree -v -r requirements/base.txt
2:52:33 PM: ERROR: Collecting psycopg2
2:52:33 PM:   Downloading psycopg2-2.8.6.tar.gz (383 kB)
2:52:33 PM:     ERROR: Command errored out with exit status 1:
2:52:33 PM:      command: /opt/buildhome/python3.7/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-jfhtm8jk/psycopg2_cec52e5ad15c4268bec741be29303445/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-jfhtm8jk/psycopg2_cec52e5ad15c4268bec741be29303445/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-7qyj29q4
2:52:33 PM:          cwd: /tmp/pip-wheel-jfhtm8jk/psycopg2_cec52e5ad15c4268bec741be29303445/
2:52:33 PM:     Complete output (23 lines):
2:52:33 PM:     running egg_info
2:52:33 PM:     creating /tmp/pip-pip-egg-info-7qyj29q4/psycopg2.egg-info
2:52:33 PM:     writing /tmp/pip-pip-egg-info-7qyj29q4/psycopg2.egg-info/PKG-INFO
2:52:33 PM:     writing dependency_links to /tmp/pip-pip-egg-info-7qyj29q4/psycopg2.egg-info/dependency_links.txt
2:52:33 PM:     writing top-level names to /tmp/pip-pip-egg-info-7qyj29q4/psycopg2.egg-info/top_level.txt
2:52:33 PM:     writing manifest file '/tmp/pip-pip-egg-info-7qyj29q4/psycopg2.egg-info/SOURCES.txt'
2:52:33 PM:     Error: pg_config executable not found.
2:52:33 PM:     pg_config is required to build psycopg2 from source.  Please add the directory
2:52:33 PM:     containing pg_config to the $PATH or specify the full executable path with the
2:52:33 PM:     option:
2:52:33 PM:         python setup.py build_ext --pg-config /path/to/pg_config build ...
2:52:33 PM:     or with the pg_config option in 'setup.cfg'.
2:52:33 PM:     If you prefer to avoid building psycopg2 from source, please install the PyPI
2:52:33 PM:     'psycopg2-binary' package instead.
2:52:33 PM:     For further information please check the 'doc/src/install.rst' file (also at
2:52:33 PM:     <https://www.psycopg.org/docs/install.html>).
2:52:33 PM:     ----------------------------------------
2:52:33 PM: ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
2:52:33 PM: Error: Command '['/opt/buildhome/python3.7/bin/python3.7', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/opt/buildhome/.cache/pip/wheels/pipgrip', '--index-url', 'https://pypi.org/simple', '--trusted-host', 'pypi.org', '--progress-bar=off', 'psycopg2']' returned non-zero exit status 1.

The requirements/base.txt.

# consciously added packages
Django==1.10.8
djangorestframework==3.4.0
pycountry==0.14.2
python-dateutil==2.6.0
wagtail==1.9
wagtail-pg-search-backend==1.3.2

# dependencies
beautifulsoup4==4.5.3
django-modelcluster==3.1
django-taggit==0.22.0
django-treebeard==4.0.1
html5lib==0.999999999
Pillow==3.4.2
pytz==2016.6.1
requests==2.25.1
six==1.10.0
Unidecode==0.4.19
urllib3==1.26.3
webencodings==0.5
Willow==0.4

[ddelange]

Can you try adding psycopg2-binary to your requirements.txt and running again?

[abitrolly]

The link to log with -vvv https://app.netlify.com/sites/filmfest/deploys/6047829454699f00087f2569#L85

Yes, I was going to add the -binary but needed to understand who invokes it. Give me a minute.

[ddelange]

Got it: wagtail-pg-search-backend requests psycopg2 (without binary), meaning that the dependency psycopg2 will have to be checked either way for dependency resolution, so my suggestion above won't avoid error.

You could try to open an issue with wagtail for changing their requirement to psycopg2-binary, as it's more common judging by pypistats.org and will arguably lead to less installation issues for their users in general.

I will close for now as it's a downstream issue. On pipgrip's side, a workaround to avoid building wheels for dependency resolution is currently still blocked by some open issues on PyPA, for more info see #40.

[abitrolly]

Yes, the process of building wheels is unexpected. I hoped it does static analysis without executing Python code.

Could the pipgrip at least give the partial graph before failing?

[abitrolly]

For the reference, I was able to get a graph for each dependency separately by removing all empty and comment lines from requirements.txt and feeding them through xargs.

cat requirements/base.txt | xargs -n 1 pipgrip --tree -v

[ddelange]

partial graph before failing

I'm afraid this is rather cumbersome, as the tree is built after the full resolution, and it happens recursively and exhaustively (so would error on missing metadata when being passed a partially resolved package_source).

For the reference, I was able to get a graph for each dependency separately by removing all empty and comment lines from requirements.txt and feeding them through xargs.

cat requirements/base.txt | xargs -n 1 pipgrip --tree -v

Interesting! On my machine, running pipgrip --tree wagtail-pg-search-backend fails with the same traceback you posted above.

[abitrolly]

It appeared I had psycopg2 installed when running the xargs command.

wagtail-pg-search-backend==1.3.2 (1.3.2)
├── django>=1.10 (3.1.7)
│   ├── asgiref<4,>=3.2.10 (3.3.1)
│   ├── pytz (2021.1)
│   └── sqlparse>=0.2.2 (0.4.1)
├── psycopg2 (2.8.6)
├── six (1.15.0)