Feature: Adding dependencies through CLI

[ntjess]

What's the problem this feature will solve?

Some libraries allow many backing modules for the same functionality. Two very popular examples are OpenCv and Qt. The python module cv2 is provided by opencv-python-headless, opencv-python-contrib, opencv-python, and a few more pypi packages. Similarly, Qt APIs are available via PySide5, PySide6, etc.

Often, rather than adding an extra argument to pip installs, developers will simply check if one of many compatible libraries is available and raise an error otherwise. qtpy and pyqtgraph are popular examples of this: Qt is not specified in their requirements file; the user is expected to install this separately and the import fails if it isn't found. pip-compile doesn't handle these cases well -- currently, the only way requirements can be specified is through a requirements file.

TLDR: It is cumbersome to install packages that should exist in the user's environment, but did not come from requirements.txt.

Describe the solution you'd like

Ideally, pip-compile can support cli dependencies in its pip-args flag, since according to the docs:

Any valid pip flags or arguments may be passed on with pip-compile’s --pip-args option

pip-compile ./pyproject.toml --pip-args "pyside6 opencv-python-headless"
# Or a separate flag like --inline-requirements "opencv-python-headless"

This would allow pip-compile to handle these cases without forcing a requirements-tmp.txt for dependencies not managed or specified by the library at hand, and fits within the pip-args capabilities of adding cli dependencies.

Alternative Solutions

Devs can make an additional requirements file that specifies these needs, i.e.:

pip-compile pyproject.toml requirements-cv2.txt

which holds the relevant information. However, this leads to quite a bit of repo bloat if a separate additional requirements file must be specified for each similar (single) dependency. This is why e.g. packages like qtpy don't have an explicit requirements.txt file indicating this information.

Additional context

I would be happy to work on the PR if this idea is positively received.
The solution should be quite easy to implement using pip-compile's existing logic, by simply extending reqs to include these additional options. It has the nice benefit of still printing in the "autogenerated" message, so reproducibility will not be lost.

As a side note, the CLI already has an (undocumented) ability to accept requirements from stdin, so non-requirements.txt dependencies are already considered appropriate with existing logic.

As another side note, stdin dependencies are lost in the "autogeneration" comment! This solution has the added benefit of retaining those dependencies on multiple runs of pip-compile since stdin can simply be directed as a inline-requirements arg.

I.e. run echo pyside6 | pip-compile - -o requirements.txt and you will get a file with this comment at the top:

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile --output-file='.\requirements.txt' -

Note! The original stdin text, pyside6, is lost. This would not happen if it was stored as a CLI option.

[ntjess]

Another question -- it also seems like this functionality could be added to --upgrade-package. Currently, passing that argument with a package not in requirements.txt is not an error, but silently does not include it in the compiled output. Is this a bug? I.e. there should either be an error specifying an upgrade to a non-relevant package, or it should be included into the output. But no error + no inclusion seems like it should be classified as a bug

[AndydeCleyre]

Another question -- it also seems like this functionality could be added to --upgrade-package. Currently, passing that argument with a package not in requirements.txt is not an error, but silently does not include it in the compiled output. Is this a bug? I.e. there should either be an error specifying an upgrade to a non-relevant package, or it should be included into the output. But no error + no inclusion seems like it should be classified as a bug

You can see the discussions about this behavior at #759 and #1550, but the upshot is that --upgrade-package arguments are now treated like constraints.

[ntjess]

Thanks for the clarification. Looking at #759, thinking of -P as a constraint instead of a true "package dependency" is a good reason to raise no warning/error.

That makes me more on-board to use a new --add-package flag rather than repurposing -P. Is the feature proposed in #1742 still under consideration by the pip-tools team?

[vandmo]

This feature would also enable something like this in tox:

[testenv:pip-compile]
description = Compile requirements files
skip_install = true
deps =
    pip-tools==7.3.0
commands =
    pip-compile --upgrade --output-file requirements-format.txt --packages oitnb
    pip-compile --upgrade --output-file requirements-check.txt --packages flake8,flake8-bugbear,black
    pip-compile --upgrade --output-file requirements-test.txt --packages pytest,pytest-sugar

Today you are basically forced to have a requirements-x.in file since you can't do something like printf 'pytest\npytest-sugar' | pip-compile --upgrade --output-file requirements-test.txt since the | won't work.