Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Allow Environment Override #1842

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ Windows and ``pythonX.Y -m piptools compile`` on other systems.
``pip-compile`` should be run from the same virtual environment as your
project so conditional dependencies that require a specific Python version,
or other environment markers, resolve relative to your project's
environment.
environment. If you need to resolve dependencies for a different environment,
see `Cross-environment`_ for some solutions.

**Note**: If ``pip-compile`` finds an existing ``requirements.txt`` file that
fulfils the dependencies then no changes will be made, even if updates are
Expand Down Expand Up @@ -529,6 +530,8 @@ We suggest to use the ``{env}-requirements.txt`` format
(ex: ``win32-py3.7-requirements.txt``, ``macos-py3.10-requirements.txt``, etc.).


.. _Cross-environment:

Cross-environment usage of ``requirements.in``/``requirements.txt`` and ``pip-compile``
=======================================================================================

Expand All @@ -539,7 +542,7 @@ etc.). For an exact definition, refer to the possible combinations of `PEP 508
environment markers`_.

As the resulting ``requirements.txt`` can differ for each environment, users must
execute ``pip-compile`` **on each Python environment separately** to generate a
execute ``pip-compile`` **for each Python environment separately** to generate a
``requirements.txt`` valid for each said environment. The same ``requirements.in`` can
be used as the source file for all environments, using `PEP 508 environment markers`_ as
needed, the same way it would be done for regular ``pip`` cross-environment usage.
Expand All @@ -551,8 +554,30 @@ dependencies, making any newly generated ``requirements.txt`` environment-depend
As a general rule, it's advised that users should still always execute ``pip-compile``
on each targeted Python environment to avoid issues.

There is a feature (``--override-environment``) that can be used to
specify the environment when gathering dependencies, allowing for cross-environment
fetching. However, a different ``requirements.txt`` must still be generated per
environment. It is recommended to override all keys in `PEP 508 environment markers`_
when targetting a different environment so the environment is fully defined.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to have a short example in the README.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an example for a 'typical' Linux machine


.. _PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers

For example, if you wanted to evaluate ``requirements.in`` for a typical Linux machine:

.. code-block:: bash

$ pip-compile requirements.in \
--override-environment os_name posix \
--override-environment sys_platform linux \
--override-environment platform_machine x86_64 \
--override-environment platform_python_implementation CPython \
--override-environment platform_release '' \
--override-environment platform_version '' \
--override-environment python_version 3.11 \
--override-environment python_full_version 3.11.0 \
--override-environment implementation_name cpython \
--override-environment implementation_version 3.11.0

Other useful tools
==================

Expand Down
26 changes: 26 additions & 0 deletions piptools/scripts/compile.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,14 @@
from ..repositories.base import BaseRepository
from ..resolver import BacktrackingResolver, LegacyResolver
from ..utils import (
PEP508_ENVIRONMENT_MARKERS,
UNSAFE_PACKAGES,
dedup,
drop_extras,
is_pinned_requirement,
key_from_ireq,
parse_requirements_from_wheel_metadata,
validate_environment_overrides,
)
from ..writer import OutputWriter

Expand Down Expand Up @@ -302,6 +304,14 @@ def _determine_linesep(
help="Specify a package to consider unsafe; may be used more than once. "
f"Replaces default unsafe packages: {', '.join(sorted(UNSAFE_PACKAGES))}",
)
@click.option(
"--override-environment",
multiple=True,
type=(str, str),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add validation on the values that people can override? I would expect this to fail if I pass --override-environment foo bar to the CLI. If this can be shown in the help to tell the user what environment markers can be overridden, it would improve the experience as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added argument validation

help="Specify an environment marker to override."
"This can be used to fetch requirements for a different platform",
callback=validate_environment_overrides,
)
def cli(
ctx: click.Context,
verbose: int,
Expand Down Expand Up @@ -340,6 +350,7 @@ def cli(
emit_index_url: bool,
emit_options: bool,
unsafe_package: tuple[str, ...],
override_environment: dict[str, str],
) -> None:
"""
Compiles requirements.txt from requirements.in, pyproject.toml, setup.cfg,
Expand Down Expand Up @@ -429,6 +440,21 @@ def cli(
pip_args.extend(["--cache-dir", cache_dir])
pip_args.extend(right_args)

env_dict = dict(override_environment)
if len(env_dict) > 0:
# Since the environment is overriden globally, handle it here in the
# top level instead of within the resolver.
import pip._vendor.packaging.markers

default_env = pip._vendor.packaging.markers.default_environment()

def overriden_environment() -> dict[str, str]:
return {
k: env_dict.get(k, default_env[k]) for k in PEP508_ENVIRONMENT_MARKERS
}

pip._vendor.packaging.markers.default_environment = overriden_environment

repository: BaseRepository
repository = PyPIRepository(pip_args, cache_dir=cache_dir)

Expand Down
41 changes: 39 additions & 2 deletions piptools/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,25 @@
"--no-reuse-hashes",
}

# Only certain environment markers are allowed in requirement specifications.
# Validate that overrides use a valid marker in order to provide better debug
# feedback to the user.
PEP508_ENVIRONMENT_MARKERS = [
"os_name",
"sys_platform",
"platform_machine",
"platform_python_implementation",
"platform_release",
"platform_system",
"platform_version",
"python_version",
"python_full_version",
"implementation_name",
"implementation_version",
# Note that 'extra' is omitted here because that should be set at the wheel
# level, not the runtime level.
]


def key_from_ireq(ireq: InstallRequirement) -> str:
"""Get a standardized key for an InstallRequirement."""
Expand Down Expand Up @@ -389,13 +408,18 @@ def get_compile_command(click_ctx: click.Context) -> str:
else:
if isinstance(val, str) and is_url(val):
val = redact_auth_from_url(val)

if option.name == "pip_args_str":
# shlex.quote() would produce functional but noisily quoted results,
# e.g. --pip-args='--cache-dir='"'"'/tmp/with spaces'"'"''
# Instead, we try to get more legible quoting via repr:
left_args.append(f"{option_long_name}={repr(val)}")
quoted_val = repr(val)
elif isinstance(val, (tuple, list)):
quoted_val = " ".join([shlex.quote(str(v)) for v in val])
else:
left_args.append(f"{option_long_name}={shlex.quote(str(val))}")
quoted_val = shlex.quote(str(val))

left_args.append(f"{option_long_name}={quoted_val}")

return " ".join(["pip-compile", *sorted(left_args), *sorted(right_args)])

Expand Down Expand Up @@ -522,3 +546,16 @@ def parse_requirements_from_wheel_metadata(
markers=parts.markers,
extras=parts.extras,
)


def validate_environment_overrides(
_ctx: click.Context,
_param: str,
value: list[tuple[str, str]],
) -> list[tuple[str, str]]:
for key, _ in value:
if key not in PEP508_ENVIRONMENT_MARKERS:
raise click.BadParameter(
f"Override key '{key}' must be one of " f"{PEP508_ENVIRONMENT_MARKERS}!"
)
return value
Loading