-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Resolver: Be less eager about hitting the network #8905
Comments
I thought the networking implementation already handles caching on this? I recall reviewing a PR to add |
Output from running w/ /tmp/foo.txt is from #8664 (comment). Even though the page parsing code has lru_cache, what's happening is we're reaching out over the network even though we've seen the page, due to the way we're handling max_age on the pages in the download cache stuff. |
I'm not sure it's "just" network hits. The new resolver checks multiple times for the same requirement:
An example:
Also, putting a memory cache on |
That's expected, since it'll reach the requirement from multiple edges of the graph.
OH, huh. |
Adding a debugging reporter to resolvelib (one that does a
Notably, the file is empty when used without the 2020-resolver -- we're not hitting the network at all with the old resolver code. :) |
Consider the following:
Old resolver checks automat and its 2 dependencies.
So does new resolver.
Old resolver doesn't re-check six, because it "did it earlier". (Oversimplification, but basically accurate)
The new resolver does re-check. Now multiply that by 161 requirements in the actual example. That's why the old resolver has 160-odd lines and the new has twice as many. On my machine, the old resolver (160-odd checks) takes around 1-2 seconds. The new resolver with So I see two main components to the slowdown:
For (1), it's not surprising that the performance gets worse non-linearly. Dependency resolution algorithms are essentially exponential in performance, so big dependency graphs will be pretty nasty. That's fairly fundamental. The best we can likely do here is to work hard to improve the base performance so that the exponential factor doesn't hurt us quite so soon. But we're going to remain exponential. For (2), that's fundamentally the cost of correctness. We can't avoid it because the old resolver was "getting away with" simply dropping requirements. Unfortunately (!), it was often safe to do so, so the old resolver seemed to be fast and accurate in many cases (whereas it was actually just fast and lucky!)
OK, it looks like a significant component of (1) is the network hit. That means we probably can significantly improve that base performance by removing the network cost. |
Yup to everything you've said. :) |
Huh. Is there a way to configure requests so we cache responses during one pip command run, but drop the cache once the command ends? Similar to how the browser can set the cache header to “end of session” (iow each pip command run is a session, and cache clears when session ends). This is probably a good thing to do even out of the context of this performance problem—it would be very confusing if a new version becomes available right in the middle of a dependency resolution run. |
That's what I was hoping that putting a LRU cache on |
I think we prefer the already-installed version, when it satisfies a requirement except when upgrading. Looks like the entire "don't hit the network when the already installed version works" discussion was discussed in #8023. |
Slapping
Current performace for comparison:
Not nearly as good as the legacy resolver (which is 2s), but probably a good start? |
I think skipping the network requests would be give us a much bigger chunk of performance improvement -- but obviously both is best. :P |
Agreed. But the logic of caching Not calling |
Also the not calling |
192: Update pip to 20.2.4 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.3** to **20.2.4**. <details> <summary>Changelog</summary> ### 20.2.4 ``` =================== Deprecations and Removals ------------------------- - Document that certain removals can be fast tracked. (`8417 <https://github.com/pypa/pip/issues/8417>`_) - Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 <https://github.com/pypa/pip/issues/8927>`_) Features -------- - New resolver: Avoid accessing indexes when the installed candidate is preferred and considered good enough. (`8023 <https://github.com/pypa/pip/issues/8023>`_) - Improve error message friendliness when an environment has packages with corrupted metadata. (`8676 <https://github.com/pypa/pip/issues/8676>`_) - Cache package listings on index packages so they are guarenteed to stay stable during a pip command session. This also improves performance when a index page is accessed multiple times during the command session. (`8905 <https://github.com/pypa/pip/issues/8905>`_) - New resolver: Tweak resolution logic to improve user experience when user-supplied requirements conflict. (`8924 <https://github.com/pypa/pip/issues/8924>`_) Bug Fixes --------- - New resolver: Correctly respect ``Requires-Python`` metadata to reject incompatible packages in ``--no-deps`` mode. (`8758 <https://github.com/pypa/pip/issues/8758>`_) - New resolver: Pick up hash declarations in constraints files and use them to filter available distributions. (`8792 <https://github.com/pypa/pip/issues/8792>`_) - New resolver: If a package appears multiple times in user specification with different ``--hash`` options, only hashes that present in all specifications should be allowed. (`8839 <https://github.com/pypa/pip/issues/8839>`_) Improved Documentation ---------------------- - Add ux documentation (`8807 <https://github.com/pypa/pip/issues/8807>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]>
195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### 20.3 ``` - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 <https://github.com/pypa/pip/issues/8546>`_, `8377 <https://github.com/pypa/pip/issues/8377>`_) - Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 <https://github.com/pypa/pip/issues/6741>`_) - Warn if index pages have unexpected content-type (`6754 <https://github.com/pypa/pip/issues/6754>`_) - Allow specifying ``--prefer-binary`` option in a requirements file (`7693 <https://github.com/pypa/pip/issues/7693>`_) - Generate PEP 376 REQUESTED metadata for user supplied requirements installed by pip. (`7811 <https://github.com/pypa/pip/issues/7811>`_) - Warn if package url is a vcs or an archive url with invalid scheme (`8128 <https://github.com/pypa/pip/issues/8128>`_) - Parallelize network operations in ``pip list``. (`8504 <https://github.com/pypa/pip/issues/8504>`_) - Allow the new resolver to obtain dependency information through wheels lazily downloaded using HTTP range requests. To enable this feature, invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 <https://github.com/pypa/pip/issues/8588>`_) - Support ``--use-feature`` in requirements files (`8601 <https://github.com/pypa/pip/issues/8601>`_) Bug Fixes --------- - Use canonical package names while looking up already installed packages. (`5021 <https://github.com/pypa/pip/issues/5021>`_) - Fix normalizing path on Windows when installing package on another logical disk. (`7625 <https://github.com/pypa/pip/issues/7625>`_) - The VCS commands run by pip as subprocesses don't merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 <https://github.com/pypa/pip/issues/7968>`_) - Correctly treat non-ASCII entry point declarations in wheels so they can be installed on Windows. (`8342 <https://github.com/pypa/pip/issues/8342>`_) - Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 <https://github.com/pypa/pip/issues/8454>`_) - Headers provided by wheels in .data directories are now correctly installed into the user-provided locations, such as ``--prefix``, instead of the virtual environment pip is running in. (`8521 <https://github.com/pypa/pip/issues/8521>`_) Vendored Libraries ------------------ - Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3. - Upgrade appdirs to 1.4.4 - Upgrade certifi to 2020.6.20 - Upgrade distlib to 0.3.1 - Upgrade html5lib to 1.1 - Upgrade idna to 2.10 - Upgrade packaging to 20.4 - Upgrade requests to 2.24.0 - Upgrade six to 1.15.0 - Upgrade toml to 0.10.1 - Upgrade urllib3 to 1.25.9 Improved Documentation ---------------------- - Add ``--no-input`` option to pip docs (`7688 <https://github.com/pypa/pip/issues/7688>`_) - List of options supported in requirements file are extracted from source of truth, instead of being maintained manually. (`7908 <https://github.com/pypa/pip/issues/7908>`_) - Fix pip config docstring so that the subcommands render correctly in the docs (`8072 <https://github.com/pypa/pip/issues/8072>`_) - replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 <https://github.com/pypa/pip/issues/8353>`_) - Fix example for defining multiple values for options which support them (`8373 <https://github.com/pypa/pip/issues/8373>`_) - Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 <https://github.com/pypa/pip/issues/8459>`_) - Add feature flags to docs (`8512 <https://github.com/pypa/pip/issues/8512>`_) - Document how to install package extras from git branch and source distributions. (`8576 <https://github.com/pypa/pip/issues/8576>`_) ``` ### 20.3b1 ``` =================== Deprecations and Removals ------------------------- - ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version. To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Deprecate support for Python 3.5 (`8181 <https://github.com/pypa/pip/issues/8181>`_) - Document that certain removals can be fast tracked. (`8417 <https://github.com/pypa/pip/issues/8417>`_) - Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 <https://github.com/pypa/pip/issues/8927>`_) - Deprecate ``--find-links`` option in ``pip freeze`` (`9069 <https://github.com/pypa/pip/issues/9069>`_) Features -------- - Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Allow multiple values for --abi and --platform. (`6121 <https://github.com/pypa/pip/issues/6121>`_) - Add option ``--format`` to subcommand ``list`` of ``pip cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 <https://github.com/pypa/pip/issues/8355>`_) - Improve error message friendliness when an environment has packages with corrupted metadata. (`8676 <https://github.com/pypa/pip/issues/8676>`_) - Make the ``setup.py install`` deprecation warning less noisy. We warn only when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as situations where both fails are most probably irrelevant to this deprecation. (`8752 <https://github.com/pypa/pip/issues/8752>`_) - Check the download directory for existing wheels to possibly avoid fetching metadata when the ``fast-deps`` feature is used with ``pip wheel`` and ``pip download``. (`8804 <https://github.com/pypa/pip/issues/8804>`_) - When installing a git URL that refers to a commit that is not available locally after git clone, attempt to fetch it from the remote. (`8815 <https://github.com/pypa/pip/issues/8815>`_) - Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 <https://github.com/pypa/pip/issues/8892>`_) - Cache package listings on index packages so they are guarenteed to stay stable during a pip command session. This also improves performance when a index page is accessed multiple times during the command session. (`8905 <https://github.com/pypa/pip/issues/8905>`_) - New resolver: Tweak resolution logic to improve user experience when user-supplied requirements conflict. (`8924 <https://github.com/pypa/pip/issues/8924>`_) - Support Python 3.9. (`8971 <https://github.com/pypa/pip/issues/8971>`_) - Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 <https://github.com/pypa/pip/issues/8975>`_) - Switch to the new dependency resolver by default. (`9019 <https://github.com/pypa/pip/issues/9019>`_) - Remove the ``--build-dir`` option, as per the deprecation. (`9049 <https://github.com/pypa/pip/issues/9049>`_) Bug Fixes --------- - Propagate ``--extra-index-url`` from requirements file properly to session auth, so that keyring auth will work as expected. (`8103 <https://github.com/pypa/pip/issues/8103>`_) - Allow specifying verbosity and quiet level via configuration files and environment variables. Previously these options were treated as boolean values when read from there while through CLI the level can be specified. (`8578 <https://github.com/pypa/pip/issues/8578>`_) - Only converts Windows path to unicode on Python 2 to avoid regressions when a POSIX environment does not configure the file system encoding correctly. (`8658 <https://github.com/pypa/pip/issues/8658>`_) - List downloaded distributions before exiting ``pip download`` when using the new resolver to make the behavior the same as that on the legacy resolver. (`8696 <https://github.com/pypa/pip/issues/8696>`_) - New resolver: Pick up hash declarations in constraints files and use them to filter available distributions. (`8792 <https://github.com/pypa/pip/issues/8792>`_) - Avoid polluting the destination directory by resolution artifacts when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 <https://github.com/pypa/pip/issues/8827>`_) - New resolver: If a package appears multiple times in user specification with different ``--hash`` options, only hashes that present in all specifications should be allowed. (`8839 <https://github.com/pypa/pip/issues/8839>`_) - Tweak the output during dependency resolution in the new resolver. (`8861 <https://github.com/pypa/pip/issues/8861>`_) - Correctly search for installed distributions in new resolver logic in order to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 <https://github.com/pypa/pip/issues/8963>`_) - Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file with editable=True. Render it as a non-editable ``file://`` URL until modern editable installs are standardized and supported. (`8996 <https://github.com/pypa/pip/issues/8996>`_) Vendored Libraries ------------------ - Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed. It is mandatory for ``pip debug`` command. Improved Documentation ---------------------- - Add documentation for '.netrc' support. (`7231 <https://github.com/pypa/pip/issues/7231>`_) - Add OS tabs for OS-specific commands. (`7311 <https://github.com/pypa/pip/issues/7311>`_) - Add note and example on keyring support for index basic-auth (`8636 <https://github.com/pypa/pip/issues/8636>`_) - Added initial UX feedback widgets to docs. (`8783 <https://github.com/pypa/pip/issues/8783>`_, `8848 <https://github.com/pypa/pip/issues/8848>`_) - Add ux documentation (`8807 <https://github.com/pypa/pip/issues/8807>`_) - Update user docs to reflect new resolver as default in 20.3. (`9044 <https://github.com/pypa/pip/issues/9044>`_) - Improve migration guide to reflect changes in new resolver behavior. (`9056 <https://github.com/pypa/pip/issues/9056>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]>
195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### 20.3 ``` - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 <https://github.com/pypa/pip/issues/8546>`_, `8377 <https://github.com/pypa/pip/issues/8377>`_) - Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 <https://github.com/pypa/pip/issues/6741>`_) - Warn if index pages have unexpected content-type (`6754 <https://github.com/pypa/pip/issues/6754>`_) - Allow specifying ``--prefer-binary`` option in a requirements file (`7693 <https://github.com/pypa/pip/issues/7693>`_) - Generate PEP 376 REQUESTED metadata for user supplied requirements installed by pip. (`7811 <https://github.com/pypa/pip/issues/7811>`_) - Warn if package url is a vcs or an archive url with invalid scheme (`8128 <https://github.com/pypa/pip/issues/8128>`_) - Parallelize network operations in ``pip list``. (`8504 <https://github.com/pypa/pip/issues/8504>`_) - Allow the new resolver to obtain dependency information through wheels lazily downloaded using HTTP range requests. To enable this feature, invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 <https://github.com/pypa/pip/issues/8588>`_) - Support ``--use-feature`` in requirements files (`8601 <https://github.com/pypa/pip/issues/8601>`_) Bug Fixes --------- - Use canonical package names while looking up already installed packages. (`5021 <https://github.com/pypa/pip/issues/5021>`_) - Fix normalizing path on Windows when installing package on another logical disk. (`7625 <https://github.com/pypa/pip/issues/7625>`_) - The VCS commands run by pip as subprocesses don't merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 <https://github.com/pypa/pip/issues/7968>`_) - Correctly treat non-ASCII entry point declarations in wheels so they can be installed on Windows. (`8342 <https://github.com/pypa/pip/issues/8342>`_) - Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 <https://github.com/pypa/pip/issues/8454>`_) - Headers provided by wheels in .data directories are now correctly installed into the user-provided locations, such as ``--prefix``, instead of the virtual environment pip is running in. (`8521 <https://github.com/pypa/pip/issues/8521>`_) Vendored Libraries ------------------ - Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3. - Upgrade appdirs to 1.4.4 - Upgrade certifi to 2020.6.20 - Upgrade distlib to 0.3.1 - Upgrade html5lib to 1.1 - Upgrade idna to 2.10 - Upgrade packaging to 20.4 - Upgrade requests to 2.24.0 - Upgrade six to 1.15.0 - Upgrade toml to 0.10.1 - Upgrade urllib3 to 1.25.9 Improved Documentation ---------------------- - Add ``--no-input`` option to pip docs (`7688 <https://github.com/pypa/pip/issues/7688>`_) - List of options supported in requirements file are extracted from source of truth, instead of being maintained manually. (`7908 <https://github.com/pypa/pip/issues/7908>`_) - Fix pip config docstring so that the subcommands render correctly in the docs (`8072 <https://github.com/pypa/pip/issues/8072>`_) - replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 <https://github.com/pypa/pip/issues/8353>`_) - Fix example for defining multiple values for options which support them (`8373 <https://github.com/pypa/pip/issues/8373>`_) - Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 <https://github.com/pypa/pip/issues/8459>`_) - Add feature flags to docs (`8512 <https://github.com/pypa/pip/issues/8512>`_) - Document how to install package extras from git branch and source distributions. (`8576 <https://github.com/pypa/pip/issues/8576>`_) ``` ### 20.3b1 ``` =================== Deprecations and Removals ------------------------- - ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version. To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Deprecate support for Python 3.5 (`8181 <https://github.com/pypa/pip/issues/8181>`_) - Document that certain removals can be fast tracked. (`8417 <https://github.com/pypa/pip/issues/8417>`_) - Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 <https://github.com/pypa/pip/issues/8927>`_) - Deprecate ``--find-links`` option in ``pip freeze`` (`9069 <https://github.com/pypa/pip/issues/9069>`_) Features -------- - Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Allow multiple values for --abi and --platform. (`6121 <https://github.com/pypa/pip/issues/6121>`_) - Add option ``--format`` to subcommand ``list`` of ``pip cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 <https://github.com/pypa/pip/issues/8355>`_) - Improve error message friendliness when an environment has packages with corrupted metadata. (`8676 <https://github.com/pypa/pip/issues/8676>`_) - Make the ``setup.py install`` deprecation warning less noisy. We warn only when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as situations where both fails are most probably irrelevant to this deprecation. (`8752 <https://github.com/pypa/pip/issues/8752>`_) - Check the download directory for existing wheels to possibly avoid fetching metadata when the ``fast-deps`` feature is used with ``pip wheel`` and ``pip download``. (`8804 <https://github.com/pypa/pip/issues/8804>`_) - When installing a git URL that refers to a commit that is not available locally after git clone, attempt to fetch it from the remote. (`8815 <https://github.com/pypa/pip/issues/8815>`_) - Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 <https://github.com/pypa/pip/issues/8892>`_) - Cache package listings on index packages so they are guarenteed to stay stable during a pip command session. This also improves performance when a index page is accessed multiple times during the command session. (`8905 <https://github.com/pypa/pip/issues/8905>`_) - New resolver: Tweak resolution logic to improve user experience when user-supplied requirements conflict. (`8924 <https://github.com/pypa/pip/issues/8924>`_) - Support Python 3.9. (`8971 <https://github.com/pypa/pip/issues/8971>`_) - Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 <https://github.com/pypa/pip/issues/8975>`_) - Switch to the new dependency resolver by default. (`9019 <https://github.com/pypa/pip/issues/9019>`_) - Remove the ``--build-dir`` option, as per the deprecation. (`9049 <https://github.com/pypa/pip/issues/9049>`_) Bug Fixes --------- - Propagate ``--extra-index-url`` from requirements file properly to session auth, so that keyring auth will work as expected. (`8103 <https://github.com/pypa/pip/issues/8103>`_) - Allow specifying verbosity and quiet level via configuration files and environment variables. Previously these options were treated as boolean values when read from there while through CLI the level can be specified. (`8578 <https://github.com/pypa/pip/issues/8578>`_) - Only converts Windows path to unicode on Python 2 to avoid regressions when a POSIX environment does not configure the file system encoding correctly. (`8658 <https://github.com/pypa/pip/issues/8658>`_) - List downloaded distributions before exiting ``pip download`` when using the new resolver to make the behavior the same as that on the legacy resolver. (`8696 <https://github.com/pypa/pip/issues/8696>`_) - New resolver: Pick up hash declarations in constraints files and use them to filter available distributions. (`8792 <https://github.com/pypa/pip/issues/8792>`_) - Avoid polluting the destination directory by resolution artifacts when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 <https://github.com/pypa/pip/issues/8827>`_) - New resolver: If a package appears multiple times in user specification with different ``--hash`` options, only hashes that present in all specifications should be allowed. (`8839 <https://github.com/pypa/pip/issues/8839>`_) - Tweak the output during dependency resolution in the new resolver. (`8861 <https://github.com/pypa/pip/issues/8861>`_) - Correctly search for installed distributions in new resolver logic in order to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 <https://github.com/pypa/pip/issues/8963>`_) - Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file with editable=True. Render it as a non-editable ``file://`` URL until modern editable installs are standardized and supported. (`8996 <https://github.com/pypa/pip/issues/8996>`_) Vendored Libraries ------------------ - Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed. It is mandatory for ``pip debug`` command. Improved Documentation ---------------------- - Add documentation for '.netrc' support. (`7231 <https://github.com/pypa/pip/issues/7231>`_) - Add OS tabs for OS-specific commands. (`7311 <https://github.com/pypa/pip/issues/7311>`_) - Add note and example on keyring support for index basic-auth (`8636 <https://github.com/pypa/pip/issues/8636>`_) - Added initial UX feedback widgets to docs. (`8783 <https://github.com/pypa/pip/issues/8783>`_, `8848 <https://github.com/pypa/pip/issues/8848>`_) - Add ux documentation (`8807 <https://github.com/pypa/pip/issues/8807>`_) - Update user docs to reflect new resolver as default in 20.3. (`9044 <https://github.com/pypa/pip/issues/9044>`_) - Improve migration guide to reflect changes in new resolver behavior. (`9056 <https://github.com/pypa/pip/issues/9056>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]>
195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### 20.3 ``` - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 <https://github.com/pypa/pip/issues/8546>`_, `8377 <https://github.com/pypa/pip/issues/8377>`_) - Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 <https://github.com/pypa/pip/issues/6741>`_) - Warn if index pages have unexpected content-type (`6754 <https://github.com/pypa/pip/issues/6754>`_) - Allow specifying ``--prefer-binary`` option in a requirements file (`7693 <https://github.com/pypa/pip/issues/7693>`_) - Generate PEP 376 REQUESTED metadata for user supplied requirements installed by pip. (`7811 <https://github.com/pypa/pip/issues/7811>`_) - Warn if package url is a vcs or an archive url with invalid scheme (`8128 <https://github.com/pypa/pip/issues/8128>`_) - Parallelize network operations in ``pip list``. (`8504 <https://github.com/pypa/pip/issues/8504>`_) - Allow the new resolver to obtain dependency information through wheels lazily downloaded using HTTP range requests. To enable this feature, invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 <https://github.com/pypa/pip/issues/8588>`_) - Support ``--use-feature`` in requirements files (`8601 <https://github.com/pypa/pip/issues/8601>`_) Bug Fixes --------- - Use canonical package names while looking up already installed packages. (`5021 <https://github.com/pypa/pip/issues/5021>`_) - Fix normalizing path on Windows when installing package on another logical disk. (`7625 <https://github.com/pypa/pip/issues/7625>`_) - The VCS commands run by pip as subprocesses don't merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 <https://github.com/pypa/pip/issues/7968>`_) - Correctly treat non-ASCII entry point declarations in wheels so they can be installed on Windows. (`8342 <https://github.com/pypa/pip/issues/8342>`_) - Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 <https://github.com/pypa/pip/issues/8454>`_) - Headers provided by wheels in .data directories are now correctly installed into the user-provided locations, such as ``--prefix``, instead of the virtual environment pip is running in. (`8521 <https://github.com/pypa/pip/issues/8521>`_) Vendored Libraries ------------------ - Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3. - Upgrade appdirs to 1.4.4 - Upgrade certifi to 2020.6.20 - Upgrade distlib to 0.3.1 - Upgrade html5lib to 1.1 - Upgrade idna to 2.10 - Upgrade packaging to 20.4 - Upgrade requests to 2.24.0 - Upgrade six to 1.15.0 - Upgrade toml to 0.10.1 - Upgrade urllib3 to 1.25.9 Improved Documentation ---------------------- - Add ``--no-input`` option to pip docs (`7688 <https://github.com/pypa/pip/issues/7688>`_) - List of options supported in requirements file are extracted from source of truth, instead of being maintained manually. (`7908 <https://github.com/pypa/pip/issues/7908>`_) - Fix pip config docstring so that the subcommands render correctly in the docs (`8072 <https://github.com/pypa/pip/issues/8072>`_) - replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 <https://github.com/pypa/pip/issues/8353>`_) - Fix example for defining multiple values for options which support them (`8373 <https://github.com/pypa/pip/issues/8373>`_) - Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 <https://github.com/pypa/pip/issues/8459>`_) - Add feature flags to docs (`8512 <https://github.com/pypa/pip/issues/8512>`_) - Document how to install package extras from git branch and source distributions. (`8576 <https://github.com/pypa/pip/issues/8576>`_) ``` ### 20.3b1 ``` =================== Deprecations and Removals ------------------------- - ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version. To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Deprecate support for Python 3.5 (`8181 <https://github.com/pypa/pip/issues/8181>`_) - Document that certain removals can be fast tracked. (`8417 <https://github.com/pypa/pip/issues/8417>`_) - Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 <https://github.com/pypa/pip/issues/8927>`_) - Deprecate ``--find-links`` option in ``pip freeze`` (`9069 <https://github.com/pypa/pip/issues/9069>`_) Features -------- - Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Allow multiple values for --abi and --platform. (`6121 <https://github.com/pypa/pip/issues/6121>`_) - Add option ``--format`` to subcommand ``list`` of ``pip cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 <https://github.com/pypa/pip/issues/8355>`_) - Improve error message friendliness when an environment has packages with corrupted metadata. (`8676 <https://github.com/pypa/pip/issues/8676>`_) - Make the ``setup.py install`` deprecation warning less noisy. We warn only when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as situations where both fails are most probably irrelevant to this deprecation. (`8752 <https://github.com/pypa/pip/issues/8752>`_) - Check the download directory for existing wheels to possibly avoid fetching metadata when the ``fast-deps`` feature is used with ``pip wheel`` and ``pip download``. (`8804 <https://github.com/pypa/pip/issues/8804>`_) - When installing a git URL that refers to a commit that is not available locally after git clone, attempt to fetch it from the remote. (`8815 <https://github.com/pypa/pip/issues/8815>`_) - Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 <https://github.com/pypa/pip/issues/8892>`_) - Cache package listings on index packages so they are guarenteed to stay stable during a pip command session. This also improves performance when a index page is accessed multiple times during the command session. (`8905 <https://github.com/pypa/pip/issues/8905>`_) - New resolver: Tweak resolution logic to improve user experience when user-supplied requirements conflict. (`8924 <https://github.com/pypa/pip/issues/8924>`_) - Support Python 3.9. (`8971 <https://github.com/pypa/pip/issues/8971>`_) - Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 <https://github.com/pypa/pip/issues/8975>`_) - Switch to the new dependency resolver by default. (`9019 <https://github.com/pypa/pip/issues/9019>`_) - Remove the ``--build-dir`` option, as per the deprecation. (`9049 <https://github.com/pypa/pip/issues/9049>`_) Bug Fixes --------- - Propagate ``--extra-index-url`` from requirements file properly to session auth, so that keyring auth will work as expected. (`8103 <https://github.com/pypa/pip/issues/8103>`_) - Allow specifying verbosity and quiet level via configuration files and environment variables. Previously these options were treated as boolean values when read from there while through CLI the level can be specified. (`8578 <https://github.com/pypa/pip/issues/8578>`_) - Only converts Windows path to unicode on Python 2 to avoid regressions when a POSIX environment does not configure the file system encoding correctly. (`8658 <https://github.com/pypa/pip/issues/8658>`_) - List downloaded distributions before exiting ``pip download`` when using the new resolver to make the behavior the same as that on the legacy resolver. (`8696 <https://github.com/pypa/pip/issues/8696>`_) - New resolver: Pick up hash declarations in constraints files and use them to filter available distributions. (`8792 <https://github.com/pypa/pip/issues/8792>`_) - Avoid polluting the destination directory by resolution artifacts when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 <https://github.com/pypa/pip/issues/8827>`_) - New resolver: If a package appears multiple times in user specification with different ``--hash`` options, only hashes that present in all specifications should be allowed. (`8839 <https://github.com/pypa/pip/issues/8839>`_) - Tweak the output during dependency resolution in the new resolver. (`8861 <https://github.com/pypa/pip/issues/8861>`_) - Correctly search for installed distributions in new resolver logic in order to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 <https://github.com/pypa/pip/issues/8963>`_) - Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file with editable=True. Render it as a non-editable ``file://`` URL until modern editable installs are standardized and supported. (`8996 <https://github.com/pypa/pip/issues/8996>`_) Vendored Libraries ------------------ - Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed. It is mandatory for ``pip debug`` command. Improved Documentation ---------------------- - Add documentation for '.netrc' support. (`7231 <https://github.com/pypa/pip/issues/7231>`_) - Add OS tabs for OS-specific commands. (`7311 <https://github.com/pypa/pip/issues/7311>`_) - Add note and example on keyring support for index basic-auth (`8636 <https://github.com/pypa/pip/issues/8636>`_) - Added initial UX feedback widgets to docs. (`8783 <https://github.com/pypa/pip/issues/8783>`_, `8848 <https://github.com/pypa/pip/issues/8848>`_) - Add ux documentation (`8807 <https://github.com/pypa/pip/issues/8807>`_) - Update user docs to reflect new resolver as default in 20.3. (`9044 <https://github.com/pypa/pip/issues/9044>`_) - Improve migration guide to reflect changes in new resolver behavior. (`9056 <https://github.com/pypa/pip/issues/9056>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]>
194: Update pytest-pylint to 0.18.0 r=duckinator a=pyup-bot This PR updates [pytest-pylint](https://pypi.org/project/pytest-pylint) from **0.17.0** to **0.18.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest-pylint - Changelog: https://pyup.io/changelogs/pytest-pylint/ - Repo: https://github.com/carsongee/pytest-pylint </details> 197: Update pip to 20.3.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3.1**. <details> <summary>Changelog</summary> ### 20.3.1 ``` =================== Deprecations and Removals ------------------------- - The --build-dir option has been restored as a no-op, to soften the transition for tools that still used it. (`9193 <https://github.com/pypa/pip/issues/9193>`_) ``` ### 20.3 ``` - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 <https://github.com/pypa/pip/issues/8546>`_, `8377 <https://github.com/pypa/pip/issues/8377>`_) - Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 <https://github.com/pypa/pip/issues/6741>`_) - Warn if index pages have unexpected content-type (`6754 <https://github.com/pypa/pip/issues/6754>`_) - Allow specifying ``--prefer-binary`` option in a requirements file (`7693 <https://github.com/pypa/pip/issues/7693>`_) - Generate PEP 376 REQUESTED metadata for user supplied requirements installed by pip. (`7811 <https://github.com/pypa/pip/issues/7811>`_) - Warn if package url is a vcs or an archive url with invalid scheme (`8128 <https://github.com/pypa/pip/issues/8128>`_) - Parallelize network operations in ``pip list``. (`8504 <https://github.com/pypa/pip/issues/8504>`_) - Allow the new resolver to obtain dependency information through wheels lazily downloaded using HTTP range requests. To enable this feature, invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 <https://github.com/pypa/pip/issues/8588>`_) - Support ``--use-feature`` in requirements files (`8601 <https://github.com/pypa/pip/issues/8601>`_) Bug Fixes --------- - Use canonical package names while looking up already installed packages. (`5021 <https://github.com/pypa/pip/issues/5021>`_) - Fix normalizing path on Windows when installing package on another logical disk. (`7625 <https://github.com/pypa/pip/issues/7625>`_) - The VCS commands run by pip as subprocesses don't merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 <https://github.com/pypa/pip/issues/7968>`_) - Correctly treat non-ASCII entry point declarations in wheels so they can be installed on Windows. (`8342 <https://github.com/pypa/pip/issues/8342>`_) - Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 <https://github.com/pypa/pip/issues/8454>`_) - Headers provided by wheels in .data directories are now correctly installed into the user-provided locations, such as ``--prefix``, instead of the virtual environment pip is running in. (`8521 <https://github.com/pypa/pip/issues/8521>`_) Vendored Libraries ------------------ - Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3. - Upgrade appdirs to 1.4.4 - Upgrade certifi to 2020.6.20 - Upgrade distlib to 0.3.1 - Upgrade html5lib to 1.1 - Upgrade idna to 2.10 - Upgrade packaging to 20.4 - Upgrade requests to 2.24.0 - Upgrade six to 1.15.0 - Upgrade toml to 0.10.1 - Upgrade urllib3 to 1.25.9 Improved Documentation ---------------------- - Add ``--no-input`` option to pip docs (`7688 <https://github.com/pypa/pip/issues/7688>`_) - List of options supported in requirements file are extracted from source of truth, instead of being maintained manually. (`7908 <https://github.com/pypa/pip/issues/7908>`_) - Fix pip config docstring so that the subcommands render correctly in the docs (`8072 <https://github.com/pypa/pip/issues/8072>`_) - replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 <https://github.com/pypa/pip/issues/8353>`_) - Fix example for defining multiple values for options which support them (`8373 <https://github.com/pypa/pip/issues/8373>`_) - Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 <https://github.com/pypa/pip/issues/8459>`_) - Add feature flags to docs (`8512 <https://github.com/pypa/pip/issues/8512>`_) - Document how to install package extras from git branch and source distributions. (`8576 <https://github.com/pypa/pip/issues/8576>`_) ``` ### 20.3b1 ``` =================== Deprecations and Removals ------------------------- - ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version. To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Deprecate support for Python 3.5 (`8181 <https://github.com/pypa/pip/issues/8181>`_) - Document that certain removals can be fast tracked. (`8417 <https://github.com/pypa/pip/issues/8417>`_) - Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 <https://github.com/pypa/pip/issues/8927>`_) - Deprecate ``--find-links`` option in ``pip freeze`` (`9069 <https://github.com/pypa/pip/issues/9069>`_) Features -------- - Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 <https://github.com/pypa/pip/issues/4256>`_) - Allow multiple values for --abi and --platform. (`6121 <https://github.com/pypa/pip/issues/6121>`_) - Add option ``--format`` to subcommand ``list`` of ``pip cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 <https://github.com/pypa/pip/issues/8355>`_) - Improve error message friendliness when an environment has packages with corrupted metadata. (`8676 <https://github.com/pypa/pip/issues/8676>`_) - Make the ``setup.py install`` deprecation warning less noisy. We warn only when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as situations where both fails are most probably irrelevant to this deprecation. (`8752 <https://github.com/pypa/pip/issues/8752>`_) - Check the download directory for existing wheels to possibly avoid fetching metadata when the ``fast-deps`` feature is used with ``pip wheel`` and ``pip download``. (`8804 <https://github.com/pypa/pip/issues/8804>`_) - When installing a git URL that refers to a commit that is not available locally after git clone, attempt to fetch it from the remote. (`8815 <https://github.com/pypa/pip/issues/8815>`_) - Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 <https://github.com/pypa/pip/issues/8892>`_) - Cache package listings on index packages so they are guarenteed to stay stable during a pip command session. This also improves performance when a index page is accessed multiple times during the command session. (`8905 <https://github.com/pypa/pip/issues/8905>`_) - New resolver: Tweak resolution logic to improve user experience when user-supplied requirements conflict. (`8924 <https://github.com/pypa/pip/issues/8924>`_) - Support Python 3.9. (`8971 <https://github.com/pypa/pip/issues/8971>`_) - Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 <https://github.com/pypa/pip/issues/8975>`_) - Switch to the new dependency resolver by default. (`9019 <https://github.com/pypa/pip/issues/9019>`_) - Remove the ``--build-dir`` option, as per the deprecation. (`9049 <https://github.com/pypa/pip/issues/9049>`_) Bug Fixes --------- - Propagate ``--extra-index-url`` from requirements file properly to session auth, so that keyring auth will work as expected. (`8103 <https://github.com/pypa/pip/issues/8103>`_) - Allow specifying verbosity and quiet level via configuration files and environment variables. Previously these options were treated as boolean values when read from there while through CLI the level can be specified. (`8578 <https://github.com/pypa/pip/issues/8578>`_) - Only converts Windows path to unicode on Python 2 to avoid regressions when a POSIX environment does not configure the file system encoding correctly. (`8658 <https://github.com/pypa/pip/issues/8658>`_) - List downloaded distributions before exiting ``pip download`` when using the new resolver to make the behavior the same as that on the legacy resolver. (`8696 <https://github.com/pypa/pip/issues/8696>`_) - New resolver: Pick up hash declarations in constraints files and use them to filter available distributions. (`8792 <https://github.com/pypa/pip/issues/8792>`_) - Avoid polluting the destination directory by resolution artifacts when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 <https://github.com/pypa/pip/issues/8827>`_) - New resolver: If a package appears multiple times in user specification with different ``--hash`` options, only hashes that present in all specifications should be allowed. (`8839 <https://github.com/pypa/pip/issues/8839>`_) - Tweak the output during dependency resolution in the new resolver. (`8861 <https://github.com/pypa/pip/issues/8861>`_) - Correctly search for installed distributions in new resolver logic in order to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 <https://github.com/pypa/pip/issues/8963>`_) - Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file with editable=True. Render it as a non-editable ``file://`` URL until modern editable installs are standardized and supported. (`8996 <https://github.com/pypa/pip/issues/8996>`_) Vendored Libraries ------------------ - Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed. It is mandatory for ``pip debug`` command. Improved Documentation ---------------------- - Add documentation for '.netrc' support. (`7231 <https://github.com/pypa/pip/issues/7231>`_) - Add OS tabs for OS-specific commands. (`7311 <https://github.com/pypa/pip/issues/7311>`_) - Add note and example on keyring support for index basic-auth (`8636 <https://github.com/pypa/pip/issues/8636>`_) - Added initial UX feedback widgets to docs. (`8783 <https://github.com/pypa/pip/issues/8783>`_, `8848 <https://github.com/pypa/pip/issues/8848>`_) - Add ux documentation (`8807 <https://github.com/pypa/pip/issues/8807>`_) - Update user docs to reflect new resolver as default in 20.3. (`9044 <https://github.com/pypa/pip/issues/9044>`_) - Improve migration guide to reflect changes in new resolver behavior. (`9056 <https://github.com/pypa/pip/issues/9056>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]> Co-authored-by: Ellen Marie Dash <[email protected]>
Originally posted by @pradyunsg in #8664 (comment)
The text was updated successfully, but these errors were encountered: