pip 20.2 release (Q3 2020)

[pradyunsg]

Another quarter! Another release!

There has definitely been enough work done to justify a release (we have had funded developer time!).

[pradyunsg]

As always, we need a release manager for this release, from @pypa/pip-committers.

As has been the case for the past few releases (#7531 (comment)), I'd happy to be the release manager for this release as well. I'll be a bit more preemptive this time and self-assign myself as the RM for pip 20.2. Holler here if you're opposed to me being the RM. 😉

[pradyunsg]

Mapping the path from status quo, to where we want to be for the release:

• Update vendored packages for pip 20.2 release #8591 -- I need to update vendored packages
• Fail legacy install if .egg-info directory was not created #6998 -- a gone_in="20.2" deprecation that needs to be completed.
• Remove the pip search command #5216 (comment) -- no longer a blocker IMO.
• Fix the link on new-resolver dependency conflicts #8593 -- would be a good idea to not ship a broken link. :)

Other than that, looking at the new resolver work that's due for pip 20.2:

• Updating the message when the user hits a conflict w/ installed packages #8546 -- would be fixed by Improve install conflict warning #8594
• We still need a NEWS / Changelog entry for the new resolver presented behind --use-feature=2020-resolver.

(I think that's all?)

Right now, I think #6998 and the changelog entry are the only items that need to be done and haven't had someone take them up. @pypa/pip-team Any takers?

[sbidoul]

@pradyunsg we could add #8369 which is the easiest of the setup.py install deprecations. The others need more work.

[pradyunsg]

Sounds good to me!

[rouge8]

I'm hoping #8601 (or something similar) can be included to make it easier to use the new resolver 🤞

[pradyunsg]

Just did another pass with help from @brainwane. Looks like we should be on track for a release on Monday. :)

In terms of TODOs before the release, there's:

• finishing up Improve install conflict warning #8594 (which closes Updating the message when the user hits a conflict w/ installed packages #8546, wrapping up Implement step 1 in resolver rollout process #8513) -- that's a TODO for me.
• accepting/rejecting Support '--use-feature' in requirements files #8601 -- need @pypa/pip-committers' inputs/reviews on that. :)
• finishing up documentation/announcement prep for the new resolver.

[pradyunsg]

Thinking that I'd work over the weekend was a bad idea, especially since #8594 needed a lot more wrangling with the test suite than I had expected.

Once that and #8491 are merged, I'll make the release.

[brainwane]

@pradyunsg here's a draft release announcement for email/Discourse - what do you think? In particular, if you think the highlights are off, please speak up.

---

On behalf of the PyPA, I am pleased to announce that we have just released pip 20.2, a new version of pip. You can install it by running python -m pip install --upgrade pip.

The highlights for this release are:

• Implement PEP 376, REQUESTED metadata, so it can be easier to
  discover which distributions were installed by the user and which
  distributions were installed as dependencies

• Parallelize the networking operations done by pip list --outdated
  and pip list --uptodate

• Log debugging information about pip in pip install --verbose, and
  add a subcommand debug to pip config

• Allow specifying --prefer-binary option in a requirements file

• Install wheel files directly instead of extracting them to a temp
  directory

• Support --use-feature in requirements files

• The beta version of pip's next-generation dependency resolver

The new resolver is off by default because it is not yet ready for everyday use. Please test it with the --use-feature=2020-resolver flag. The new resolver is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please see our guide on how to test and migrate, and how to report issues. We are preparing to release pip 20.3, with the new resolver on by default, in October.

You can find more details (including deprecations and removals) in the changelog.

As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated.

Specific thanks go to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for their funding that enabled substantial work on the new resolver.

[xavfernandez]

Do we want to also ask for feedback on --use-feature=fast-deps that can be combined with --use-feature=2020-resolver ?

[pradyunsg]

Do we want to also ask for feedback on --use-feature=fast-deps that can be combined with --use-feature=2020-resolver ?

I think so. Let's add a couple of sentences about it. :)

[pradyunsg]

@brainwane and I collaborated on the email here. I think we're good to go -- now for me to make the release... after I eat breakfast. :)

[pradyunsg]

Made the release!

• Uploaded to PyPI
• Updated get-pip.py
• Posted on discuss.python.org
• Posted on distutils-sig

[pradyunsg]

And, because I'm thinking of automating this process everytime I go through it, I wrote a script and used that for making this release.

# make-release.sh

tag=$1

# be on latest master
nox -s prepare-release -- $tag
git push origin HEAD

echo "File the PR and merge it."
read -n 1

git pull upstream master --ff-only
git push origin $tag --no-verify
git push upstream $tag --no-verify

rm -r build/ dist/ || true
nox -s build-release -- $tag
nox -s upload-release -- $tag

cd ../get-pip

git checkout master
git reset --hard HEAD
git pull upstream master
pip install -r requirements.txt
curl -X PURGE https://pypi.org/simple/pip/
invoke generate
git add .
git commit -m "Update to $tag"
git tag $tag  # this should change to `git tag -s -m "$tag" $tag` in the future

git push origin master $tag
git push upstream master $tag

[pradyunsg]

@pypa/pip-committers There's no "embargo" on master -- it's open for merging PRs as usual. I'll cherry-pick bugfixes as needed.

[brainwane]

I just tweeted; also today I'll send a bunch of announcement emails and do other publicity.

Edited later to add -- some other time-sensitive stuff came up, so more announcements will be tomorrow.

[brainwane]

Pradyun already announced on Discourse and distutils-sig.

I've now announced:

• on pypi-announce
• on blog.python.org
• on python-announce
• on wheel-builders
• at error installing on ec2 deep learning ami napari/napari#631
• at test Piwheels with pip 20.2 piwheels/piwheels#239
• on the NumFOCUS list
• on python-list

I still need to do devpi-dev, python-dev, the psf-community & psf-volunteers lists, Carpentries-discuss, and the newsletters (Import Python, Python Weekly, and Pycoders weekly). I'll do those today.

There are many other venues I'd also like to hit but I think they'll have to wait till next week because I have other commitments today & tomorrow.

[pradyunsg]

Looks like we have at least a few reasons to make a bugfix release:

• Update documentation to reflect updated resolver feature flag #8660
• Canonicalize name in check_if_exists #8659
• Trace a better error message on installation failure due to invalid .data files in wheels #8656
• Ignore require-virtualenv in pip list #8603 (not a "reason", but worth including)

I'd like to wait over the weekend to give some more time for bug reports / PRs / reviews to roll in. Let me know if there's anything else I've missed that should be included in a bugfix release. If everything works out, I'll make a bugfix release on Monday. :)

[brainwane]

I've now announced on/to:

• devpi-dev
• python-dev
• psf-community
• Carpentries
• Import Python - done via email
• PyCoder’s Weekly - done via online form
• Python Weekly - already covered
• psf-volunteers (not yet published, I see)

[pradyunsg]

I just merged #8702 right now, so that leaves on #8687. :)

[jessecooper]

#8687 appears to be out of scope for a hotfix version and would be more qualified for a minor release.

[pradyunsg]

@jessecooper as I noted in #8687 (comment), it's a substantial enough improvement that I don't think it should wait for 3 months until 20.3.

[pradyunsg]

The discussion over at #8687 reached a concensus/conclusion. We're ready to go for 20.2.2. I'll cut the release in a bit. :)

[pradyunsg]

pip 20.2.2 has been released (#8746 for the release PR).

[jessecooper]

@pradyunsg I can understand not wanting to wait 3 months to get something out but regardless of how substantial the improvement I don't think it should delay the release of a hotfix for a regression as that has a larger impact to the overall Python community. Either way, thank you guys for getting this release out it is greatly appreciated.

[pradyunsg]

#8752 is a PR in line for a 20.2.3 release. I'll cut the release sometime tomorrow for it. Holler here if there's more stuff for the 20.2.3 bugfix. :)

[hugovk]

Something to consider for 20.2.3, but not urgent: #8778.

[pradyunsg]

I'mma cut 20.2.3 with #8778 and #8752 later tonight/early tomorrow. Let me know if anyone has any concerns with that. :)

[pradyunsg]

No one hollered, and I missed my reminder again. Oops.

[pradyunsg]

With #8853, I reckon this release cycle has run long enough and we can wrap it up now! Onward to 20.3!

[brainwane]

@pradyunsg this is my own fault for doing things in the wrong order, but per https://discuss.python.org/t/an-update-on-pip-and-dependency-resolution/1898/11 would it be easy to make a 20.2.4 point release, mostly for the sake of updating the front page of pip.pypa.io?

[pradyunsg]

Can do! Let me know which PRs you'd want to include on top of 20.2.3 (likely all documentation-only?) and I'll cut the 20.2.4 release with those updates. :)

[brainwane]

@pradyunsg Right - #8942 (not yet merged), #8807, and #8933. And if you've made any improvements to performance related to #8664 it's probably worth including those as well.

Or you could just merge #8942 and cut a release of master!

[brainwane]

In today's meeting @pradyunsg and I decided that

• there will be a 20.2.4 release sometime before Oct 15th, ideally sooner (before the 20.3 release pip 20.3 release (Q4 2020) #8936 which will probably be in the second half of the month)
• since there are some things in master that should not go out in a point release, 20.2.4 will be cut off 20.2.3 rather than off master
• 20.2.4 will bundle docs & performance improvements, including those listed above, and ideally the resolution to New resolver: Build automated testing to check for acceptable performance #8664 as well

[brainwane]

Per conversation with @pradyunsg just now -- things to include in 20.2.4:

docs:

• docs: Add how to default to new resolver #8942 (how to default to new resolver)
• Add ux docs to pip documentation #8807 (ux docs)
• Update docs index page about resolver and UX work #8933 (index page about resolver and UX work)

additional docs suggestions:

• Add documentation for netrc support #8829
• Add feature flag documentation #8561
• Document Python support policy #8927
• Architecture Documentation: CLI deep dive #7859
• Document fast-track deprecation policy #8873
• Update Caching documentation #8780
• Reformat a few spots in user guide #8795

performance:

• New resolver try user requested combinations first #8924
• Don't crash on 'check' when METADATA is missing #8926
• Cache found candidates #8912
• Pick up hashes from contraints files and intersect #8839

[pradyunsg]

#8990 is the 20.2.4 release PR. Some changes listed in the previous comment were already part of other 20.2 releases. The exact list for the 20.2.4 release is available in the PR.

[pradyunsg]

Alrighty! pip 20.2.4 has been out for 3 hours now, and seems to have been well received so far (i.e. no new bug reports). As mentioned, the changelog has the list of exact changed included in the bugfix, and so does #8990. :)

Gonna wrap this up proper, and focus on getting to out 20.3 later this month!