pip 20.3 release (Q4 2020)

[pradyunsg]

Another quarter! Another release!

We're definitely not skipping this release, because we have a new resolver to roll out -- and lots of really nice other enhancements as well.

[pradyunsg]

As always, need a release manager (RM) for this from @pypa/pip-committers.

I've self-assigned myself as the RM for this release (#7531 (comment)) -- as usual, if anyone is opposed to this, let me know. :)

[brainwane]

Per our discussion in our meeting today, @pradyunsg believes we should be able to make this release in the second half of October, assuming we don't find a showstopper in a bug report before then.

[brainwane]

I just spoke with @pradyunsg about the timing of the 20.3 release. Conclusion: @pradyunsg and I are leaning towards modifying the plans for 20.3 to:

• a pre-release this week
• a stable release in mid-November, around Nov. 11-12

Reasons: In terms of features and showstoppers, we are set to be ready to release this week. However, a few things have come up, just now, at the end of the 20.2 beta period:

• New resolver: Build automated testing to check for acceptable performance #8664 (comment) We need to nail down our policy and implementation re: Python 2 support.
• Azure macOS job is failing + "pip search" throttling #9030 and Add rate limiting to the XMLRPC Api pypi/warehouse#4321 (comment) We've just run into some CI blips/glitches that have gotten in our way and are not yet completely resolved.
• As Election Day in the United States (Tuesday, 3 November) approaches, we're seeing an increasing risk of civil unrest in the US (which may occur approaching, on, and immediately following 3 November), and we're considering the risk that a disruptive change to pip, released only a week or less before the election, would make it harder for civic tech developers/administrators and data scientists to use Python (in tools to assess/counter voting suppression and violence).

We planned an October release, in accordance with the usual release cadence, before the full scope of the pandemic and the US election situation emerged. But, as the release policy states,

The release manager may, at their discretion, choose whether or not there will be a pre-release period for a release, and if there is may extend that period into the next month if needed.

@pypa/pip-committers heads-up in case you are inclined to agree or disagree.

[brainwane]

Looks like we should expect the prerelease in the next couple days (cc @pradyunsg).

[brainwane]

We're being blocked (in making the 20.3.beta1 release); it's held up by a Python 2.7-specific CI failure that Pradyun is unable to reproduce locally. So 20.3.beta1 may not be out till Saturday.

https://dev.azure.com/pypa/pip/_build/results?buildId=29831&view=logs&j=f801bdd4-2ff4-5da8-3bd6-deea013b4470&t=89b17dec-01ee-56ca-032e-26fbc447775d&l=70

[pradyunsg]

Alrighty, that's resolved now (see #9019 for details, and a very-annoyed-at-CI-stuff Pradyun). I'll be making the release (20.3.beta1) in the next few hours!

[pradyunsg]

pip 20.3 beta1 is up on PyPI. get-pip.py has not been updated.

Happy Halloween folks! :)

[brainwane]

Here's a rough plan for publicizing this release (separate from the script or checklist for actually making the release and uploading it to PyPI and updating get-pip.py):

• blog post, to be cross-posted on pyfound.blogspot.com and blog.python.org (Ernest or other PSF staff can help post this to the PSF blog, and CPython core devs to the Python Insider blog, in case I'm not available) -- DONE, https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html and https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html
• post a somewhat detailed, technical release announcement to Discourse, and distutils-sig and python-dev mailing lists -- should include release highlights, links to the changelog and surveys, thanks to the funders, and a brief paragraph about what to expect in 21.0 -- DONE, https://mail.python.org/archives/list/[email protected]/thread/56ILRE26U4KU55YNVY2GQMGYYCEXPW74/ , https://discuss.python.org/t/announcement-pip-20-3-release/5948 , https://mail.python.org/archives/list/[email protected]/thread/OU6SUVG5EU6LOAVMROAAZO4AI2G6S7EJ/
• update pip needs a dependency resolver #988 and New Resolver: Rollout, Feedback Loops and Development Flow #6536 with GitHub comments announcing the release and pointing to the blog post
• update our meeting/update table on the wiki to add items about our blog post and Discourse announcement (Ernest or Ewa can do this)
• have twitter.com/ThePyPA and twitter.com/ThePSF tweet about the release (I think Dustin can do the former, and Betsy W. at the PSF can do the latter; give Betsy specific wording to tweet), linking to the blog post -- DONE in https://twitter.com/ThePyPA/status/1333400979409137664 (asked Betsy to retweet from the PSF account)
• post a LESS detailed release announcement (just the most crucial information, with a link to the blog post for more) to some further Python mailing lists:

1. pypi-announce
2. python-announce
3. wheel-builders
4. python-list
5. psf-community
6. psf-volunteers

• forward the more detailed release announcement to Python Weekly, Import Python, PyCoder's Weekly, The Changelog, and LWN
• also send the less detailed release announcement to some other mailing lists: NumFOCUS and the Carpentries discuss list -- DONE https://carpentries.topicbox.com/groups/discuss/T4bf0a9d08b83415f/python-users-important-changes-in-pip-20-3-dependency-resolver & https://groups.google.com/g/numfocus/c/D3JCVJuHQbU/m/PQXcDyPxBwAJ
• update the /topic in #pypa and #pypa-dev on Freenode IRC to reflect the release
• ask Betsy W. at PSF to forward our less detailed release announcement to PyPI and PSF sponsors
• post a note in the CZI-Essential Open Source for Science chat announcing the release and linking to the blog post
• reach out to Chinese-, Japanese-, Portugese- and Spanish-speaking contacts for
  1. Facebook/Telegram/Reddit outreach
  2. Brazil list(s) like https://groups.google.com/forum/#!forum/python-brasil
  3. language-specific lists like
  - https://mail.python.org/mailman/listinfo/python-de
  - https://mail.python.org/mailman/listinfo/python-es

[brainwane]

@pypa/pip-committers @pypa/pip-helpers @di Heads-up: we decided in today's meeting that we will probably be releasing 20.3 tomorrow or Friday. We'll of course know more tomorrow.

[brainwane]

Pradyun has been chasing down several issues that have proven nastier or more critical than they originally seemed, notably #9011. (See the milestone.) We may be able to cut the new release on Sunday or Monday. I would really hate for it to be delayed much longer than that, such as going into December, but if we release something with known regressions that will create hard-to-debug trouble for enough users, then that'll cause high support costs for us and lowered credibility for future releases.

[pelson]

@brainwane - just wanted to say that although sometimes it might feel like you are speaking to the void with these updates, in reality they are extremely helpful for downstream planning and long may they continue! Keep up the great work (and to all the pip devs!) 👍

[duckinator]

I 100% agree with @pelson. It's so nice that I can just go to the issue tracker, find an issue about the pending release, and just... learn exactly what the state of it is. I can't think of any other project that does that so reliably. 🙂

[webknjaz]

Sounds like it's fixed in the resolver upstream: #9011 (comment) 🎉

Have you considered having another beta/rc release before going final?

[joelberkeley-secondmind]

ok, thanks. But it isn't necessary if the whole workflow is pip install x y z; pip check?

[uranusjr]

If you start with an empty environment, yes.

[pradyunsg]

pip 20.3 has been released!

• Uploaded to PyPI.
• Pushed updates/tags to GitHub.
• get-pip.py has been updated, and the change will propagate over the next few minutes.

[groodt]

@pradyunsg @uranusjr and the many others involved. Thank you! What a huge achievement!

[brainwane]

We're planning a 20.3.1 release for some followup bugfixes.

I went most of the way through the release publicity checklist I made in #8936 (comment) .

I screwed up -- I forgot to mention in our announcements and other materials that the old resolver is still the default when using pip with Python 2.x.

I am going to:

• update the blog posts
• update the Discourse post
• send an update to distutils-sig
• send an update to python-list
• send an update to python-announce - deciding whether or not to do this
• send an update to wheel-builders
• update New Resolver: Rollout, Feedback Loops and Development Flow #6536
• send an update to python-dev
• see what needs updating in the user guide docs at pip.pypa.io and submit a pull request - done, Update docs to clarify that old resolver is default with Python 2 #9269

[pradyunsg]

There's a 20.3.2 release that I'd likely make this week (and likely a 20.3.3 release).

@pypa/pip-committers Please don't merge non-bugfix stuff until 20.3.2 is out, since I'd like to be lazy and cut the release directly off of the default branch instead of the tag. :)

[pradyunsg]

There's a 20.3.2 release that I'd likely make this week (and likely a 20.3.3 release).

Done!

• Uploaded to PyPI.
• Pushed updates/tags to GitHub.
• get-pip.py has been updated, and the change will propagate over the next few minutes.

---

@pypa/pip-committers consider the master branch to be open for pip 21.0 now. I'll cut the remaining bugfixes via cherry-picked commits.

[pradyunsg]

For anyone following along, pip 20.3.2 was yanked, and pip 20.3.3 will be released today, fixing the underlying issue that led to us yanking pip 20.3.2.

See #9293 for relevant details.

[hugovk]

As an aside, I'm a little surprised that pip install -U pip installed the yanked version (albeit with a warning):

$ pip install -U pip
Requirement already satisfied: pip in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (20.3.1)
Collecting pip
  Downloading pip-20.3.2-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 2.2 MB/s
WARNING: The candidate selected for download or install is a yanked version: 'pip' candidate (version 20.3.2 at https://files.pythonhosted.org/packages/3d/0c/01014c0442830eb38d6baef0932fdcb389279ce74295350ecb9fe09e048a/pip-20.3.2-py2.py3-none-any.whl#sha256=8d779b6a85770bc5f624b5c8d4d922ea2e3cd9ce6ee92aa260f12a9f072477bc (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*))
Reason for being yanked: <none given>
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.1
    Uninstalling pip-20.3.1:
      Successfully uninstalled pip-20.3.1
Successfully installed pip-20.3.2

The docs say:

A yanked release is a release that is always ignored by an installer, unless it is the only release that matches a version specifier (using either == or ===). See PEP 592 for more information.

https://pypi.org/help/#yanked

Is this a bug?

[pradyunsg]

Yup. That's a bug in the new resolver's handling in pip 20.3.1, that was fixed in 20.3.2. Sadly, pip 20.3.2 is borked in a different way. :)

[pradyunsg]

For anyone following eagerly -- #9302 is the PR for the 20.3.3 release. Once CI says "OK to go", I'll upload it. :)

[pradyunsg]

20.3.3 is out!

• Uploaded to PyPI.
• Pushed updates/tags to GitHub.
• get-pip.py has been updated, and the change will propagate over the next few minutes.

[pradyunsg]

21.0 is imminent, so I'll cut the 20.3.4 sometime in the next few days.

Note that (unless something really weird happens), 20.3.4 will be the last release of 20.3.* series, and the last Python 2-compatible release of pip.

[pradyunsg]

20.3.4 is out!

• Uploaded to PyPI.
• Pushed updates/tags to GitHub.
• get-pip.py has been updated, and the change will propagate over the next few minutes.

Calling that the end of 20.3's lifecycle. Expect a 21.0 today/tomorrow. :)