Add an upgrade-strategy option #3972

pradyunsg · 2016-09-16T16:12:41Z

Resolves #3871

Associate an upgrade-strategy with pip install --upgrade. The upgrade-
strategy determines how upgrading of the dependencies.

Provides 2 upgrade-strategies "eager" and "non-eager". "eager" upgrades
dependencies regardless of whether they currently satisfy the new parent
version-specification. "non-eager" upgrades dependencies if and only if
they do not satisfy the new parent version-specification.

PS: I've allowed write access to maintainers (there was this checkbox) on this branch but I don't think that would be necessary.

Pinging (sorry if this is noise): @njsmith @xavfernandez @pfmoore @dstufft @rbtcollins @rgommers @dholth

As before, I would like it if we discussed all the implementation related stuff on this PR and use #3871 for all the other related discussions.

pradyunsg · 2016-09-16T16:15:48Z

pip/commands/install.py

@@ -224,6 +236,14 @@ def run(self, options, args):
        if options.build_dir:
            options.build_dir = os.path.abspath(options.build_dir)

+        allowed_strategies = ["eager", "non-eager"]


I don't think we'll ever add any new ones but this list should probably be kept close to the code actually performing the upgrade-or-not decision and be visible from here.

Please suggest what would be the correct/appropriate place to put this list of allowed upgrade strategies?

You could instead just set add choices=["eager", "non-eager"] to the add_option call above. It should handle the error handling for you.

Yes! I was wondering if I'm missing something obvious there...

I don't think we'll ever add any new ones

--no-deps can also be considered an upgrade strategy.

--no-deps can also be considered an upgrade strategy.

For the sake of simplicity, not in this PR. ;P Maybe add a new issue for this?

pradyunsg · 2016-09-16T17:37:30Z

Travis failed on nightly for reasons I don't understand and pep8 because the test names are too long.

dstufft

This looks like it's pretty close to being able to land.

dstufft · 2016-09-17T17:08:56Z

pip/commands/install.py

@@ -224,6 +236,14 @@ def run(self, options, args):
        if options.build_dir:
            options.build_dir = os.path.abspath(options.build_dir)

+        allowed_strategies = ["eager", "non-eager"]


You could instead just set add choices=["eager", "non-eager"] to the add_option call above. It should handle the error handling for you.

dstufft · 2016-09-17T17:10:54Z

pip/commands/install.py

+        cmd_opts.add_option(
+            '--upgrade-strategy',
+            dest='upgrade_strategy',
+            default='non-eager',


As I mentioned in #3871 (comment) I think we'll want this to default to eager for this time and we'll make sure to file an issue to swap the default for the next X+1 release.

I'll push a commit for the same.

dstufft · 2016-09-17T17:18:19Z

You can ignore the Python 3.6/nightly errors. That's not a problem with this branch. I'm talking to the Travis folks about getting an upgraded version of that available. Worst case I'll disable the matrix line.

pradyunsg · 2016-09-17T18:29:32Z

@dstufft Thanks for the review.

I've made the changes that you suggested.

PS: Considering the size of the patch, I really think this should be squashed and merged...

pradyunsg · 2016-09-17T18:43:27Z

Oh, and if this merges today, I have some time everyday for a week free to tackle any random issue. Any that comes to your head?

dstufft · 2016-09-18T01:14:37Z

I generally squash merge changes anyways yea.

dstufft · 2016-09-18T01:15:07Z

I need to get the 3.6 stuff fixed, going to poke at that now. If you have a chance to write a change log entry for this that'd be great. Otherwise I'll do it before I merge.

pradyunsg · 2016-09-18T03:27:27Z

I think that the build is fixed. Should I rebase?

dstufft · 2016-09-18T03:50:48Z

Go for it!

Associate an upgrade-strategy with `pip install --upgrade`. The upgrade- strategy determines how upgrading of the dependencies. Provides 2 upgrade-strategies "eager" and "non-eager". "eager" upgrades dependencies regardless of whether they currently satisfy the new parent version-specification. "non-eager" upgrades dependencies if and only if they do not satisfy the new parent version-specification.

pradyunsg · 2016-09-18T06:26:07Z

Done! Build is passing.

rgommers · 2016-09-18T10:48:17Z

pip/commands/install.py

+            '--upgrade-strategy',
+            dest='upgrade_strategy',
+            default='eager',
+            choices=['non-eager', 'eager'],


You may want to consider whether "non-eager" is clear to a user. The original proposal of "only-if-needed" is not much longer. That's what the pip docs call it now: https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade

Makes sense. I'll make the change.

rgommers · 2016-09-18T11:04:52Z

pip/req/req_set.py

+            if best_installed:
+                skip_reason = 'already up-to-date'
+            elif self.upgrade_strategy == "non-eager":
+                skip_reason = 'not upgraded as not directly required'


This gets appended to "Requirement already" instead of to "Requirement". Now it looks a bit weird: Requirement already not upgraded as not directly required

and same two lines above, gives Requirement already already up-to-date

I missed that line. 😅 I thought I had changed it...

rgommers · 2016-09-18T11:07:25Z

pip/req/req_set.py

+            elif self.upgrade_strategy == "non-eager":
+                skip_reason = 'not upgraded as not directly required'
+            else:
+                skip_reason = 'already satisfied'


This can't happen - if the above two clauses are false, then the latest version is not installed and strategy is eager. Hence can't skip but need to upgrade.

Now that I think about it, yes. This is a need-to-upgrade. I don't know what reasoning I was applying earlier for this...

Yeah... I did that because of the assert on line 484. And because I could never hit that branch in any of my testing earlier, for the previous PR.

rgommers · 2016-09-18T11:10:38Z

Played around with this a little bit - behaves as expected. LGTM

pradyunsg · 2016-09-18T11:31:58Z

Thanks for the review @rgommers!

pradyunsg · 2016-09-18T12:29:21Z

The build broke for reasons not related to these changes (some 404 on hitting the web for generating docs).

pfmoore · 2016-09-18T13:11:24Z

I restarted the doc build

pfmoore

LGTM, one minor nit noted, which is fine left as is.

pfmoore · 2016-09-18T13:15:45Z

pip/req/req_set.py

                # For link based requirements we have to pull the
                # tree down and inspect to assess the version #, so
                # its handled way down.
                if not (self.force_reinstall or req_to_install.link):
                    try:
-                        finder.find_requirement(req_to_install, self.upgrade)
+                        finder.find_requirement(
+                            req_to_install, upgrade_allowed)


Won't upgrade_allowed always be true here (line 414)? I guess the same was true of self.upgrade before, though...

I wondered the same initially. Then I decided to keep it as it was with upgrade.

pfmoore · 2016-09-18T13:23:10Z

Weird, I reran the doc build twice, but it failed both times. I can fetch the offending file manually, though. Travis weirdness?

pradyunsg · 2016-09-18T14:24:55Z

Thanks for the review @pfmoore!

Travis weirdness?

I think so...

matthew-brett · 2016-09-18T20:53:16Z

Nice ! Many thanks for this.

pradyunsg · 2016-12-07T07:20:30Z

Would it be useful to update a section of the documentation as updated here: https://github.com/pypa/pip/pull/3806/files#diff-0fcaa96196be65c5d918fd5099ea19a8 to mention this new option?

rgommers · 2016-12-07T10:06:24Z

that makes sense to me. Later on (1-2 years?) the comments on history can be removed again, but for now they will be helpful

Follow up on pypa#3972 (comment)

pradyunsg · 2017-01-14T16:27:40Z

Made the PR: #4232

Follow up on #3972 (comment)

To prevent unnecessary upgrades of libraries. See pypa/pip#3972

lock · 2019-06-02T13:05:12Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

pradyunsg commented Sep 16, 2016

View reviewed changes

pradyunsg mentioned this pull request Sep 16, 2016

Make install command upgrade packages by default #3786

Closed

pradyunsg mentioned this pull request Sep 17, 2016

Add "upgrade" and "upgrade-all" commands #59

Closed

dstufft requested changes Sep 17, 2016

View reviewed changes

dstufft approved these changes Sep 18, 2016

View reviewed changes

pradyunsg added 4 commits September 18, 2016 09:55

Shorten test names, correct descriptions

61d8d6a

Delegate the checking of strategy to option parser

25d9866

Switch the default upgrade strategy

09fe699

pradyunsg force-pushed the non-eager-upgrades branch from 7c962d6 to 09fe699 Compare September 18, 2016 04:25

Correct the test for change in error message

6550631

rgommers reviewed Sep 18, 2016

View reviewed changes

pradyunsg added 2 commits September 18, 2016 16:59

Rename non-eager to only-if-needed

21c98ef

Remove the repetitive "already"

932ecd2

pfmoore approved these changes Sep 18, 2016

View reviewed changes

dstufft merged commit 807f879 into pypa:master Sep 18, 2016

arcivanov mentioned this pull request Sep 22, 2016

Uptake new PIP upgrade features pybuilder/pybuilder#409

Closed

xavfernandez mentioned this pull request Nov 13, 2016

Change install command's default behaviour to upgrade packages by default #3806

Closed

12 tasks

pradyunsg added a commit to pradyunsg/pip that referenced this pull request Jan 14, 2017

Update documentation to mention upgrade-strategy

950e54e

Follow up on pypa#3972 (comment)

pradyunsg mentioned this pull request Jan 14, 2017

Update documentation to mention upgrade-strategy #4232

Merged

saimn mentioned this pull request Jan 18, 2017

new_table() astropy/astropy#5704

Closed

xavfernandez pushed a commit that referenced this pull request Jan 27, 2017

Update documentation to mention upgrade-strategy

1b0df50

Follow up on #3972 (comment)

pradyunsg deleted the non-eager-upgrades branch February 7, 2017 04:41

gotgenes mentioned this pull request Apr 4, 2017

--upgrade-strategy unmentioned in Release Notes #4418

Closed

pradyunsg mentioned this pull request May 19, 2017

Switch default upgrade strategy to only-if-needed #4500

Merged

ncoghlan mentioned this pull request Jun 25, 2017

Add a pip-style --upgrade-strategy setting to pipenv lock? pypa/pipenv#418

Closed

guewen added a commit to guewen/docker-odoo-project that referenced this pull request Jan 4, 2018

Use pip development version

82b5287

To prevent unnecessary upgrades of libraries. See pypa/pip#3972

guewen mentioned this pull request Jan 4, 2018

Use pip development version camptocamp/docker-odoo-project#44

Merged

guewen added a commit to TDu/docker-odoo-project that referenced this pull request Jan 11, 2018

Use pip development version

9caa632

To prevent unnecessary upgrades of libraries. See pypa/pip#3972

pradyunsg added the C: upgrade The logic of upgrading packages label May 11, 2018

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 2, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 2, 2019

Add an upgrade-strategy option #3972

Add an upgrade-strategy option #3972

Conversation

pradyunsg commented Sep 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg commented Sep 16, 2016

dstufft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dstufft commented Sep 17, 2016

pradyunsg commented Sep 17, 2016

pradyunsg commented Sep 17, 2016

dstufft commented Sep 18, 2016

dstufft commented Sep 18, 2016

pradyunsg commented Sep 18, 2016

dstufft commented Sep 18, 2016

pradyunsg commented Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

rgommers commented Sep 18, 2016 • edited Loading

pradyunsg commented Sep 18, 2016

pradyunsg commented Sep 18, 2016

pfmoore commented Sep 18, 2016

pfmoore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pfmoore commented Sep 18, 2016

pradyunsg commented Sep 18, 2016

matthew-brett commented Sep 18, 2016

pradyunsg commented Dec 7, 2016

rgommers commented Dec 7, 2016

pradyunsg commented Jan 14, 2017

lock bot commented Jun 2, 2019

pradyunsg commented Sep 16, 2016 •

edited

Loading

pradyunsg commented Sep 18, 2016 •

edited

Loading

pradyunsg Sep 18, 2016 •

edited

Loading

pradyunsg Sep 18, 2016 •

edited

Loading

rgommers commented Sep 18, 2016 •

edited

Loading