Log an informational message when backtracking takes multiple rounds on a specific package

[pradyunsg]

Closes #8975
Toward #8713

Deviates from @ei8fdb's suggestion by using different messages when hitting the "trigger" numbers:

pip is looking at multiple versions of this package to determine
which version is compatible with other requirements.
This could take a while.

This is taking longer than usual. You might need to provide the
dependency resolver with stricter constraints to reduce runtime.
If you want to abort this run, you can press Ctrl + C to do so.

Further, this also doesn't constantly print the same message (based on some prototyping + having implemented this) and instead prints it only when the actual "trigger" number occurs.

[uranusjr]

LGTM. What’s wrong with the news bot?

[pfmoore]

There's no news file and it's not tagged as "trivial".

It may be worth actually having a news entry for this, it's an important user-visible change. SO I won't add the label myself, I'll leave it to @pradyunsg to decide.

[pradyunsg]

MacOS runs are being flaky.

[pradyunsg]

Okay, Azure has changed something in their MacOS images and that's causing our build-env tests to fail due to temporary files:

-- created: -------------------
  tmp/com.apple.dyld
          uname-8487F8338686B8B010BF7BAEFA1A50C27FCA318D2FED3E6B363D8D561583F290.closure  (36168 bytes)

[ei8fdb]

@pradyunsg One thing I just noticed: you've removed the feedback loop line so the user can tell us what's happened in a GH issue. Why was that removed? Gathering data will be more difficult.

[pradyunsg]

Happy to add it in -- I wasn't sure what exact link to use and the proposals all had placeholder links.

Lemme know and I'll add it in. :)

[ei8fdb]

Happy to add it in -- I wasn't sure what exact link to use and the proposals all had placeholder links.

Lemme know and I'll add it in. :)

OK, Ill create one and let you know.

[ei8fdb]

OK, Ill create one and let you know.

[ei8fdb]

Happy to add it in -- I wasn't sure what exact link to use and the proposals all had placeholder links.
Lemme know and I'll add it in. :)

OK, Ill create one and let you know.

I've created a GH issue template PR for users to report backtracking. What are you're thoughts on using Github's URL shortener to shorten the issue template URL?

[pfmoore]

So just to be clear, what's our workflow here?

• User tries an install, gets a message saying "pip is backtracking, please let us know"
• User files an issue report
• We do what exactly with that issue report? My understanding is that there's no presumption that this is a bug, pip backtracking is normal and expected behaviour. We want the information for data collection and analysis purposes. So what do we do? Close the issue with a "thank you for helping us to understand how pip is behaving in the real world" comment? Or leave it open?
• Who will collate these issues and turn the information into something actionable?
• Once we have "enough" information (I have no feel for what would be enough, but I'm assuming we'll know it when we get there) what's the process for informing users "thanks, we have enough now, please don't raise any more issues"? Presumably we update pip's message and wait for the next release for it to be available to users?

My concern here is twofold - one, that we get the tracker filled with informational tickets (my "pip issues" mail folder is already hard to manage, this might push it to a point where I switch off tracking the repo as a whole and rely on explicit pings to get notifications), and two, that we set users' expectations wrongly (by directing them to the bug tracker, we give the impression that something's wrong and pip shouldn't be doing this).

[ei8fdb]

We do what exactly with that issue report? My understanding is that there's no presumption that this is a bug, pip backtracking is normal and expected behaviour. We want the information for data collection and analysis purposes. So what do we do?

The intended purpose of this data gathering was to inform the team so you can make decisions based on data. e.g. the initial trigger points we decided on were plucked out of the air. In order to make any decisions to improve any usability, we need some data.

Close the issue with a "thank you for helping us to understand how pip is behaving in the real world" comment? Or leave it open?

Close it. My preference would have be to create a short form asking users to complete it in order for you to have this data. Since there's no infrastructure for creating forms/surveys I chose to create a GH issue template.

Who will collate these issues and turn the information into something actionable?

While the UX team is working on pip it will be us, after we are no longer working fulltime on pip, it will be UX contributors, or the pip maintainers team.

Once we have "enough" information (I have no feel for what would be enough, but I'm assuming we'll know it when we get there) what's the process for informing users "thanks, we have enough now, please don't raise any more issues"? Presumably we update pip's message and wait for the next release for it to be available to users?

Correct. This is part of designing and building software in a user centred way. When the maintainers have enough information about backtracking behaviour (I do not know that point. I am not a pip maintainer) the link to the GH issue should be removed. I don't see why that would be an onerous task.

two, that we set users' expectations wrongly (by directing them to the bug tracker, we give the impression that something's wrong and pip shouldn't be doing this).

This is why careful design, and content that provides clear expectation setting is important.

What's your suggested alternative to the data gathering suggestion? (There are other, better ways, but I am working with what I have available) If you'd prefer to not gather this data then let me know.

[pfmoore]

Thanks @ei8fdb for the detailed explanation, that answers my questions.

Personally, I won't be looking at the data from this exercise, because I have no real feel for what we could usefully do with it. Backtracking is just something pip has to do if we want the new correct resolution behaviour, and knowing how often it happens to users won't materially affect pip, in any way that I can see. (It might be useful information for "someone else", to guide efforts to push project developers to define their dependencies better/differently, for example, but I don't know who that "someone else" would be).

So I guess that for me, I'm in favour of collecting information like this, but I see it as more useful to the broader "packaging ecosystem" than of specific usefulness to pip. Others may have ideas how pip could use it, though, so that's just a personal opinion.

And no, I don't have a better suggestion for an approach. Like you say, it would be better to gather the data somewhere else, but the tracker's essentially the only option we have 🙁

I'm happy to deal with this by closing issues with a "thanks for the feedback" message, and leave the follow-up to others.

[ewdurbin]

A separate tracker for these issues may be significantly better option, and they’re free 🤗

pypa/pip-backtracking or something

[ewdurbin]

Also, FWIW if a form is what is desired... there’s very little stopping us from spinning up an app if someone builds it. PSF infra can host such a simple thing and we have the pypa.io domain to serve such a thing from

[dstufft]

We could also use a google form or something.

[pradyunsg]

PR updated based on all the existing feedback. CI is basically passing (ignoring failures due to #9030). The only missing bit is that there's no URL pointing the user to give us info about their failure. AFAICT, there's #9029 adding an issue template, but there's some concerns around doing so; and no concensus on what the "give us info" mechanism looks like.

Here's my suggestion: Let's put together a feedback-survey-like thing for this, with free form text fields, so that we can have users share the instances with us? I think that'd be nicer than an issue tracker, since "hey, we'd like some info to better understand stuff" is not what filing an issue on a tracker usually suggests (where functional assumption is that users would expect we'd "solve and close the issue" for them).

---

I'm basically thinking something like:

"hi, we're collecting data to better understand situations where pip's issue tracker would backtrack 'in the wild' so that we can prioritize future improvements in this area":

• requirements.txt file, if applicable
• command line invocation
• relevant part of the output
• free form text field for describing what behavior they see
• email to contact you (optional)

[ei8fdb]

The only missing bit is that there's no URL pointing the user to give us info about their failure. AFAICT, there's #9029 adding an issue template, but there's some concerns around doing so; and no concensus on what the "give us info" mechanism looks like.

@ewdurbin @dstufft I had forgotten that PSF uses Google services, I've put together this Google form (https://forms.gle/LkZP95S4CfqBAU1N6) to gather the information.

So final line of message 2 reads:

To improve how pip performs, tell us why this happened here: https://forms.gle/LkZP95S4CfqBAU1N6

---

Access to results: I've shared edit and read access with the other pip team members.

[pradyunsg]

Awesome! Thanks @ei8fdb for creating this so quickly! The form looks good to me. :)

To improve how pip performs, tell us why this happened here: forms.gle/LkZP95S4CfqBAU1N6

I've tweaked this slightly to:

To improve how pip performs, tell us that this happened here: https://pip.pypa.io/surveys/backtracking

Please holler if you think these tweaks aren't correct / appropriate. :)

[ei8fdb]

I've tweaked this slightly to:

To improve how pip performs, tell us that this happened here: https://pip.pypa.io/surveys/backtracking

Please holler if you think these tweaks aren't correct / appropriate. :)

2 things:

1. using the https://pip.pypa.io/surveys/backtracking URL is good! 👍
2. "tell us that this happened here" was confusing to read for me. My preference would be to use the previous wording

"To improve how pip performs, tell us why this happened here:"

but I'm OK with the wording you've used if you prefer it.

[pfmoore]

Or "tell us what happened here"?

[pradyunsg]

(CI failed coz I merged this too soon, oops)