--find-link urls are not cached

[sbidoul]

Environment

• pip version: 20.3b1
• Python version: 3
• OS: Linux

Description

Pip disables HTTP caching for index and find-links URLs, due to Cache-Control: max-age=0 being passed to http requests.

pip/src/pip/_internal/index/collector.py

Lines 142 to 155 in 5ee7caf

	# We don't want to blindly returned cached data for
	# /simple/, because authors generally expecting that
	# twine upload && pip install will function, but if
	# they've done a pip install in the last ~10 minutes
	# it won't. Thus by setting this to zero we will not
	# blindly use any cached data, however the benefit of
	# using max-age=0 instead of no-cache, is that we will
	# still support conditional requests, so we will still
	# minimize traffic sent in cases where the page hasn't
	# changed at all, we will just always incur the round
	# trip for the conditional GET now instead of only
	# once per 10 minutes.
	# For more information, please see pypa/pip#5670.
	"Cache-Control": "max-age=0",

While this is not an issue for accessing PEP 503 indexes where each project has different URL, for find-links URLs this generates a lot of useless HTTP fetches as pip repeatedly downloads identical find-links URLs for each requirement.

Expected behavior

Caching of --find-links URLs within a single pip session.

How to Reproduce

Obtain a somewhat long list of requirements and a https find-links URL (not necessarily related to the requirements).

Run pip wheel --no-deps and notice the find-links web server gets an identical hit for each requirement in the list.

Possible solution

A @lru_cache on _get_html_page improves pip wheel execution time by 50% for a list of 50 requirements.