-
-
Notifications
You must be signed in to change notification settings - Fork 625
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lockfiles: Should we use pip-compile or teach Pex to generate lockfiles? #12470
Comments
I think mixing the format of the lock files (PEP 665) in with this is not a great idea. First and foremost we need to generate locks that work. They need to handle interpreter constraints and platforms. Once we know how to do that, then the data needed to support those locks flows. The format to hold that data then flows from there. |
Fair. I rewrote this ticket, PTAL. I think PEP 665 is still relevant, such as wondering if/when pip-compile will add support. But I see what you mean and it's deprioritized. |
@jsirois any insights or opinions on how expensive lockfile generation would be for Pex to implement/maintain? |
Sounds like pip-compile will not plan to add PEP 665 support until the PEP is accepted, whereas we're interested in being an early adopter and helping to push the standardization movement forward. We also have not heard back on our offer to switch to the new resolver, despite activity on the thread. For both these reasons, I think we should lean into Pex/Pants generating PEP 665-compatible lockfiles. |
Instead of pip-compile vs Slack, I think the solution is: Poetry. As mentioned in Slack, it looks like Poetry robustly handles our concerns with environment markers! (#12200)
Poetry is battle-tested and ready out-of-the-box, whereas improving pip-compile or teaching Pex to generate lockfiles are both large, risky projects. -- Two known downsides:
|
…inor_versions()` (#12515) Because Poetry already robustly handles generating a single lockfile that works with all requested Python interpreters (#12470 (comment)), there is no need for the "pessimistic generation" proposed in #12463. [ci skip-rust] [ci skip-build-wheels]
…-compile (#12549) **Disclaimer**: This is not a formal commitment to Poetry, as we still need a more rigorous assessment it can handle everything we need. Instead, this is an incremental improvement in that Poetry handles things much better than pip-compile. It gets us closer to the final result we want, and makes it much more ergonomic to use the experimental feature—like `generate_all_lockfiles.sh` now not needing any manual editing. But we may decide to switch from Poetry to something like pdb or Pex. -- See #12470 for why we are not using pip-compile. One of the major motivations is that Poetry generates lockfiles compatible with all requested Python interpreter versions, along with Linux, macOS, and Windows. Meaning, you no longer need to generate the lockfile in each requested environment and manually merge like we used to. This solves #12200 and obviates the need for #12463. -- This PR adds only basic initial support. If we do decide to stick with Poetry, some of the remaining TODOs: - Handle PEP 440-style requirements. - Hook up to `[python-setup]` and `[python-repos]` options. - Hook up to caching. - Support `--platform` via post-processing `poetry.lock`: #12557 - Possibly remove un-used deps/hashes to reduce supply chain attack risk: #12458 -- Poetry is more rigorous than pip-compile in ensuring that all interpreter constraints are valid, which prompted needing to tweak a few of our tools' constraints.
Status quo & issues
We're using pip-compile. While it has a good UX, we've encountered some issues:
--platform
pip option does not do anything when passed via--pip-args
. We likely need this flag when you're using--platform
w/ Pants, e.g. theplatforms
field ofpex_binary
.We've offered to fix the resolver in jazzband/pip-tools#1190 (comment), but haven't heard back.
It's also unclear how they will respond to PEP 665. While it hasn't been accepted yet, the format has many advantages over pip-compile's
requirements.txt
style (including formalizing what environments the lock is valid for), and pantsbuild would also like to support standardization in the Python ecosystem.Proposal: Pex generating lockfiles
PEX is well-positioned to generate lockfiles. We already use pip's new resolver and have experience with the packaging ecosystem.
A unique benefit Pex could bring: robustly handle using multiple environments, #12200. Pex already has technology to resolve against multiple Python versions in parallel, e.g. 3.5-3.9. It can use this + possibly post-processing resolves to robustly generate a lockfile that works in multiple environments. This compatibility info will be set in
metadata.tags
and/ormetadata.marker
.Originally, some Pants team members were concerned with bifurcating the Python ecosystem even more, but PEP 665 instead allows Pex to help build momentum around standardization in the community. Rather than us introducing Yet Another format, we'll be using a standardized format.
The major downside seems to be development time and maintenance burden. It's plausible pip may eventually gain the ability to generate PEP 665 lockfiles and we could leverage that, but in the meantime, the functionality would likely be added to Pex.
The text was updated successfully, but these errors were encountered: