Build Python packages using the limited API

[vyasr]

Python has a limited API that is guaranteed to be stable across minor releases. Any code using the Python C API that limits itself to using code in the limited API is guaranteed to also compile on future minor versions of Python within the same major family. More importantly, all symbols in the current (and some historical) version of the limited API are part of Python's stable ABI, which also does not change between Python minor versions and allows extensions compiled against one Python version to continue working on future versions of Python.

Currently RAPIDS builds a single wheel per Python version. If we were to compile using the Python stable ABI, we would be able to instead build a single wheel that works for all Python versions that we support. There would be a number of benefits here:

• Reduced build time: This benefit is largely reduced by Support dynamic linking between RAPIDS wheels #33, since if we build the C++ components as standalone wheels they are already Python-independent (except when we actually use the Python C API in our own C libraries; the only example that I'm currently aware of in RAPIDS is ucxx). The Python components alone are generally small and easy to build. We'll still benefit, but the benefits will be much smaller.
• Reduced testing time: Currently we run test across a number of Python versions for our packages on every PR. We often struggle with what versions need to be tested each time. If we were to only build a single wheel that runs on all Python versions, it would be much easier to justify a consistent strategy of always testing e.g. the earliest and latest Python versions. We may still want to test more broadly in nightlies, but really the only failure mode here is if a patch release is made for a Python version that is neither the earliest nor the latest, and that patch release contains breaking changes. That is certainly possible (e.g. the recent dask failure that forced us to make a last-minute patch), but it's infrequent enough that we don't need to be testing regularly.
• Wider support matrix: Since we'll have a single binary that works for all Python versions, maintaining the full support matrix will be a lot easier and we won't feel as much pressure to drop earlier versions in order to support newer ones.
• Day 0 support: Our wheels will work for new Python versions as soon as they're released. Of course, if there are breaking changes then we'll have to address those, but in the average case where things do work users won't be stuck waiting on us.
• Better installation experience: Having a wheel that automatically works across Python versions will reduce the frequency of issues that are raised around our pip installs.

Here are the tasks (some ours, some external) that need to be accomplished to make this possible:

• Making Cython compatible with the limited API: Cython has preliminary support for the limited API. However, this support is still experimental, and most code still won't compile. I have been making improvements to Cython itself to fix this, and I now have a local development branch of Cython where I can compile most of RAPIDS (with additional changes to RAPIDS libraries). We won't be able to move forward with releasing production abi3 wheels until this support in Cython is released. This is going to be the biggest bottleneck for us.
• nanobind support for the limited API: nanobind can already produce abi3 wheels when compiled with Python 3.12 or later. Right now we use nanobind in pylibcugraphops, and nowhere else.
• Removing C API usage in our code: RAPIDS makes very minimal direct usage of the Python C API. The predominant use case that I see is creating memoryviews in order to access some buffers directly. We can fix this by constructing buffers directly. The other thing we'll want to do is remove usage of the NumPy C API, which has no promise of supporting the limited API AFAIK. That will be addressed in Remove usage of the NumPy C API #41. Other use cases can be addressed incrementally.
• Intermediate vs. long-term: If Cython support for the limited API ends up being released before RAPIDS drops support for Python 3.10, we may be in an intermediate state where we still need to build a version-specific wheel for 3.10 while building an abi3 wheel for 3.11+ (and 3.12+ for pylibcugraphops due to nanobind). If that is the case, it shouldn't cause much difficulty since it'll just involve adding a tiny bit of logic on top of our existing GH workflows.

At this stage, it is not yet clear whether the tradeoffs required will be worthwhile, or at what point the ecosystem's support for the limited API will be reliable enough for us to use in production. However, it shouldn't be too much work to get us to the point of at least being able to experiment with limited API builds, so we can start answering questions around performance and complexity fairly soon. I expect that we can pretty easily remove explicit reliance on any APIs that are not part of the stable ABI, at which point this really becomes a question of the level of support our binding tools provide and if/when we're comfortable with those.

[jakirkham]

It is worth noting that the Python Buffer Protocol C API landed in Python 3.11 (additional ref). So think that is a minimum for us

Also find this listing of functions in the Limited and Stable API quite helpful

[vyasr]

Yes. I have been able to build most of RAPIDS using Cython's limited API supported (along with some additional changes I have locally) in Python 3.11. Python 3.11 is definitely a must. But as I said above in the "intermediate vs long-term" bullet, we could still benefit before dropping Python<3.11 support by building one wheel for each older Python version and then build an abi3 wheel to be used for Python 3.11+.

[vyasr]

I've made PRs ro rmm, raft, and cuml that address the issues in those repos. I've also taken steps to remove ucxx's usage of the the numpy C API (#41), which in turn removes one of its primary incompatibilities. The last major issue in RAPIDS code that I see is the usage of the array module in the Array class that is vendored by both kvikio and ucxx (and ucx-py). If that can be removed, then I think we'll be in good shape on the RAPIDS end, and we'll just be waiting on support for this feature in Cython itself. @jakirkham expressed interest in helping out with that in the process of making that Array class more broadly usable.

[da-woods]

A small warning here:

There's definitely places where Cython is substituting private C API for private Python API, so future compatibility definitely isn't guaranteed (it'll just be a runtime failure rather than a compile-time failure). We'll see how that evolves - I hope to be able to make some of these warnings rather than failures (since it's largely just non-essential introspection support).

We're also having to build a few more runtime version-checks into our code. Which is obviously a little risky because although you're compiling the same thing, you're taking different paths on different Python versions.

So the upshot is that your testing matrix probably doesn't reduce to a single version. (From Cython's point of view the testing matrix probably expands, because we really should be testing combinations like Py_LIMITED_API=0x03090000 with Python 3.12 and that gets big quite quickly so I don't know how we're going to do that)

[vyasr]

Thanks for chiming in here @da-woods! I appreciate your comments. I agree that there is more complexity around testing here than simply a set and forget single version. At present, RAPIDS typically supports 2 or 3 Python versions at a time. We tend to lag a bit behind NEP 29/SPEC 0 timelines, so we support older versions a bit longer at the expense of not supporting new ones until they've been out for a bit. A significant part of the resource constraint equation for us is certainly on the testing side since running our full test suites on multiple Python versions adds up quickly. The way that I had envisioned this working, if we did move forward, would be that we built on the oldest supported Python (e.g. Py_LIMITED_API=0x03090000) and then we ran tests on the earliest and latest Python we supported (e.g. 3.9 and 3.11). The big benefit of using the limited API in this space would be that we could bump up the latest supported Python version without needing to move the earliest. The assumption would be that by the time a new Python version was released (e.g. 3.12), we would have gone through enough patch release of the previous release (3.11) to trust that nothing would be breaking in future patch releases. Of course, in practice that's probably not true: CPython certainly doesn't always strictly follow SemVer rules for patch releases, and to be fair Hyrum's law certainly applies to a project at that scale. Beyond that Cython's use of CPython internals certainly means that we could be broken even by patch releases. In practice what this would probably mean is that we would run tests as mentioned above on a frequent basis (on every PR), then run a larger test matrix infrequently (say, nightly or weekly). IOW even with limited API builds we would definitely still want to do broader testing to ensure that such builds are actually as compatible as they claim to be. However, I'd hope that the scale of that testing would be reduced.