Remove NumPy <2 pin #1441
Remove NumPy <2 pin #1441
Conversation
seberg
added
non-breaking
github-actions
bot
added
conda
|
I merged in the upstream to see if CI is failing in the same way we see here: rapidsai/docker#709 (comment) |
|
Closing and reopening to restart CI now that cuDF is done: rapidsai/cudf#16300 |
|
Seeing some unexpected and unrelated failures that appear to be due to a cuDF change Running CI in a no change PR to see if this reproduces there: #1444 Edit: Seeing the same error over there ( #1444 (comment) ). Following up separately |
|
On the Conda side, it looks like we are still seeing NumPy 1 in CI. Think we are missing another dependency. Will need to take a closer look |
|
Took all of the dependencies on CI (except RAPIDS ones) and tried to create a Conda environment with them locally on Linux ARM (though there should be no difference between x86_64 in this regard). It created the environment using NumPy 2 Didn't see anything obviously missing in this repo The other RAPIDS dependencies are all migrated Not sure what we are missing. May need to dig deeper later |
Strongly suspect this is a result of the changes from rapidsai/cudf#15483. I'll work on adapting |
|
Depends on PR: #1447 |
Contributes to rapidsai/build-planning#33. Proposes the following for `cuspatial` wheels: * add build and runtime dependencies on `libcudf` wheels * stop vendoring copies of `libcudf.so`, `libnvcomp.so`, `libnvcomp_bitcomp.so`, and `libnvcomp_gdeflate.so` - *(load `libcudf.so` dynamically at runtime instead)* And other related changes for development/CI: * combine all `pip install` calls into 1 in wheel-testing scripts - *like rapidsai/cudf#16575 - *to improve the chance that packaging issues are discovered in CI* * `dependencies.yaml` changes: - more use of YAML anchors = less duplication - use dedicated `depends_on_librmm` and `depends_on_libcudf` groups * explicitly pass a package type to `gha-tools` wheel uploading/downloading scripts ## Notes for Reviewers ### Benefits of these changes Unblocks CI in this repo (ref: #1444 (comment), #1441 (comment)). Reduces wheel sizes for `cuspatial` wheels by about 125MB 😁 | wheel | size (before) | size (this PR) | |:-----------:|-------------:|---------------:| | `cuspatial` | 146.0M | 21M | | `cuproj ` | 0.9M | 0.9M | |**TOTAL** | **146.9M** | **21.9M** | *NOTES: size = compressed, "before" = 2024-08-21 nightlies (c60bd4d), CUDA = 12, Python = 3.11* <details><summary>how I calculated those (click me)</summary> ```shell # note: 2024-08-21 because that was the most recent date with # successfully-built cuspatial nightlies # docker run \ --rm \ -v $(pwd):/opt/work:ro \ -w /opt/work \ --network host \ --env RAPIDS_NIGHTLY_DATE=2024-08-21 \ --env RAPIDS_NIGHTLY_SHA=c60bd4d \ --env RAPIDS_PR_NUMBER=1447 \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env RAPIDS_REPOSITORY=rapidsai/cuspatial \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \ bash mkdir -p "${WHEEL_DIR_BEFORE}" mkdir -p "${WHEEL_DIR_AFTER}" py_projects=( cuspatial cuproj ) for project in "${py_projects[@]}"; do # before RAPIDS_BUILD_TYPE=nightly \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="branch-24.10" \ RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}" # after RAPIDS_BUILD_TYPE=pull-request \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}" done du -sh ${WHEEL_DIR_BEFORE}/* du -sh ${WHEEL_DIR_BEFORE} du -sh ${WHEEL_DIR_AFTER}/* du -sh ${WHEEL_DIR_AFTER} ``` </details> Reduces the amount of additional work required to start shipping `libcuspatial` wheels. ### Background This is part of ongoing work towards packaging `libcuspatial` as a wheel. relevant prior work: * packaging `libcudf` wheels: rapidsai/cudf#15483 * consolidating `pip install` calls in CI scripts for `cudf`: rapidsai/cudf#16575 * `cudf` dropping its Arrow library dependency: rapidsai/cudf#16640 ### How I tested this Confirmed in local builds and CI logs that `cudf` is being *found*, not *built*, in `cuspatial` builds. ```text -- CPM: Using local package [email protected] ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10602971716/job/29386288614?pr=1447#step:9:23472)) Built `cuspatial` wheels locally and ran all the unit tests, without issue. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Roeschke (https://github.com/mroeschke) URL: #1447
|
#1447 is merged and I've pulled in those changes here |
|
Ok think I understand what is happening with NumPy 1 being installed on CI now, OSMnx 1.9.4 (currently latest) pins to NumPy 1 IIUC OSMnx 2.0 will relax this constraint I don't think there is a lot we can do about that Looking at the code here, it appears the cuSpatial library doesn't actually use OSMnx. Only the notebooks do Perhaps there is a way to move that dependency out of the main CI paths so we can test with NumPy 2 here (ignoring notebooks which will have to stay on NumPy 1) Think if we can do that, we can move forward Edit: Nvm this is already only installed in the notebook job. So this is not an issue |
|
Should add another piece is This can only happen if While |
There was a problem hiding this comment.
I just checked, and looks like the conda-python-tests jobs are still pulling in numpy==1.26.4 at runtime:
wheel-tests-cuspatial are pulling in numpy==2.0.2
wheel-tests-cuproj also pulling in numpy==2.0.2
This PR is up to date with latest branch-24.10 (so includes the proj pin removal from #1449 and other packaging changes). I think we need to figure out what else is holding numpy under 2.0. @jakirkham @seberg I'm going to push a commit here changing these pins to numpy>=2.1.0,<3.0a0 just to hopefully get more direct information about this via solver errors.
|
In the wheel builds, looks to me like Seems like it isn't even considering Just pushed c1f5845 bumping to |
|
Is the reason that something is pinning llvmlite to: |
seberg
force-pushed
the
my_new_branch
branch
from
a9ced90
to
eaee711
Compare
seberg
force-pushed
the
my_new_branch
branch
from
eaee711
to
a58d1e6
Compare
|
Tried removing all new things and rebasing once more... I am confuse because it seems like there is something that just doesn't resolve in conda, but I am not sure what. |
|
@seberg This is blocked because we're pinning to an old Line 183 in 2a30f5f This will be unblocked once we migrate to a newer fmt/spdlog. See rapidsai/build-planning#56. cc: @jameslamb |
This PR removes the NumPy<2 pin which is expected to work for
RAPIDS projects once CuPy 13.3.0 is released (CuPy 13.2.0 had
some issues preventing the use with NumPy 2).