Distribute libcuspatial wheels #1450

jameslamb · 2024-08-30T19:24:36Z

Description

Contributes to rapidsai/build-planning#33

Adds libcuspatial wheels, and switches cuspatial wheels to using them.

Notes for Reviewers

Benefits of these changes

Faster CI runs and smaller total footprint on package repositories (because now libcuspatial no longer needs to be compiled once per Python version).

Smaller cuspatial wheels.

whee. l	size (before)	size (this PR)
`libcuspatial`	---	17.0M
`cuspatial`.	21.0M	4.1M
`cuproj`	0.9M	0.9M
TOTAL	21.9M	22.0M

NOTES: size = compressed, "before" = 2024-09-02 nightlies (1544e7b), CUDA = 12, Python = 3.11

how I calculated those (click me)

docker run \
    --rm \
    -v $(pwd):/opt/work:ro \
    -w /opt/work \
    --network host \
    --env RAPIDS_NIGHTLY_DATE=2024-09-02 \
    --env RAPIDS_NIGHTLY_SHA=1544e7b \
    --env RAPIDS_PR_NUMBER=1450 \
    --env RAPIDS_PY_CUDA_SUFFIX=cu12 \
    --env RAPIDS_REPOSITORY=rapidsai/cuspatial \
    --env WHEEL_DIR_BEFORE=/tmp/wheels-before \
    --env WHEEL_DIR_AFTER=/tmp/wheels-after \
    -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \
    bash

mkdir -p "${WHEEL_DIR_BEFORE}"
mkdir -p "${WHEEL_DIR_AFTER}"

py_projects=(
    cuspatial
    cuproj
)

for project in "${py_projects[@]}"; do
    # before
    RAPIDS_BUILD_TYPE=nightly \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="branch-24.10" \
    RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \
        rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}"

    # after
    RAPIDS_BUILD_TYPE=pull-request \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
        rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}"
done

# after
RAPIDS_BUILD_TYPE=pull-request \
RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
    rapids-download-wheels-from-s3 cpp "${WHEEL_DIR_AFTER}"

du -sh ${WHEEL_DIR_BEFORE}/*
du -sh ${WHEEL_DIR_BEFORE}
du -sh ${WHEEL_DIR_AFTER}/*
du -sh ${WHEEL_DIR_AFTER}

devcontainers job?

Once this PR is close to ready, let's merge the devcontainers PR and then re-run the devcontainers CI here.

devcontainers PR: rapidsai/devcontainers#387

`rapids-metadata` changes?

Not necessary, libcuspatial is already there: https://github.com/rapidsai/rapids-metadata/blob/9b6307e708511cd9a1990d8bb36606df53bc9e1b/src/rapids_metadata/__init__.py#L89

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

cpp/cmake/thirdparty/get_ranger.cmake

ci/build_wheel.sh

bdice

Re-approving.

harrism · 2024-09-03T23:53:29Z

My review is requested but I have no expertise on wheels/pip.

harrism

Approving cmake changes.

ci/build_wheel.sh

cpp/cmake/thirdparty/get_ranger.cmake

python/libcuspatial/CMakeLists.txt

vyasr · 2024-09-04T00:02:55Z

python/libcuspatial/CMakeLists.txt

+set(CUSPATIAL_USE_CUDF_STATIC OFF)
+set(CUSPATIAL_EXCLUDE_CUDF_FROM_ALL ON)
+
+set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/lib)


Could you remind me why we're setting this? Is it strictly necessary? Why aren't the install rules that come from add_subdirectory sufficient? If we do have to set this variable, why do we have to set it but also do a separate install below? It seems like either one or the other should be sufficient. I know this code is in the libcudf wheel now as well, but I don't understand why.

I copied this in here from libcudf because I think I misunderstood the purpose there.

There, this is used to ensure all the nvcomp libraries and libcudf.so all get put together into the same place: https://github.com/rapidsai/cudf/blob/26091a44b3dbf0f56fc0dfc5f081077f2d00681f/python/libcudf/CMakeLists.txt#L50-L53

That isn't necessary here, and you're right that the install rules from the add_subdirectory() are sufficient. Removed this stuff in fb307ba

I suspect that the libcudf install rules are also a little overspecified now. Simultaneously setting both the CMake var and doing the install of cudf in particular seems redundant. The nvcomp installation might be fine as is. We can revisit those later, though.

vyasr · 2024-09-04T00:04:26Z

Mostly confused by the CMake installation, otherwise everything else looks OK to me.

jameslamb · 2024-09-04T23:34:13Z

I've merged rapidsai/devcontainers#387 and restarted the pip devcontainer CI job here. Will merge this if it passes.

build link: https://github.com/rapidsai/cuspatial/actions/runs/10705830747/job/29698846144?pr=1450

jameslamb · 2024-09-04T23:40:05Z

/merge

Fixes #1455 devcontainer conda CI jobs are failing in this project because of the following mix of characteristics for thoes jobs: * all build and runtime dependencies for `libcuspatial`, `cuspatial`, `cuproj` are installed via conda * `libcuspatial`, `cuspatial`, and `cuproj` wheels are then built with `pip install -e --no-deps --no-build-isolation` * `import libcuspatial` results in unconditionally running `import libcudf` * `libcudf` is provided by the `libcudf` **conda** package, which does not have any Python modules, so that import fails This fixes that, and restores the ability to mix a `pip install`'d `cuspatial` / `cuproj` with a `conda`-installed `libcudf`. ## Notes for Reviewers ### How did CI not catch this before? When rapidsai/devcontainers#387 was merged, I only re-ran the **pip** devcontainers CI job on #1450. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1456

Follow-up to #1448. This proposes some `dependencies.yaml` changes that I noticed while working on adding `libcuspatial` wheels in #1450. I've left inline comments with more details, but in short: * not including any cuspatial projects in the conda environment files checked into this repo (which are intended to be used to set up a development environment) * removing an unnecessary conda-only list of dependencies from pyproject.toml-specific config for `cuproj` * introduction of `depends_on_libcuspatial` and `depends_on_cuproj` lists in `dependencies.yaml`, to reduce duplication Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1451

jameslamb added 7 commits August 29, 2024 10:42

initial library setup

3727dcb

dependencies.yaml updates

c37508f

update build scripts, cuspatial cmake

59f14f1

add VERSION

ad7cd4e

do not mark ranger for export

d6eba42

add library-loading, fix auditwheel exclusions

8a39de5

revert unrelated changes, remove now-unnecessary logic in build.sh

8a09b3a

jameslamb added 2 - In Progress Currenty a work in progress 5 - DO NOT MERGE Hold off on merging; see PR for details improvement Improvement / enhancement to an existing function non-breaking Non-breaking change libcuspatial Relates to the cuSpatial C++ library labels Aug 30, 2024

github-actions bot added cmake Related to CMake code or build configuration Python Related to Python code ci labels Aug 30, 2024

jameslamb removed the 2 - In Progress Currenty a work in progress label Aug 30, 2024

jameslamb added 3 commits August 30, 2024 14:26

Merge branch 'branch-24.10' into libcuspatial-wheel

af2096c

add job to pr-builder

2021ac6

fix build script, revert unrelated change

2d78d79

jameslamb mentioned this pull request Aug 30, 2024

rearrange dependencies.yaml, fix development instructions #1451

Merged

3 tasks

jameslamb added 3 commits August 30, 2024 13:50

remove commented-out debugging code

815c82a

reduce number of builds

4539088

add comment on matrix filter

fe85ee2

jameslamb mentioned this pull request Aug 30, 2024

add manifest entry for libcuspatial wheels rapidsai/devcontainers#387

Merged

jameslamb commented Aug 30, 2024

View reviewed changes

cpp/cmake/thirdparty/get_ranger.cmake Show resolved Hide resolved

fix copyright

6856666

jameslamb changed the title ~~WIP: [NOT READY FOR REVIEW] Distribute libcuspatial wheels~~ Distribute libcuspatial wheels Sep 3, 2024

jameslamb added 3 - Ready for Review Ready for review by team and removed 5 - DO NOT MERGE Hold off on merging; see PR for details labels Sep 3, 2024

jameslamb marked this pull request as ready for review September 3, 2024 14:15

jameslamb requested review from a team as code owners September 3, 2024 14:15

jameslamb requested review from AyodeAwe, KyleFromNVIDIA, thomcom, harrism, robertmaynard and bdice September 3, 2024 14:15

bdice approved these changes Sep 3, 2024

View reviewed changes

ci/build_wheel.sh Show resolved Hide resolved

statically link cudart

1d00c91

bdice approved these changes Sep 3, 2024

View reviewed changes

harrism removed their request for review September 3, 2024 23:53

harrism approved these changes Sep 3, 2024

View reviewed changes

vyasr requested changes Sep 4, 2024

View reviewed changes

remove unnecessary extra install()

fb307ba

jameslamb requested a review from vyasr September 4, 2024 16:37

vyasr approved these changes Sep 4, 2024

View reviewed changes

rapids-bot bot merged commit 0e9d36c into rapidsai:branch-24.10 Sep 4, 2024
64 checks passed

jameslamb deleted the libcuspatial-wheel branch September 4, 2024 23:40

This was referenced Sep 5, 2024

Use RMM get_current_device_resource_ref() #1454

Open

[BUG]: conda devcontainers build failing: "No module named 'libcudf'" #1455

Closed

re-enable mixing libcuspatial wheels with libcudf conda packages #1456

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distribute libcuspatial wheels #1450

Distribute libcuspatial wheels #1450

jameslamb commented Aug 30, 2024 •

edited

Loading

bdice left a comment

harrism commented Sep 3, 2024

harrism left a comment

vyasr Sep 4, 2024

jameslamb Sep 4, 2024

vyasr Sep 4, 2024

vyasr commented Sep 4, 2024

jameslamb commented Sep 4, 2024

jameslamb commented Sep 4, 2024

Distribute libcuspatial wheels #1450

Distribute libcuspatial wheels #1450

Conversation

jameslamb commented Aug 30, 2024 • edited Loading

Description

Notes for Reviewers

Benefits of these changes

devcontainers job?

rapids-metadata changes?

Checklist

bdice left a comment

Choose a reason for hiding this comment

harrism commented Sep 3, 2024

harrism left a comment

Choose a reason for hiding this comment

vyasr Sep 4, 2024

Choose a reason for hiding this comment

jameslamb Sep 4, 2024

Choose a reason for hiding this comment

vyasr Sep 4, 2024

Choose a reason for hiding this comment

vyasr commented Sep 4, 2024

jameslamb commented Sep 4, 2024

jameslamb commented Sep 4, 2024

jameslamb commented Aug 30, 2024 •

edited

Loading

`rapids-metadata` changes?