Treat clobber warnings as errors for conda builds #54
Assignees
Comments
|
Adding an example for reference on how to investigate the contents of these files. # choose a conda package from https://anaconda.org/rapidsai/librmm/files
RMM_CONDA_PACKAGE=https://anaconda.org/rapidsai/librmm/24.04.00/download/linux-64/librmm-24.04.00-cuda12_240410_g8f19c9c3_0.tar.bz2
# download it
wget \
"${RMM_CONDA_PACKAGE}" \
-O librmm.tar.bz2
# list its contents, filtering down for paths mentioned in the clobber errors
tar jtf ./librmm.tar.bz2 \
| grep -E 'fmt|spdlog'For a recent version of fmt and spdlog headers and static libraries (click me) |
|
Open PRs:
|
This was referenced Apr 23, 2024
Short SummaryIt looks to me like the root cause of the
And I think there are 2 classes of fixes we could pursue.
DetailsI'm able to reproduce the errors about clobbering docker run \
--rm \
-v $(pwd):/opt/rmm \
-w /opt/rmm \
-it rapidsai/ci-conda:cuda12.2.2-ubuntu22.04-py3.9-amd64 \
bash
# do a 'librmm' conda build
ci/build_cpp.shI removed the 
(link to diff on my rapids-cmake fork) We could immediately resolve these
But separately, we should try to permanently fix this so the clobbering issues don't re-appear in similar situations in the future. If we want to preserve the ability to substitute in sources with not-yet-upstreamed patches in our conda builds, I can think of 2 options:
Downloading to a project-specific path would be fine for headers and source files, but I'm not sure how to handle other types of files that we're getting clobbering warnings for like:
Note I'm saying "project-specific" here because every RAPIDS package using For example, look at the conda builds on rapidsai/cuml#5821. It's getting 3 packages all trying to install the same So if NotesThere are other clobber warnings, just starting with |
|
Linking these related things (thanks @vyasr for the pointer). There is already a |
We don't publish |
|
Ah ok. I missed the "as CI artifacts" part of the description in rapidsai/rapids-cmake#414. Thank you, edited my original post. |
|
I just put up a PR to build the latest version of That'll be helpful whenever we decide to upgrade and drop the patch in |
|
I've looked across all the open PRs that add this setting: conda config --set path_conflict preventAnd see the following common issues. 1:
|
This was referenced Apr 24, 2024
rapids-bot bot
pushed a commit
to rapidsai/cuxfilter
that referenced
this issue
Apr 26, 2024
Contributes to rapidsai/build-planning#54 Ensures that conda builds will fail in the presence of multiple packages with identical paths Authors: - Jake Awe (https://github.com/AyodeAwe) - James Lamb (https://github.com/jameslamb) Approvers: - Ajay Thorve (https://github.com/AjayThorve) - Ray Douglass (https://github.com/raydouglass) URL: #583
This was referenced Apr 26, 2024
rapids-bot bot
pushed a commit
to rapidsai/raft
that referenced
this issue
May 2, 2024
Contributes to rapidsai/build-planning#54. The `libraft-headers` and `libraft-headers-only` conda packages are bundling `rmm`'s headers. I believe that's because the `librmm` conda package isn't available in the `libraft*` conda build environment, and as a result it's getting `rmm` via CPM (thanks to `rapids-cmake`). As a result, this project and any that depend on it are seeing warnings like the following in conda builds where `conda`'s `path_conflict` setting is set to `warn` or `prevent` (like #2245): ```text This transaction has incompatible packages due to a shared path. packages: rapidsai-nightly/linux-64::librmm-24.04.00a38-cuda11_240326_ga98931b9_38, rapidsai-nightly/linux-64::libraft-headers-only-24.04.00a93-cuda11_240326_g9637b3c2_93 path: 'include/rmm/mr/device/arena_memory_resource.hpp' ``` To fix that, this proposes the following changes: * make `librmm` a `host` dependency of the following conda packages: `libraft-headers-only`, `libraft-headers` ### Benefits of this change * slightly smaller `libraft-headers` and `libraft-headers-only` conda packages * reduced risk of runtime and installation issues caused by file clobbering ## Notes for reviewers ### History of changes to the `librmm` dependency for `libraft-headers`: * both `run` and `host`: #508 * both `run` and `host`, but ignoring its `run_exports`: #1264 * just `run`, but ignoring its `run_exports`: #2102 In particular, #2102 referred to the `host` dependency on `librmm` as "extraneous" but from a packaging perspective, I don't think it is. `librmm` being in `host` means it'll be present in the build environment, which means its headers will be *found* instead of *downloaded*, and therefore not packaging into the `libraft*` conda packages. ### How I tested this Built all the `raft` conda packages locally from `branch-24.06` and confirmed that they contain `rmm` headers. Then again from this branch and confirmed they were gone. ```shell docker run \ --rm \ --env-file "${PWD}/aws-creds.env" \ -v $(pwd):/opt/work \ -w /opt/work \ -it rapidsai/ci-conda:cuda12.2.2-ubuntu22.04-py3.10-amd64 \ bash CI="true" \ ci/build_cpp.sh # On 'branch-24.06', this showed the rmm headers being packaged. # On this branch, they're omitted. tar --bzip2 -tf \ /tmp/conda-bld-output/linux-64/libraft-headers-only-24.06.00a50-cuda12_240430_g1e0e2283_50.tar.bz2 \ | grep 'include/rmm' \ | wc -l ``` Also checked the CI logs from `conda-cpp-build` jobs here. On other recent PRs, I see CPM downloading `rmm` ... ```text -- CPM: Adding package [email protected] (branch-24.06) ``` ... and all the `rmm` headers getting installed as part of the `libraft-headers` package ```text -- Installing: /opt/conda/conda-bld/_h_env_placehold_placehold_..._placeho/include/rmm/cuda_stream.hpp ``` ([build link](https://github.com/rapidsai/raft/actions/runs/8904352932)) Here, I see `librmm` coming through via the conda package requirements ... ```text The following NEW packages will be INSTALLED: ... librmm: 24.06.00a17-cuda12_240430_g26fa9ecb_17 rapidsai-nightly ``` ... and being used instead of downloads via CPM ... ```text -- CPM: Using local package [email protected] ``` ... and no installation of the `rmm` headers as part of building any `libraft` packages. ([build link](https://github.com/rapidsai/raft/actions/runs/8910675575/job/24470450187?pr=2284)) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Ray Douglass (https://github.com/raydouglass) URL: #2284
|
I'm pausing on this for now to help out with some of the other initiatives (like #31 and #33). I believe rapidsai/raft#2284 fixed the issues with |
|
Thanks for the update James! I agree that those two are more immediate priorities, but the progress you've already made on here is great. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(Apr 23, 2024) moved from an internal tracking board. The description here is @ajschmidt8 's original write-up of the issue.
Problem
Some of our conda builds suffer from
ClobberWarnings (e.g. here and here).Ignoring these clobber warnings can sometimes result in packaging issues. Therefore, we'd like to begin treating these clobber warnings as errors.
Ideally, this setting should go in the
.condarcfile of our CI images (i.e. here) so that it applies to every RAPIDS repository.However, since many repositories already suffer from clobber warnings, this would break CI for a lot of people.
Therefore, we need to plan the rollout carefully.
Solution
To roll out this new configuration setting safely, we should do the following:
For each of the repositories in the section below:
a. Open a PR and add the following line to any build scripts (e.g.
ci/build_{cpp,python}.sh):conda config --set path_conflict preventb. Resolve any build issues that result from this change
c. If build issues occur, fix them and merge the PR (leaving in the new conda config line, that will be cleaned up later)
d. If no build issues occur, simply close the PR
e. Link the PR in this issue (do not link this issue in the PR, since this issue is in a private repository)
Once all of the repositories are successfully using the new conda config setting, add the setting to https://github.com/rapidsai/ci-imgs/blob/main/context/condarc.tmpl
Go back and clean up all of the extraneous
conda config --set path_conflict preventlines in each repositoryRepositories
Look at the repos at this search query.
The text was updated successfully, but these errors were encountered: