-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update fmt
and spdlog
across RAPIDS
#56
Comments
@bdice drew my attention to these global pins in fmt:
- '10'
spdlog:
- '1.12' I think the I think we should NOT update any RAPIDS packages to depending on the References:
|
Yes, we should try to keep RAPIDS in sync with conda-forge. For these, I would move them forward once the migrator passes some critical threshold (for spdlog, I might base it on when packages like For the purpose of fixing our conda clobbering issues, we may want to work on some kind of vendoring solution (centralized via Also, if we vendor these and remove the dependencies from the RAPIDS conda packages, it would also make it possible to decouple from the global conda-forge pinnings and manage version updates solely through rapids-cmake (the status quo requires changes in both rapids-cmake and conda recipes). |
Modifies pinning tests from #530: * to only test projects that were downloaded by CPM (e.g. ignoring the `fmt` that might already exist in the build environment) * to echo out `pinned_versions.json` and `versions.json` in logs from failed tests, to make debugging faster ## Notes for Reviewers #592 proposes some testing changes that aren't specific to the goals of that PR. Since that PR might be stuck for a bit (rapidsai/build-planning#56 (comment)), this proposes pulling those out into a separate PR: * so that other changes in this project benefit from them * to shrink the diff of #592 and therefore the risk of merge conflicts ### How I tested this Pushed a commit with the new test error message content changes but keeping `fmt` in the failing tests, to confirm that the expected tests failed. <details><summary>got the expected outputs (click me)</summary> ```text The following tests FAILED: 698 - cpm_generate_pins-nested-makefile (Failed) 700 - cpm_generate_pins-nested-ninja (Failed) 702 - cpm_generate_pins-nested-ninja_multi-config (Failed) 722 - cpm_generate_pins-simple-makefile (Failed) 724 - cpm_generate_pins-simple-ninja (Failed) 726 - cpm_generate_pins-simple-ninja_multi-config (Failed) ``` And they failed in the expected way more informative logs! ```text CMake Error at CMakeLists.txt:51 (message): pinned fmt tag (10.2.1) should differ compared to baseline 10.2.1 pinned_versions.json: { "always_download" : true, "git_shallow" : false, "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } versions.json: { "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } ``` ([build link](https://github.com/rapidsai/rapids-cmake/actions/runs/8837613213/job/24267079292?pr=592)) </details> # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #599
FYI -- There's active migrations for spdlog 1.14 and fmt 11 going on in conda-forge right now:
I believe RAPIDS is now multiple versions behind on spdlog which will make solving environments painful. |
Thanks for the heads-up @kkraus14! We'll discuss and try to land an update for fmt/spdlog in the next RAPIDS release. |
Description
RAPIDS projects that need
fmt
andspdlog
userapids-cmake
to find it, via functions likerapids_cpm_fmt()
(docs link).rapids-cmake
is currently carrying patches to both of those libraries, and as a result always downloads sources of them. (see rapidsai/rapids-cmake#525 for details).Those patches have since been upstreamed and made it into official releases of those projects (in their main source control and on
conda-forge
). This proposes upgrading to those new versions across RAPIDS and dropping the patches inrapids-cmake
.Benefits of this work
conda
builds across many RAPIDS projectsrmm
to ensure future such conflicts raise a loud errorrapids-cmake
and therefore the need to reason about them in future upgradesAcceptance Criteria
rapids-cmake
is pulling infmt >= 10.2.1
rapids-cmake
is pulling inspdlog >= 1.13
fmt
andspdlog
inrmm
's conda build CI logsApproach
On a branch, modify
rapids-cmake/cpm/versions.json
(code link) such that:fmt
andspdlog
are bumped to these new onesFollow "Overriding RAPIDS CMake" (rapids-cmake docs) to point builds of at least
rmm
,cudf
, andraft
at your branch ofrapids-cmake
with these changes. Also modify those projects'dependencies.yaml
and/or conda recipemeta.yaml
to change their version constraints onfmt
andspdlog
.NOTE: this is a great task for
rapids-reviser
(https://github.com/rapidsai/rapids-reviser/tree/main/examples/fmt-10-spdlog-1.12).Confirm that CI succeeds and that the correct versions of
fmt
andspdlog
are being pulled in.For
rmm
, re-use this PR for that purpose: rapidsai/rmm#1508. If this change also results in there being 0 remaining conda clobbering warnings inrmm
, leave in theconda config --set path_conflict prevent
in that PR's build scripts, so we'll be alerted by a CI failure if something around this changes in the future.Notes
What makes you confident that these patches can be removed?
From #54 (comment)
fmt
's most recent release was10.2.1
, so that patch we're carrying around for the10.1.1
version could be dropped.rapids-cmake
: (fmt/fix_10_1_1_version.diff)v10.2.1
release on conda-forge removing a similar patch: fmt v10.2.1 conda-forge/fmt-feedstock#48The
spdlog
patch we've been carrying around has been upstreamedrapids-cmake
: (spdlog/nvcc_constexpr_fix.diff)v1.13.0
of the conda-forge recipe is up: spdlog v1.13.0 conda-forge/spdlog-feedstock#61What are "clobber warnings" and how does this help with them?
As described in #54 and rapidsai/rmm#1508, conda builds of
rmm
and other RAPIDS packages downstream of it are emitting dozens of warnings like these:These are saying that the
librmm
conda package contains files whose paths exactly conflict with those from thefmt
andspdlog
packages fromconda-forge
. When such packages are installed together, one will overwrite the other, which could lead to runtime issues.Removing the patches in
rapids-cmake
makes it more likely that builds will find the files they need from thefmt
andspdlog
conda-forge packages already present in the build environment, and therefore not vendor them, and therefore not ship a copy that causes conflicts.The text was updated successfully, but these errors were encountered: