Use libkvikio conda packages in libcudf, add explicit libcufile dependency.

[bdice]

Description

Provides a partial solution to #13230. During conda builds of libcudf, libkvikio is currently being fetched by CPM rather than supplied by a conda dependency. This means libkvikio headers are being shipped as part of libcudf's packages, which is not desirable.

I also added libcufile[-dev] as an explicit dependency of libcudf. Note that this is only available on linux64 (amd64), not aarch64 (arm64). We should always make the cuFile library available at build time for conda packages on amd64.

The impacts of this change are:

• libcudf conda packages will no longer ship libkvikio headers (those headers will instead be supplied by libkvikio packages). This reduces the size of libcudf and prevents clobbering files from libkvikio.
• libcudf will have a dependency on libcufile from the nvidia channel on linux64 (but not aarch64, since libcufile packages currently do not exist for aarch64).

This change does not impact builds outside of conda-build.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[bdice]

Verified that builds show the libkvikio conda package being used on amd64 and arm64.

-- CPM: using local package [email protected]

Also adding the cuFile dependency caused -DCUFILE_FOUND to be added to every compile line on amd64. This meant it had to build without sccache and took a long time! However, that's also a confirmation that cuFile support should now be working in packages built by CI. 😄 cc: @madsbk @vuule

[bdice]

FYI @ajschmidt8 -- you requested some changes on cuspatial and ucxx for recent changes to the update-version.sh script. I started to make those changes here but decided it's too large of a scope for this PR. Also, cudf's update script is significant more complex than ucxx/cuspatial and I didn't want to set the wrong example (since so many packages look to cudf for guidance). If you want this to be updated in the near term, I would appreciate your help!

[ajschmidt8]

Sounds good.

There are a lot of process changes I want to make around these scripts in the future.

Realistically I probably won't start that initiative until 23.08 or even 23.10.

[bdice]

For the record, pip wheels are built with cuFile support. I checked both CUDA 11.8 and CUDA 12.0 builds. AMD64 only, because cuFile isn't supported on ARM64.

  -- CPM: adding package [email protected] (branch-23.06)
  -- Found cuFile: /usr/local/cuda/lib64/libcufile.so

[vuule]

FWIW, looks good to me :)
This is an important fix for cuFile integration, thank you @bdice !

[bdice]

/merge