Simplify and standardize (or replace) build.sh scripts

[vyasr]

Every RAPIDS project has a build.sh script at the root. It is a bash script that simplifies the process of building the various components of the project. It was designed with the idea of being run in an environment where all the dependencies are installed, but we want to build both the C++ and Python components with the Python packages using the built C++ library rather than prebuilt artifacts. Over time, the scope of this script has increased: it is invoked in all the conda builds, and it supports a number of additional local user conveniences on a per-repo basis such as building Python packages in editable mode, passing arbitrary extra CMake arguments through, building tests and/or benchmarks, etc. However, this script is probably not the best place for most of this logic. #52 should help address one of the most common use cases for build.sh: accumulating a set of CMake options into a common, easy-to-use grouping. Other cases may no longer be as important as they once were due to improvements in our Python build infrastructure since we now use scikit-build-core to effectively use CMake to coordinate the build of the compiled components of the Python package with the C++ package. Once #52 is complete we should do a thorough audit of build.sh scripts to see how far down we can simplify them. To the extent possible, we should find ways to make these scripts uniform across RAPIDS. If the scripts wind up being truly trivial, then we should consider if there are alternative ways to educate developers to obviate the need for these scripts.

[betatim]

One input related to considering changes to build.sh: I am always puzzled/frustrated-then-read-the-docs by the fact that I can't run pip install -e. from the top level directory to get going working on a "python project" like cuml. While it contains code that isn't Python code, for me it is a "Python project" because the main way you use it is from Python. For me as a Python developer the "obvious" way to get started is to run pip install -e. as this works for most projects.

The main benefit of "fitting in" is that when you arrive in a new-to-you project you end up feeling at home when things work the way they work in most other places. While this might only be a small comfort factor, when the opposite is true (standard way doesn't work) my thought process is "oh boy, this is going to take a while" - so the reason to enable "standard tooling" isn't because it is such a big win, more to avoid a big loss.

Maybe I am getting too old and cynical :D But I do feel like anec-data-ly "custom build commands" tend to correlate with a less good developer experience. I often find "not excellent documentation", "exactly one way to do it and no other", "only works on one platform", "requires complex tools like bazel before you can do anything", "running the tests is not just pytest but some custom thing", etc all correlate with "custom build command". And on top of it I am unsure if it is really needed. Plenty of complex build stuff goes on when you build scikit-learn, but it can all be triggered by pip install --no-build-isolation -e. (it doesn't achieve the gold standard and it pains me...).

TL;DR: Old man Tim yells at cloud build.sh ;) If we do touch it, maybe we can move closer to pip install -e. just working.

[vyasr]

One of my main efforts over the years has been to try to get us closer to being able to use standard build mechanisms (run cmake in the cpp dir, run pip install in the Python dir) and have those "just work". I don't think we'll ever be able to make pip install work in the top-level directory because RAPIDS libraries in general have multiple Python packages per repo, so you will at least have to cd to the right dir, but we can definitely get a lot closer. A combination of #33 (which will enable installing the C++ library as a wheel, so you won't have to build it if you only want to do Python dev), #52 (so that build settings are captured in CMake presets), and #31 (so that you don't have to deal with the CUDA suffixes of dependencies) should get us much closer though.

I know @rlratzel is one of the usual proponents of build.sh, so I'll let him chime in if he wants to respond to your other thoughts 😄

[betatim]

That sounds great!

[rlratzel]

I'm in favor of these changes, especially if they make build.sh smaller and easier to further standardize across RAPIDS.

I suspect build.sh is being classified as a "custom build command" simply because this issue isn't closed yet :) Ideally, build.sh is just easily discovered, runnable build documentation that provides a convenience wrapper around standard commands. build.sh and standard/modern build commands are not mutually exclusive.

A bit more history: build.sh was meant to be a façade to provide a consistent (and greatly simplified at the time) interface for devs to build the libraries in their repo. Quickly after these scripts were added ~5 years ago, questions about how to build various libraries, especially from new devs, dropped dramatically (granted, docs could have been better too, but still…) As cmake options were added and python build tooling changed, build.sh was updated and devs didn't miss a beat. build.sh was especially successful for handling build tool bug workarounds too, like this one.

This might seem trivial but I like being able to switch to any RAPIDS repo and run build.sh -h to see what libraries and build options are available. This is easier and faster than searching the README/CONTRIBUTING for the same info. I'd hate to see this go.

One comment I'm confused about is:

It was designed with the idea of being run in an environment where all the dependencies are installed, but we want to build both the C++ and Python components with the Python packages using the built C++ library rather than prebuilt artifacts.

build.sh does not set up a build environment, but it will build python extensions for the C++ libraries using the artifacts it installed in a prior build step from the source tree. And if you have prebuilt C++ libraries installed, the python extension builds will use those instead and not require you to build the C++ libraries. This has been the case for years and appears to work for me this way today (unless I'm misunderstanding something).

[vyasr]

I suspect build.sh is being classified as a "custom build command" simply because this issue isn't closed yet

Well to be fair this issue is only two weeks old, so I doubt it's responsible for triggering much discussion yet :)

build.sh was meant to be a façade to provide a consistent (and greatly simplified at the time) interface for devs to build the libraries in their repo. Quickly after these scripts were added ~5 years ago, questions about how to build various libraries, especially from new devs, dropped dramatically (granted, docs could have been better too, but still…)... As cmake options were added and python build tooling changed, build.sh was updated and devs didn't miss a beat. build.sh was especially successful for handling build tool bug workarounds too

I think this anecdote illustrates a few things. First, build.sh was absolutely valuable at some point for reducing the complexity of building when onboarding new developers. It collected common knowledge into a script so that not everyone needed to know the ins and outs of each build. Second, our builds were too specialized and complicated, and build.sh was used because fixing the underlying issues with our CMake or pip-based builds would have been much harder. That is no longer true, though; we ought to be able to create just as good of an experience with standard tools now as our build systems have matured. Third, we need to properly document our CMake options (and can improve that using presets, see #52). Also, we need to document our Python builds. I strongly prefer projects that use standard commands (autoconf/cmake/etc) and document their options to projects that use a custom script. That leads me to the notes below.

This might seem trivial but I like being able to switch to any RAPIDS repo and run build.sh -h to see what libraries and build options are available. This is easier and faster than searching the README/CONTRIBUTING for the same info. I'd hate to see this go.

I'd argue for a broader point of view. I like being able to look at any piece of OSS and know that I can probably get started with a small, simple set of commands: make, ./configure && make, cmake..., bazel, etc for C++, or pip install... for Python, etc. If I have to use something more custom like build.sh I immediately assume the project's build is fragile and complicated (also my immediate response is to actually read the build.sh script to see what it's doing, but I realize that's probably more my personal idiosyncrasies than a normal response). More often than not it means it's going to be very easy to bork my environment if I use the script slightly incorrectly, so I end up stepping more gingerly around these scripts than I would around standard build commands that invoke some level of trust.