Make per-folder build.sc files a first-class feature to support large projects

[lihaoyi]

Having the build.sc file being monolithic is a problem for larger projects. Huge monolithic build.sc files are confusing to the developer, cause problems for distributing ownership via OWNERs files, compile slowly and non-incrementally, and invalidate the entire project's targets if so much as a newline is added.

None of these are problems for small projects, but become increasingly problematic when the project and engineering organization working on it grows. IMO a lot of the larger OSS projects using Mill already have reached the point where the size of build.sc is starting to become problematic, e.g. the build.sc for Mill itself, Ammonite, Scala-CLI are all over a thousand lines of rather dense code despite some effort to move logic into separate script files.

A lot of progress has already been made to allow Mill builds to be broken down into multiple build files:

1. Make builds able to depend on external projects #291 added support for "foreign modules" which are defined in build files that are not the root build.sc

2. Granular cache invalidation with multiple build files #1663 allows the import graph between multiple build files to be used for cache-invalidation purposes, so a change to one build file only invalidates targets defined in downstream builds

3. Remove Ammonite as a dependency, handle script running and bootstrapping ourselves #2377 removes the overhead that previously existed in Ammonite's handling of multiple scripts and making multiple scripts re-compile incrementally on a per-file granularity, making it even faster than re-compiling one monolithic script

These are all big steps towards making per-folder build.sc files a reality, but we're not there yet. The last things that are missing to make "per-folder" build.sc files a first class citizen include:

1. Enable targets in sub-folders to be run directly from the CLI, perhaps via a syntax such as ./mill foo/bar/baz.qux (if we want to strictly separate sub-folders and sub-modules) or foo.bar.baz.qux (if we don't mind mixing them together) or foo/bar:baz.qux (Bazel/Pants/Buck-style) to run the baz.qux target defined in foo/bar/build.sc

2. Have a story for discovering/skipping nested folders that contain build files, either opt-in (as is done in Gradle's settings.gradle) or opt out (as is done is Bazel's .bazelignore)

3. (optional) Allow modules and targets in nested build.sc files to be reference-able without an explicit import $file. e.g. Maybe just being able to say def moduleDeps = Seq($file.foo.bar.build.myModule), or def myTarget = T{... $file.foo.bar.build.myTarget() ...}

Once this is done, breaking up a large build.sc file into multiple smaller per-folder files would have the following benefits:

1. Avoiding huge multi-thousand-line build.scs in favor of smaller ones
2. Keeping build logic defined closer to the sub-folder that logic is building, making it easier to find
3. Allowing Zinc to perform incremental compilation at a per-file granularity
4. Allowing script-import-based target cache invalidation to avoid invalidating everything every time a build file is touched.
5. Be more aligned with other tools like SBT/Maven/Gradle/Bazel/Pants/Buck, which all allow per-folder build files

[DamianReeves]

Glad to hear this is being thought about.

Here is another example of a large Mill project: https://github.com/finos/morphir-scala .