Planning for breaking changes in 0.12.x?

[lihaoyi]

It's been a bit over a year since 0.11.0 was released (June 2023), and I'd like to start thinking about what breaking changes we'd like to do so we can bundle them into a 0.12.0 release. Exact timeline isn't set, but I'm thinking end-2024 or so. We can continue maintaining a 0.11.x branch if we wish, but at some point we'll need to break compat and it would be good to consolidate the breakage so it happens as few times as possible

Some off the top of my head:

• Migrate build.sc to Scala 3 (1000USD Bounty) #3152 will be a major change in the Scala version used by the build.sc files. Any OSS libraries you use in your build.sc files may need to be updated, any internal libraries you use may need to be migrated (e.g. if they use macros). Scala 3.x is not source compatible with 2.13.x, so build.sc files may need to be adjusted to satisfy the new syntax. Scala 3 also introduces new capabilities that could change how we want to implement some things, e.g. some modules that are currently modelled as nested traits could be better modelled as traits taking parameters instead.

• Make per-folder build.sc files a first-class feature to support large projects #2637 will likely require changes in the build.sc file desugaring. Not sure exact details, but e.g. we would want to move RootModule from target-resolution-time to codegen-time so that it would apply not just to CLI invocations but also to in-code programmatic references between build.sc files as well (these do not exist today because only one build.sc is supported)

• Should we get rid of Agg in user-facing APIs, or just make it a type alias for Seq? It felt like a good idea at the time, and it's definitely a handy abstraction, but at this point I'm not sure it's worth the complexity and novelty cost v.s. just using Seq everywhere

• Would Make CompilationResult more generic #2781 be a breaking change?

• Redesign tasks that get an evaluator instance #502 does not have any plan or design ready to go yet, but if we do come up with something it would probably be breaking

• Should we make -j0 the default so parallelism becomes opt-out rather than opt-in? That would match the behavior of other tools like SBT or Bazel, and I think at this point parallelism is battle-tested enough we can turn it on by default

• Rework handling of mill-launcher in MillBuildModule #2703 would be a breaking change if we decided to expose the minimal classpath

• First class support for Java and Kotlin: bring mill-kotoin plugin into main repo, and split out a mill.javalib package for java projects to use

• If we don't expect to continue making major change in Mill 0.11.0, we could resolve Complete versioned documentation generation process for dynamic pages since 0.11.0-M8 #2455 by just generating it manually as manually saving the output as a sub-folder in the gh-pages branch that we link to from the main docs

We might not get to all of these, but it's definitely worth thinking about. Any other things that people may want to change that would involve breakage? @lefou @lolgab

[lolgab]

I wanted to change the way Mill environment variables to not propagate the Environment anymore.
It's a breaking change so I would be more confident to do in a major version change.
#2882

[lefou]

All these points sound good to me. Although in this summer I will probably not have so much time to implement anything of this.

Some of these changes are probably not necessarily breaking compatibility, e.g. making --jobs 0 the default.

More on -j0: I'd like to make our output easier readable, e.g. drop the self-erasing progress in favor of some nicer progress which shows the state of not only the last started job but also of some currently running jobs. Maybe, group and delay output of non-interactive tasks. But I have no clear idea yet. Also, I'd like to change how we handle commands, e.g. make them non-concurrent to other jobs and interactive by default.

I'd also like to see some native Mill launcher, so that repeated mill executions get instant and don't need to fire up the JVM each time. Unfortunately, I don't have much experience yet with either Scala Native or Graal native images. I think the latter might be our go-to solution, since we use junix-socket for Mill client-server communications and it looks like it comes with ready to use native image configs.

[lihaoyi]

More on -j0: I'd like to make our output easier readable, e.g. drop the self-erasing progress in favor of some nicer progress which shows the state of not only the last started job but also of some currently running jobs. Maybe, group and delay output of non-interactive tasks. But I have no clear idea yet.

SBT and Bazel both have similar TUIs for displaying the concurrent tasks going on; we could build something similar

Also, I'd like to change how we handle commands, e.g. make them non-concurrent to other jobs and interactive by default.

This is what Bazel does. Sounds reasonable to me

[sequencer]

My first wish is removing absolute Path dependency(maybe change to RelPath to workspace) to allow moving cache around different CI machines and even allow remote build support.

[lihaoyi]

All these points sound good to me. Although in this summer I will probably not have so much time to implement anything of this.

That's fine, I have some bandwidth, I can maybe put up some bounties, and we might not achieve everything in the list. As long as we have general agreement on the general direction we can move forward at whatever speed is feasible

Some of these changes are probably not necessarily breaking compatibility, e.g. making --jobs 0 the default.

It doesn't break binary compatibility, but it does break semantic compatibility. There historically hasn't been a requirement that build.sc logic is multithreading-safe, even if it happens mostly "for free" due to functional programming. Introducing such a requirement could require a bunch of third-party code to break in subtle ways and is worth a big version bump

[lihaoyi]

@lefou what do you think of targeting non-Scala deveopers with Mill? e.g. mainlining Kotlin support, breaking out a dedicated mill.javalib package, etc. We used to have some half-baked docs suggesting Mill could be used for Java, but I mean doing it properly. e.g. Properly meaning full length doc pages for getting started with Mill for Java, Java Module configs and Java Build Examples, and the same for Kotlin

That would be following the Gradle playbook: nobody uses Groovy or Grails these days, but they use Gradle anyway since it's better than Maven sometimes. Maybe Mill could find traction in Java/Kotlin land too; it's not like people there are in love with Maven/Gradle/Bazel.

I think there's a space for a build tool somewhere between those three: more scalable and flexible than Maven, more simple than Gradle or Bazel, even if a bit less scalable. e.g. Maybe targeting codebases up to 1-1000 developers with 1-1000 modules, but not for monorepos with >1000 developers which probably need Bazel anyway. Currently a lot of projects get started using Maven or Gradle and switch to Bazel later, and if they get started using Mill they could have a better experience up front and also delay the inevitable Bazel migration since Mill works fine with hundreds of modules (e.g. com-lihaoyi/mill has around 500 modules and still works great)

This would open up Mill to some avenues for growth beyond the small Scala community, while also giving folks in the broader JVM community a taste of Scala if Mill manages to become popular

Again, ignoring the work hours necessary. That's orthogonal and maybe can be sorted out later

[lefou]

You're preaching to the converted. I'm all in, since I think Mill is the better alternative for many Gradle users. About the mill.javalib package, we have some prior discussions (#1826 (comment) to just link one of them); in summary, it will be hard to de-entangle Java and Scala concerns and maybe not worth it. That's mainly, because we use Zinc and its workers also for pure-Java modules. There is also some more work needed to get the stackable-trait concept a better match to stackable compiler tool chains. I've seen lots of those stacked tool chains already, and also the issues that can occur. But these technically details aside (which can be solved) I'm all for it! Especially, since it's mostly the Scala-free documentation it needs to speak to the Java audience.

[lihaoyi]

Hah i missed that thread, thanks for the link

I think using Zinc isn't a blocker. We can have JavaModule use javac (in memory or via subprocess) and ScalaModule override it to use Zinc.

I'm thinking that even just having a mill.javalib package in the same scalalib module would be fine. The user will see "java", and won't see "scala" at all even if its used under the hood

[lefou]

About mainlining Kotlin, which is currently usable in Mill via https://github.com/lefou/mill-kotlin. That would be great! Unfortunately, non-Jetbrains tools are still second class citizens to the Kotlin API. There is currently no way to use the fast incremental compilation from Mill. Gradle and IntelliJ use some internal API. I opened issues and discussions about it, but there is still nothing we can use to date. Without that, it might get hard to come up with a stable API around Kotlin settings.

[lefou]

We could also provide a GroovyModule. Not that it's needed nowadays, but maybe it provides the needed support for bespoken legacy tool chains. I don't have insights in any larger closed-source Gradle project, so this is wild guessing.

[lihaoyi]

I think Java/Scala/Kotlin are the three main ones to target. Other languages, which do exist, are generally much smaller. Groovy, Clojure, and so on

I think having limitations around kotlin is reasonable. I don't know how fast kotlin changes, but I assume once tue dust has settled around kotlin 2.0 things shouldnt be moving that quickly. Even for internal APIs

But good to know you're in agreement with the direction.

[lefou]

• Support an alternative to $-imports #3062

I think this is a good opportunity to migrate to using directives.

Pros:

• Easier to parse
• IDEs, code formatter and hightlighters already support using directives
• No need to mangle the import statements before compilation
• No false error markers in IDEs
• Opportunity to add and unify more configuration options

[lefou]

• Refactor CompileModule out of JavaModule #3237

[lihaoyi]

To add to this list, bumping uPickle to 4.x can be part of Mill 0.12.x as well

[lolgab]

Remove default value from debugEnabled: Boolean in Logger.scala

[lolgab]

Remove all methods deprecated before Mill 0.11.0