Moving Clarity to its own repo

[cylewitruk]

While implementing WASM for clarity it has become relatively clear for those of us working with it that "clarity" would be best served if it were to be extracted to its own repository to clearly separate the development of the clarity language/parsers/runtimes from that of the stacks blockchain itself.

This suggestion entails creating a new clarity repository within the stacks-blockchain org containing the following crates:

• clarity (core types, traits, high-level abstract business logic, etc.) - the current clarity crate, in other words, with some subsets of code extracted to separate modular crates as listed below.
• clar2wasm (the implementations for generating wasm from clarity)
• clarity-runtime-v1 (the current runtime to be replaced with wasm)
• clarity-runtime-wasm
• clarity-parser-v1 (the first version of the parser)
• clarity-parser-v2 (the current version of the parser)

Such a move would allow the blockchain to maintain a pinned version of clarity + parser + runtime while the development of Clarity and related continues, while also allowing smaller improvements in individual crates to be adopted by the blockchain/clarinet individually.

Not to mention testing which would become more isolated and to-the-point.

Thoughts?

[obycode]

Thanks Cyle! I agree, this would be very helpful for development and testing of Clarity.

[jcnelson]

I think Clarity should stay in the stacks-network/stacks-blockchain repo to facilitate development (in particular, Clarity depends on crates in stacks-blockchain), but it could be separated further from the blockchain than it currently is. Perhaps we could come up with a list of crates that we'd like to see built from the Clarity source?

[obycode]

I'm not quite clear how separating clarity out would make the coupling "more implicit and thus harder to identify and address." I thought it would be more explicit, and thus easier to identify and address. Can you explain that a bit, Jude?

[jcnelson]

Sure. The Clarity VM depends on the Stacks blockchain's storage facilities and data types to run, but not directly. For example:

• the Clarity storage back-end depends on a BurnStateDB and a HeadersDB which are really just traits specifically tailored for database access methods in the Stacks blockchain

• the Clarity DB is designed to work with a MARF, and provides trait wrappers around the Stacks blockchain MARF implementation

• the Clarity VM makes liberal use of Stacks data types in stacks_common, including epochs, block identifiers, cryptographic routines, and so on.

It would be misleading to say that Clarity is currently an independent virtual machine project from the Stacks blockchain -- it was definitely tailored to Stacks' needs, even though it currently makes an attempt to avoid directly depending on Stacks blockchain crates.

None of the above changes if we moved Clarity to its own repository tomorrow; it just gets more obfuscated. This, in turn, makes it harder to make contributions to Clarity, because contributors would still find themselves having to work within constraints imposed upon the VM by the blockchain implementation, even though the implementation isn't part of the same repository. I think this is a worse outcome than simply keeping the VM within the blockchain repo until such a time where these implicit couplings and others not listed can be addressed.

[jcnelson]

Like, maintainers of Clarity would have to NACK any PR that changes the Clarity DB in a way that would make it impossible to use with the Stacks MARF. This is not at all obvious to contributors if Clarity lives in a different repository. It's more obvious if Clarity stays where it is, because they can see how the above traits are used to wrap access to the Stacks blockchain (e.g. via the stackslib::clarity_vm module), and thus more quickly comprehend the design constraints that shape the VM implementation.

[obycode]

Thanks, great reasoning. I can see how it makes sense to work towards cleaning that up first.

[jcnelson]

Put another way, modularity-for-the-sake-of-modularity is an anti-pattern. The fact that these implicit couplings exist mean that we don't yet know of a good "narrow waist" API that could cleanly separate the Clarity VM from the Stacks blockchain. This is the fundamental problem that needs to be addressed, and it is not addressed -- and I'd argue worsened -- by moving clarity into a separate code repository.

EDIT: I see we replied at the same time @obycode ;)

[kantai]

In my opinion, repository choice primarily impacts the following:

1. Code ownership
2. Continuous Integration
3. Release management
4. Branch protection
5. Visibility

Notice, I don't think that it really impacts code architecture. I know that all of the components in the stacks-blockchain repo share a single cargo workspace (the top-level Cargo.toml), and that this does impose some constraints, I don't think that it matters much in practice.

Thinking through the above, though, I don't think item 1 is really relevant -- a new clarity repo would presumably be in stacks-network, and continue to be owned by Stacks Foundation, and I don't think any change there is really a technical question at all.

Now, for items 2-4, there may be something worth thinking about. Clarity and the rest of the stacks-blockchain do not obviously need to share a CI. Today, this isn't really much of an issue, because like the workspace member discussion, there really isn't a practical limit to how independently clarity can be tested with the same top-level github action workflows as the rest of the stacks-blockchain.

Release management and branch protection may have more benefits, though. Thinking about release management, clarity already produces an independent library on crates.io, but that seems kind of painful in the current repo setup (https://crates.io/crates/clarity-vm) -- maybe folks who have worked with getting those releases built can speak more to that (@obycode). But, it does seem at least doable for releases on crates.io to work even in the current setup (and could probably be made easier even with clarity remaining this repo). Where releases really just cannot be done is through github releases -- clarity cannot have its own releases there in a way that makes sense. I'm kind of ambivalent about this because I'm not sure how much that matters. Branch protection is maybe the most important issue here: for the stacks-blockchain repo, there's a pretty heavy PR process, and for good reason. Development for clarity on its own does not necessarily need to share this review process. The stacks-blockchain repo would apply its review process to clarity changes with a PR that updated its linked clarity version. Honestly, I'm conflicted here. Clarity is really fundamental to the stacks-blockchain in a way that no other dependency could be -- an update to the clarity version is super impactful to the stacks-blockchain, and waiting until the last moment to initiate a heavy review process would likely mean that the review process would be unsatisfactory. But on the other hand, maybe that just means that the review process in Clarity development should be rigorous enough to make that less of a burden in the end.

Anyways, this got kind of long and rambley, but I think my point is that a new repo probably won't make much of a difference in any event -- it would probably just be a lot of work to set up a new set of github actions, top-level files, etc., and the actual benefit of cleaning up interfaces could just be done in the current setup instead. Improvements to release management would probably be worthwhile, but I think they can also work within a single repo. If you're concerned that you want to maintain a clarity "mainline" independent of the stacks-blockchain/master, then we could just use a new branch for a clarity mainline.

[cylewitruk]

So, in general @jcnelson I do agree with you that modularity "for the sake of", or "premature modularization" usually entails more complexity than it's worth. However, in this case, I do see a clear domain boundary between blockchain and Clarity, though as you guys have been in on it's not as clear code-wise today as it could be.

The monorepo vs. multi-repo has been debated in hundreds of articles over the years and in the end comes down to preference. I personally (having come originally from svn+monorepo) have learned to embrace smaller, module-oriented repos primarily from a versioning+CI/CD perspective, but also from an access control perspective, which @kantai brought up above.

Clarity today has external projects dependent on its crate (e.g. Clarinet, which uses a newer version than stacks blockchain), which is something that "speaks to me" that it should be its own repo, with its own CI, releases and dependents can pin to the version they feel comfortable with. I don't see why the review process for Clarity would need to be any less rigorous than that of the blockchain core, or performed by different people - it's the same code, just in a different place, so it doesn't require all that much more effort.

I am as OP of course in-favor of working towards Clarity living in its own repo (under stacks-network of course). If we were only talking about one project then I probably wouldn't have the same view, but as we're looking at multiple crates being introduced it would feel better if these were in a clarity-centric repo vs. "on-the-side". Note that here I'm advocating for a clarity-monorepo ;)

[jcnelson]

Having read @kantai's reply, I'm even more inclined to keep Clarity as a crate within stacks-blockchain. What sticks out to me in particular is the fact that Clarity is an evolving piece of software that is almost entirely on consensus-critical code paths within Stacks. This does not apply to any other crate; for all intents and purposes, Clarity acts like a module within Stacks more than it acts like an independent VM project. Therefore, we should be treating it as part of the blockchain from an administrative standpoint -- it should be subject to the same code review process, CI test battery, SIP process, and so on that we currently apply to Stacks. This is much more practical if Clarity continues to live in the same repository as Stacks.

@cylewitruk Clarity today has external projects dependent on its crate (e.g. Clarinet, which uses a newer version than stacks blockchain)

Be that as it may, the "true" version of Clarity is always the one that ships with Stacks. If Clarinet's fork of Clarity (or any fork for that matter) behaves differently than Clarity-in-Stacks, then it's treated as a bug in the fork. This would be true regardless of where Clarity lived.

I am as OP of course in-favor of working towards Clarity living in its own repo (under stacks-network of course).

May I ask what specific problem you are trying to solve by doing so, which cannot be solved by continuing to allow Clarity to live in the stacks-blockchain repo? Related, can you elaborate on why it is insufficient for your needs to simply fork stacks-blockchain and strip out everything that isn't the clarity crate?