Reconsider whether we need statically nested subpackages

[bgrant0607]

This has come up a few times, so I am creating an issue for it.

Statically nested subpackages raised some tricky cases in the v1 design, such as pipeline execution order, functions needing to execute multiple times over the same content, providing inputs to functions in nested subpackages, demand fetching of subpackages, independent upgrades of subpackages, and so on. Some issues are still outstanding, such as #1084 and #2451.

Now that we have the package orchestrator, config sync, and multiple repos and syncs in config sync (i.e., not just a monorepo), we should reconsider whether we want nested subpackages. Nested subpackages are not yet supported in porch (#3241), and we may not want to support them in porch.

Obvious alternatives to static subpackages are:

• The "app of apps" pattern, as it's described in ArgoCD. We could put RootSyncs or RepoSyncs into packages. One big disadvantage of that in most GitOps tools is that then those sync specs need to be manually maintained, such as to orchestrate rollouts or updates of pinned versions.
• Dynamic package dependencies. I'm sure some other packaging systems could serve as examples we could look at, such as https://crossplane.io/docs/v1.8/concepts/packages.html#configuration-packages

I view sets of variants of the same package as a specific pattern that deserves a more specific solution (package sets, as mentioned in #3121 (comment)) and #3347.

Some of the challenges with related packages would still exist, such as managing inputs to the package functions, but flat packages seems simpler.

cc @justinsb @mortent @droot

[justinsb]

I share your suspicion that it is unneeded.

I propose we continue with our app-based dogfooding and wait until we need it ourselves.

If we find that a bunch of things aren't needed, or (for example) we want to change the default update behaviour, then we declare a v2.

[bgrant0607]

@justinsb One case where nested subpackages has come up without is Config Connector: #3344

[yuwenma]

To capture what we discussed today for #3398.

The nested Ghost package may actually be desired.

(context: The ghost package has two nested packages: Ghost host and MariaDB.)

Pros:

• MariaDB can be replaced by other DB component. This pluggable component structure gives package authoring more flexibility.
• The two packages can be applied to different functions (e.g. set-label), and can be selected by different app names.
• Ghost host and MariaDB can come from different git repo. nested package allows them to have different upstreams and thus be updated differently.

Cons:

• They are highly correlated and need be deployed as a whole. Otherwise users are very likely to have a non-workable kpt package. (e.g. Shared env vars between Ghost host Deployment and Mariadab StatefulSet.
• Nested packages increase the complexity of multi package updates.