Uniformity of kcl run and kcl mod add CLI, etc

[zong-zhe]

Uniformity of kcl mod add and kcl run CLI

1. This issue is created for

1. Support for seamless user experience with https/http.
2. Support for flexible referencing across multiple repositories including OCI, Git, and Localpath.
3. Uniformity in the style of add/run/push/pull commands to reduce user cognitive load.

2. The command design

Taking the kcl mod add command as an example:

2.1 Direct use of https/http

We support the direct use of protocols starting with http/https, and other information can be identified using a suffix in the form of ?<key>=<value>.

Taking https as an example:

kcl mod add https://ghcr.io/kcl-lang/k8s?version=0.0.1

However, with this syntax,kpm cannot determine whether https://ghcr.io/kcl-lang/k8s?version=0.0.1 originates from an OCI Registry or a Git Repo. Therefore, kpm will attempt to fetch the content sequentially following the order of OCI -> Git. If the OCI fetch is successful, package from oci will be used; if a Git repo fetch is successful, package from git will be used. If both fail, an error will be raised.

2.2 Specifying using the oci/git protocol

kpm also supports the use of specific protocols to designate the source.

Specifying the source using the oci:// protocol:

kcl mod add oci://ghcr.io/kcl-lang/k8s?version=0.0.1
kcl mod add oci://ghcr.io/kcl-lang/k8s?digest=0.0.1

Specifying the source using the git:// protocol:

kcl mod add git://github.com/test/aaa?tag=0.0.1
kcl mod add git://github.com/test/aaa?commit=kj8f7k
kcl mod add git://github.com/test/aaa?branch=main

For private git repo, ssh:// protocol is supported.

kcl mod add ssh://[email protected]/test/aaa?tag=0.0.1

For more than one parameters, & is used to connect different parameters.

kcl mod add ssh://[email protected]/test/aaa?tag=0.0.1&commit=f78sf8g

3. The dependencies in kcl.mod

For kcl.mod, kpm supports writing dependencies as below, take k8s as example:

3.1 The default source

The following show the KCL dependencies from default source, it can be oci registry or git repo.

k8s = "1.27"

If the default source is OCI registry, the configuration for kpm in ~/.kcl/kpm/config.json as following:

{
    "DefaultRepo": {
        “Oci”: {
            "DefaultOciRegistry": "localhost:5001",
            "DefaultOciRepo": "test",
        }
    }
}

If the default source is Git registry, the configuration for kpm in ~/.kcl/kpm/config.json as following:

{
    "DefaultRepo": {
        “Git”: {
            "DefaultHost”: “github.com,
            "DefaultPath”: "test",
        }
    }
}

3.2 OCI Registry

Specify the KCL package source is OCI Registry

version it the oci tag

k8s = { oci=“oci://ghcr.io/kcl-lang/k8s”, version=“0.0.1”}

3.3 Git Registry

The design before:

#266

4. API

In KpmClient, we provide a Visit method to simplify the package download process, which allows you to directly access the contents of a KclPkg. You can access the KCL packages from different sources through this method, where source refers to the various addresses of the KCL packages mentioned earlier, including local paths and the OCI or GIT addresses in URL format.

To facilitate understanding, this Visit works similarly to the workflow of web browsers, providing it with a URL, it will return the specific webpage located at that address.

For example:

// Create a `KpmClient` to use the `Visit`
Kpmcli, err := client.NewKpmClient()
if err != nil {
    // Handle error
    return
}

err = Kpmcli.Visit(

    // The first argument is the source of the KCL package you wish to access, just like a browser's URL.
    "oci://ghcr.io/kcl-lang/helloworld?tag=0.0.1",

    // The second argument is a method to access the KCL package from the source
    //     - The first argument corresponds to the KCL package of the source.
    //     - The second argument is error information, which means if an error occurs while accessing the source, you can obtain the specific error logs through this err.
    func(kclPkg *pkg.KclPkg, err error) error {
        if err != nil {
            return err
        }

        // TODO: Do you own operation for the KCL package here ! ! !

        return nil
    },

)

Some examples of other tools are referenced

• Go-getter: github.com/hashicorp/go-getter?ref=abcd1234
• Github: https://github.com/zong-zhe/kpm/releases/tag/v0.2.8
• Oras: localhost:5000/hello-artifact:v2
• Helm: helm pull oci://localhost:5000/helm-charts/mychart --version 0.1.0
• FluxCD:
• • flux pull artifact oci://ghcr.io/org/manifests/app:v0.0.1
• • flux create secret git podinfo-auth --url=ssh://[email protected]/stefanprodan/podinfo \

Considerations in Design

• Why do we need to specify fields through ?version=0.0.1 rather than following the oci style of using :v0.0.1?

We aim to provide users with a uniform style, where one kind URL can deduce another. :v0.0.1 corresponds to version=v0.0.1. If applied to other sources that do not have the concept of version, e.g. git repo, this would add a cognitive burden. That is, would :v0.0.1 still be effective if the source lacks the concept of version? Therefore, in the current version, we consider not adding this feature. If there is a strong demand for adding :v0.0.1 in the future, we will reconsider whether to add it.

The Progress

[ ] - Remove the http/https switch and adjust oci access layer in kpm to support http/https non-awareness
[ ] - Adjusting the cli to supports unified commands.
[ ] - Add default source to avoid overwriting by OCI source and Git source.
[ ] - Complete all documents about the new cli.
[ ] - Refer to how KCL-KRM requests resources

[Vanshikav123]

Hello ! @zong-zhe @Peefy is this issue still relevant can i work on it?

[zong-zhe]

Hi @Vanshikav123 😃

Welcome! You can begin with this task.

Remove the http/https switch and adjust oci access layer in kpm to support http/https non-awareness

I provide some more detailed information. Currently, kpm supports OCI Registries with URLs beginning with either 'https' or 'http' protocols.

• More details about 'kpm oci registry supports' - https://www.kcl-lang.io/docs/user_docs/guides/package-management/how-to/kpm_oci

kpm requires the environment variable OCI_REG_PLAIN_HTTP to be set, or you can adjust the field DefaultOciPlainHttp in kpm configuration file ~/.kcl/kpm/.kpm/config/kpm.json for its setup. Both OCI_REG_PLAIN_HTTP and DefaultOciPlainHttp influence the value of a switch variable within kpm.

• The switch variable is here:

  kpm/pkg/oci/oci.go

  Line 118 in bb39f68

  repo.PlainHTTP = settings.DefaultOciPlainHttp()

The state of this switch variable restricts kpm to effect only one protocol at a time—either https or http. This is quite detrimental to the user experience. It makes the user feel very confused. Users should not need to manually control this switch; kpm should support both http and https starting OCI registry urls without a switch control.

kpm utilizes the third-party library oras-go to implement the oci registry supports. I have noticed that oras is capable of working with OCI registry URLs starting with 'http' or 'https'.

• oras is https://github.com/oras-project/oras
• oras-go is https://github.com/oras-project/oras-go

Therefore, you could begin by research how the ORAS tool achieves this, and then try to implement the same functionality in kpm.

If you can find the apis used by oras, using those apis directly in kpm to implement the same functionality will make it very easy to solve this problem. 😊😊😊

[Vanshikav123]

Hi @Vanshikav123 😃

Welcome! You can begin with this task.

Remove the http/https switch and adjust oci access layer in kpm to support http/https non-awareness

I provide some more detailed information. Currently, kpm supports OCI Registries with URLs beginning with either 'https' or 'http' protocols.

• More details about 'kpm oci registry supports' - https://www.kcl-lang.io/docs/user_docs/guides/package-management/how-to/kpm_oci

kpm requires the environment variable OCI_REG_PLAIN_HTTP to be set, or you can adjust the field DefaultOciPlainHttp in kpm configuration file ~/.kcl/kpm/.kpm/config/kpm.json for its setup. Both OCI_REG_PLAIN_HTTP and DefaultOciPlainHttp influence the value of a switch variable within kpm.

• The switch variable is here:

  kpm/pkg/oci/oci.go

  Line 118 in bb39f68

  repo.PlainHTTP = settings.DefaultOciPlainHttp()

The state of this switch variable restricts kpm to effect only one protocol at a time—either https or http. This is quite detrimental to the user experience. It makes the user feel very confused. Users should not need to manually control this switch; kpm should support both http and https starting OCI registry urls without a switch control.

kpm utilizes the third-party library oras-go to implement the oci registry supports. I have noticed that oras is capable of working with OCI registry URLs starting with 'http' or 'https'.

• oras is https://github.com/oras-project/oras
• oras-go is https://github.com/oras-project/oras-go

Therefore, you could begin by research how the ORAS tool achieves this, and then try to implement the same functionality in kpm.

If you can find the apis used by oras, using those apis directly in kpm to implement the same functionality will make it very easy to solve this problem. 😊😊😊

Thanks for providing me with the references will come up with a solution soon !

[Peefy]

Can git://github.com be simplified to github.com and git://github.com is also allowed. Ref kubectl apply