Merge pull request #1 from kaweezle/feature/generators

✨Add generators
kaweezle · Jan 23, 2023 · aff71b3 · aff71b3
2 parents fe221e9 + ab61acd
commit aff71b3
Show file tree

Hide file tree

Showing 14 changed files with 584 additions and 70 deletions.
diff --git a/README.md b/README.md
@@ -6,6 +6,22 @@ krmfnbuiltin is a
 [kustomize plugin](https://kubectl.docs.kubernetes.io/guides/extending_kustomize/)
 that you can use to perform in place transformation in your kustomize projects.
 
+<!-- markdownlint-disable MD033 -->
+
+<!-- TABLE OF CONTENTS -->
+<details open="true">
+  <summary>Table of Contents</summary>
+  <ol>
+    <li><a href="#rationale">Rationale</a></li>
+    <li><a href="#usage-example">Usage Example</a></li>
+    <li><a href="#use-of-generators">Use of generators</a></li>
+    <li><a href="#installation">Installation</a></li>
+    <li><a href="#argo-cd-integration">Argo CD integration</a></li>
+    <li><a href="#related-projects">Related projects</a></li>
+  </ol>
+</details>
+<!-- markdownlint-enable MD033 -->
+
 ## Rationale
 
 `kustomize fn run` allows performing _in place_ transformation of KRM
@@ -16,7 +32,7 @@ a `container` or `exec` annotation in the transformer resource pointing to a krm
 function docker image or executable.
 
 `krmfnbuiltin` provides both the image and executable allowing the use of any
-builtin transformer.
+builtin transformer or generator.
 
 ## Usage Example
 
@@ -142,6 +158,200 @@ source:
 You now can commit the 10 modified manifests in your branch and deploy the
 applications.
 
+## Use of generators
+
+Let's imagine that one or more of your applications use an Helm chart that in
+turn creates applications. You pass the repo URL and target branch as values to
+the Helm Chart with the following:
+
+```yaml
+helm:
+  parameters:
+    - name: common.targetRevision
+      value: main
+    - name: common.repoURL
+      value: https://github.com/kaweezle/autocloud.git
+```
+
+For that particular transformation, JSON patches are not practical:
+
+```yaml
+patch: |-
+  - op: replace
+    path: /spec/source/repoURL
+    value: https://github.com/antoinemartin/autocloud.git
+  - op: replace
+    path: /spec/source/targetRevision
+    value: deploy/citest
+  - op: replace
+    path: /spec/source/helm/parameters/1/value
+    value: https://github.com/antoinemartin/autocloud.git
+  - op: replace
+    path: /spec/source/helm/parameters/0/value
+    value: deploy/citest
+```
+
+You need to hardcode the index of the value to replace in the array, which is
+error prone, and you start duplicating values.
+
+It would be better to have a unique _source_ with the right values, and do
+`replacements` where needed. You can inject the values with a
+`ConfigMapGenerator`
+
+```yaml
+# 01_configmap-generator.yaml
+apiVersion: builtin
+# Use this to inject current git values
+# kind: GitConfigMapGenerator
+kind: ConfigMapGenerator
+metadata:
+  name: configuration-map
+  annotations:
+    config.kubernetes.io/function: |
+      exec:
+        path: krmfnbuiltin
+# When using GitConfigMapGenerator, these are automatically injected
+literals:
+  - repoURL=https://github.com/kaweezle/autocloud.git
+  - targetRevision=deploy/citest
+```
+
+And then use a `ReplacementTransformer` to inject the values:
+
+```yaml
+# 02_replacement-transformer.yaml
+apiVersion: builtin
+kind: ReplacementTransformer
+metadata:
+  name: replacement-transformer
+  namespace: argocd
+  annotations:
+    # Put this annotation in the last transformation to remove generated resources
+    config.kubernetes.io/prune-local: "true"
+    config.kubernetes.io/function: |
+      exec:
+        path: krmfnbuiltin
+replacements:
+  - source:
+      kind: ConfigMap
+      fieldPath: data.repoURL
+    targets:
+      - select:
+          kind: Application
+          annotationSelector: "autocloud/local=true"
+        fieldPaths:
+          - spec.source.repoURL
+          # This field specification is not related to the index
+          - spec.source.helm.parameters.[name=common.repoURL].value
+  - source:
+      kind: ConfigMap
+      fieldPath: data.targetRevision
+    targets:
+      - select:
+          kind: Application
+          annotationSelector: "autocloud/local=true"
+        fieldPaths:
+          - spec.source.targetRevision
+          - spec.source.helm.parameters.[name=common.targetRevision].value
+```
+
+Some remarks:
+
+- ✔️ The actual values (repo url and revision) are only specified once.
+- ✔️ `spec.source.helm.parameters.[name=common.repoURL].value` is path more
+  specific than `/spec/source/helm/parameters/1/value`.
+- ✔️ The functions file names are prefixed with a number prefix (`01_`, `02_`)
+  in order to ensure that the functions are executed in the right order.
+- ✔️ In the last transformation, we add the following annotation:
+
+  ```yaml
+  config.kubernetes.io/prune-local: "true"
+  ```
+
+  In order to avoid saving the generated resources. This is due to an issue in
+  kustomize that doesn't filter out resources annotated with
+  `config.kubernetes.io/local-config` in the case you are using
+  `kustomize fn run` (although it works with `kustomize build`).
+
+As a convenience, for this specific use case, we have added a
+`GitConfigMapGenerator` that automatically adds the relevant resources, while
+some people may consider this overkill.
+
+## Installation
+
+With each [Release](https://github.com/kaweezle/krmfnbuiltin/releases), we
+provide binaries for most platforms as well as Alpine based packages. Typically,
+you would install it on linux with the following command:
+
+```console
+> KRMFNBUILTIN_VERSION="v0.1.0"
+> curl -sLo /usr/local/bin/krmfnbuiltin https://github.com/kaweezle/krmfnbuiltin/releases/download/${KRMFNBUILTIN_VERSION}/krmfnbuiltin_${KRMFNBUILTIN_VERSION}_linux_amd64
+```
+
+## Argo CD integration
+
+`krmfnbuiltin` is **NOT** primarily meant to be used inside Argo CD, but instead
+to perform _structural_ modifications to the source **BEFORE** the commit.
+
+Anyway, to use `krmfnbuiltin` with Argo CD, you need to:
+
+- Make the `krmfnbuiltin` binary available to the `argo-repo-server` pod.
+- Have Argo CD run kustomize with the `--enable-alpha-plugins --enable-exec`
+  parameters.
+
+To add krmfnbuiltin on argo-repo-server, the
+[Argo CD documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/custom_tools/)
+provides different methods to make custom tools available.
+
+If you get serious about Argo CD, you will probably end up cooking your own
+image. This
+[docker file](https://github.com/antoinemartin/autocloud/blob/deploy/citest/repo-server/Dockerfile#L45)
+shows how to use the above installation instructions in your image. To
+summarize:
+
+```Dockerfile
+FROM argoproj/argocd:latest
+
+ARG KRMFNBUILTIN_VERSION=v0.1.0
+
+# Switch to root for the ability to perform install
+USER root
+
+# Install tools
+RUN apt-get update && \
+    apt-get install -y curl && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/* && \
+    curl -sLo /usr/local/bin/krmfnbuiltin https://github.com/kaweezle/krmfnbuiltin/releases/download/${KRMFNBUILTIN_VERSION}/krmfnbuiltin_${KRMFNBUILTIN_VERSION}_linux_amd64
+
+USER argocd
+```
+
+You also need to patch the `argo-cm` config map to add the parameters. The
+following is a strategic merge patch for it:
+
+```yaml
+# argocd-cm.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-cm
+data:
+  # Options to enable exec plugins (krmfnsops).
+  kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
+  ...
+```
+
+## Related projects
+
+[kpt], from Google, takes this in place transformation principle to another
+level by making resource configuration packages similar to docker images. In
+this model, a generator or a transformer along its parameters in much like a
+line in a dockerfile. It takes a current configuration as source and generates a
+new configuration after transformation.
+
+While it has not been tested, krmfnbuiltin should work with [kpt].
+
 <!-- prettier-ignore-start -->
 
 [kpt]: https://kpt.dev/guides/rationale

diff --git a/go.mod b/go.mod
@@ -3,43 +3,57 @@ module github.com/kaweezle/krmfnbuiltin
 go 1.18
 
 require (
+	github.com/go-git/go-git/v5 v5.5.2
 	golang.org/x/tools v0.5.0
 	sigs.k8s.io/kustomize/api v0.12.1
 	sigs.k8s.io/kustomize/kyaml v0.13.10
+	sigs.k8s.io/yaml v1.2.0
 )
 
 require (
+	github.com/Microsoft/go-winio v0.5.2 // indirect
+	github.com/ProtonMail/go-crypto v0.0.0-20221026131551-cf6655e29de4 // indirect
 	github.com/PuerkitoBio/purell v1.1.1 // indirect
 	github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
+	github.com/acomagu/bufpipe v1.0.3 // indirect
 	github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a // indirect
+	github.com/cloudflare/circl v1.1.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/emirpasic/gods v1.18.1 // indirect
 	github.com/evanphx/json-patch v4.11.0+incompatible // indirect
 	github.com/go-errors/errors v1.4.2 // indirect
+	github.com/go-git/gcfg v1.5.0 // indirect
+	github.com/go-git/go-billy/v5 v5.4.0 // indirect
 	github.com/go-openapi/jsonpointer v0.19.5 // indirect
 	github.com/go-openapi/jsonreference v0.19.5 // indirect
 	github.com/go-openapi/swag v0.22.3 // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/google/gnostic v0.5.7-v3refs // indirect
-	github.com/google/go-cmp v0.5.7 // indirect
-	github.com/imdario/mergo v0.3.6 // indirect
+	github.com/imdario/mergo v0.3.13 // indirect
 	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
+	github.com/kevinburke/ssh_config v1.2.0 // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
 	github.com/mitchellh/mapstructure v1.4.3 // indirect
 	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
+	github.com/pjbgf/sha1cd v0.2.3 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
+	github.com/sergi/go-diff v1.1.0 // indirect
+	github.com/skeema/knownhosts v1.1.0 // indirect
 	github.com/spf13/cobra v1.4.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/xanzy/ssh-agent v0.3.3 // indirect
 	github.com/xlab/treeprint v1.1.0 // indirect
+	golang.org/x/crypto v0.3.0 // indirect
 	golang.org/x/mod v0.7.0 // indirect
 	golang.org/x/net v0.5.0 // indirect
 	golang.org/x/sys v0.4.0 // indirect
 	golang.org/x/text v0.6.0 // indirect
-	golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect
 	google.golang.org/protobuf v1.28.0 // indirect
+	gopkg.in/warnings.v0 v0.1.2 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	k8s.io/kube-openapi v0.0.0-20220401212409-b28bf2818661 // indirect
 	k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 // indirect
-	sigs.k8s.io/yaml v1.2.0 // indirect
 )