Support kustomize hydration in skaffold

[yuwenma]

• Provide basic kustomize images for kpt hydration pipeline.
• Enable all the following kustomize build features.
• Document the usages, including the flags/args passed into the kustomize image.

[mikebz]

HI @yuwenma I think that kind of a request is more of a project than a feature. Are we talking about making it easy for people who use kustomize to use kpt? What is the user scenario here?

[mikebz]

I think this is too broad of an issue and we would need to decompose this to make it actionable. I am going to close this as "won't fix" for now but happy to talk in more detail about the workflows we want to support.

[yuwenma]

HI Yuwen Ma I think that kind of a request is more of a project than a feature. Are we talking about making it easy for people who use kustomize to use kpt? What is the user scenario here?

@mikebz Yeah, agree this would be a big effort. This is for skaffold to use kustomize in a kpt hydration pipeline. We would need a progress tracker. What do you suggest on tracking the kustomize function?

[selfmanagingresource]

I suggest we keep this open until we can create a project for it. Kustomize support is a very high priority

[bgrant0607]

There are scenarios other than skaffold for using kpt and kustomize together. Ideally, custom integrations of kpt and kustomize wouldn't be needed for each such scenario.

What's the most challenging part of wrapping kustomize with a KRM function?

[mikebz]

I think there are different approaches to solving this problem. If we are talking about just creating a function that calls kustomize that this task is not as big. If we are talking about having feature parity between kpt and kustomize then it's a bigger issue. I read it as the latter at first.

I am reactivating this and looping in @mengqiy for the function implementation.

Just so the community is clear about our goal here I would suggest updating the original request text - I am happy to do that as well @yuwenma if you don't mind me doing the edits.

[mengqiy]

It should fairly straightforward to wrap kustomize in a kpt function. We can use kustomize's existing CRD (apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization) as the function config.

[mikebz]

@selfmanagingresource mentioned large customers demand, now we have strong signal from @bgrant0607 and @yuwenma . I'd consider this as a higher priority.

CC: @frankfarzan

[Shell32-Natsu]

We used to have a kustomize function in the function catalog but for some reason (I don't remember) we removed it.

[mikebz]

is this something we can potentially restore and provide a support path? Seems like an easy way to create a bridge for folks who have some of their config done with kustomize to adopt kpt hydration pipeline.

[mengqiy]

Ref: #1078
We may want to hear from @frankfarzan why he suggested to delete the function.

[bgrant0607]

Do we have a way to measure how many times functions are used? That could help us assess utility.

[mengqiy]

No at the moment.
GCR doesn't natively support pull stats. We can get it from GCS logs by ourselves. We have a plan to do it, but we don't have bandwidth to implement it right now. We can try to do it in early Q2.

[frankfarzan]

There are multiples aspects to consider here:

1. UX

• What's the desired e2e user journey here? This deserves a brief user guide before we can about function impl.
• In this user journey, what are the personae involved? Is the same person maintaining the kustomize directory and a kpt package? If so, why would they want this complexity? If not, are they treating kustomize directory as reusable off-the-shelf-buncle like a Helm chart developed by another persona?
• What's the day 0 and day N stories?
• Is this being run imperatively once by a human to hydrate a kustomize directory?
• Is this function being run declaratively as a generator on every hydration?
• Where's the kustomize directory? Does the user want to vendor everything to make hydration deterministic? If the kustomize directory is on the local filesystem, how does it mesh with kpt packaging?

2. Implementation

(Depending on UX questions above)

If running declaratively, the function need to mount the local filesystem or can all the resources be passed in as input KRM?
If running declaratively, does the function need network access to resolve remote bases?

The reason previous function 1 was removed because:

a) There weren't clears answers to UX questions above. As a result, it was confusing to users.
b) It's effectively just a thin wrapper around kustomize build which is run imperatively requiring mounting local filesystem and network access.

[marshall007]

@frankfarzan we find this direction of stripping away native kustomize support confusing (i.e. #1078 and this following discussion). I opened up #1545 separately because I didn't want to derail this discussion with my naivety towards the situation.

b) It's effectively just a thin wrapper around kustomize build which is run imperatively requiring mounting local filesystem and network access.

I understand the challenge from kpt's perspective here, particularly related to network access. As I noted in #1545 we deliberately don't use remote refs in our kustomizations but obviously it would have to be supported even if it is not considered a best practice at apply-time.

To us it feels like many challenges arise from lack of native kustomize support. Here is an example workflow that proves challenging:

# cat my-package/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- inventory-template.yaml
- subpackage-1/
- ingress-nginx/
components:
- ingress-nginx/aws/

---
# cat my-package/ingress-nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- upstream/

---
# cat my-package/ingress-nginx/aws/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
# https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
- name: ingress-nginx-controller
  behavior: merge
  literals:
  - use-proxy-protocol="true"
  - real-ip-header=proxy_protocol
patches:
- path: aws-service-annotations.yaml
  target:
    kind: Service
    name: ingress-nginx-controller

$ kustomize build my-package/ | kpt live apply # this is slightly annoying, but fine
$ kpt live diff my-package/ingress-nginx # something like this is basically impossible
$ kpt live destroy my-package/ingress-nginx # this is similarly impossible

In other words, because kpt lacks any understanding of the package structure as far as kustomize is concerned, it is impossible for it to operate on or inspect subtrees of the structure that was originally applied. Given that kpt is fundamentally a package manager, this is an example of the sort of thing I would expect it to be intelligent about. Currently if you want to operate on subpackages you must first manipulate your root kustomization.yaml accordingly and then pipe the output of kustomize build which is error-prone and confusing.

In case that example wasn't clear, note that the root my-package/ingress-nginx/kustomization.yaml of the ingress package does not include the aws-specific component by default. That is only referenced in my-package/kustomization.yaml. This is the technically interesting/challenging bit I am trying to focus the discussion on.

[yuwenma]

Frank Farzan we find this direction of stripping away native kustomize support confusing (i.e. #1078 and this following discussion). I opened up #1545 separately because I didn't want to derail this discussion with my naivety towards the situation.

b) It's effectively just a thin wrapper around kustomize build which is run imperatively requiring mounting local filesystem and network access.

I understand the challenge from kpt's perspective here, particularly related to network access. As I noted in #1545 we deliberately don't use remote refs in our kustomizations but obviously it would have to be supported even if it is not considered a best practice at apply-time.

To us it feels like many challenges arise from lack of native kustomize support. Here is an example workflow that proves challenging:

# cat my-package/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- inventory-template.yaml
- subpackage-1/
- ingress-nginx/
components:
- ingress-nginx/aws/

---
# cat my-package/ingress-nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- upstream/

---
# cat my-package/ingress-nginx/aws/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
# https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
- name: ingress-nginx-controller
  behavior: merge
  literals:
  - use-proxy-protocol="true"
  - real-ip-header=proxy_protocol
patches:
- path: aws-service-annotations.yaml
  target:
    kind: Service
    name: ingress-nginx-controller

$ kustomize build my-package/ | kpt live apply # this is slightly annoying, but fine
$ kpt live diff my-package/ingress-nginx # something like this is basically impossible
$ kpt live destroy my-package/ingress-nginx # this is similarly impossible

Thank you for the input @marshall007, just a tip for your kpt live diff/destry use case, you can try using a tmp dir as an intermediate place to hold the kustomize output before they are "kpt-live" applied to the cluster. e.g.

$ kpt live init tmp-dir
$ kustomize build my-package/ >>  tmp-dir/manifests.yaml
$ kpt live apply/diff/destroy tmp-dir

[mikebz]

@marshall007 just had more time to think through your scenario. First of all I want to say that kpt is focused on in place hydration while kustomize by design does out of place hydration. I think the functionality where one issues commands like kpt live diff against a folder that doesn't have fully hydrated config (WET config) poses a few challenges. I think to be consistent with the rest of kpt UX and having the final WET config on disk we should see if having a kustomize function would be sufficient for your purposes.

Let's say there was a kustomize function that given a folder to do a kustomize build and an output of that folder could enable someone to produce the WET config. The question then becomes if the source of kustomize build (DRY) and the result are in the same kpt package (maybe as sub-packages) how do you then let kpt know to ignore the DRY config?

When you have the pattern of

1. kpt pkg get
2. kustomize build
3. kpt live apply / diff / etc.

The communication from step 2 to step 3 generally happens by having a separate output directory (as @yuwenma mentions) or via STDIN/STDOUT.

There are solutions here by telling kpt to ignore directories or maybe making sure that the kustomize sources are remote, but then is the solution more straight forward than the problem?

[viglesiasce]

Here are my use cases:

Use Case 1

(Roughly 70% of all k8s users fit in this boat)

1. I dont use Kpt today, I use Helm or Kustomize.
2. I want to use Kpt for just validations and persistent mutations (labels, namespaces)

Use Case 2

1. I use an upstream COTS project that has a Helm chart (Redis/MySQL/CockroachDB)
2. I have all of my config in Kustomize at the moment
3. I'd like to transition parts of the Kustomize managed files to Kpt over time

[mikebz]

@viglesiasce there is a way to introduce kpt to kustomize users that is covered by this example: https://github.com/GoogleContainerTools/kpt/tree/main/package-examples/kustomize
also could be great to get your feedback on the FAQ: https://kpt.dev/faq/?id=whats-the-difference-between-kpt-and-kustomize