add GEP-1323: Response Header Filter #1326

aryan9600 · 2022-08-15T15:47:29Z

Signed-off-by: Sanskar Jaiswal [email protected]

What type of PR is this?
/kind gep

What this PR does / why we need it:
Adds a GEP proposing a way of modifying response headers via a new field responseHeaderModifier.

Which issue(s) this PR fixes:

Fixes #1323

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2022-08-15T15:47:37Z

Hi @aryan9600. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

shaneutt

I think it's a good proposal. I particularly like the usage examples, thanks for taking the time to write all this out.

site-src/geps/gep-1323.md

robscott

Thanks @aryan9600!

site-src/geps/gep-1323.md

candita · 2022-08-15T23:47:18Z

site-src/geps/gep-1323.md

+Adding support for this unlocks a lot of real world use cases. Let’s review a couple of them:
+
+* For example, if a team has a frontend web app that along with two different versions of their backends exposed as Kubernetes services, and the frontend needs to know which backend it’s talking to, this can be easily achieved without any modifications to the application code.
+


Could you describe how this might work if the backend application needs to deliver a custom value in the response? E.g. the backend destination determines a cookie value?

I'm sorry, I don't understand fully. If the backend application needs to deliver a custom cookie that it determined using some business logic, it'd need to set it in while returning the response itself. If I'm not answering your question clearly here, could you kindly elaborate a bit more? thanks

It sounds like maybe you're describing something like the backend app setting a cookie, and then the Gateway implementation needing to do something with that value, @candita?

If that is correct, I know some proxies support setting a header value using the value from another, but I don't feel like that's common enough to be included in the API.

@aryan9600 for example, this HTTPRoute has two hostnames, and the header response depends on the value of the hostname that delivered this traffic. If response1.header.example was used, return header value 1, but if response2.header.example was used, return header value 2. The backend gets to decide what value is sent back in the response header. Can this be set up to hold a variable name in the value of the add stanza?

Suggested change

filters:

- type: ResponseHeaderModifier

responseHeaderModifier:

add:

name: number1or2

value: $someVariableSetAtTheBackEnd

I guess I'm aware of more response headers that are set after arriving at the backend, and based on what the application determines. This proposal is based more on a hard-coded response header value, which is less challenging, but perhaps also less useful.

okay, so from what I understand, you're suggesting parsing with logical conditioning? this is a very cool feature, but i doubt if it's generic enough to be included into the API. could you please point me to a couple of service meshes/ingresses that have support for something similar today?

Would setting this proposed ResponseHeaderModifier filter with different values on two separate HTTPBackendRefs be sufficient to enable some of the desired functionality here? The configuration would be static still and a bit more verbose than specifying a filter with a dynamic value at the HTTPRouteRule level, but maybe enough for some needs, such as determining if a response was served by a canary backend?

For the hostname matching specifically, I suppose two separate HTTPRouteRules (identical except for the hostnames field) could be configured as one way to implement that instead of relying on a value from the backend - that feels like it could be a use case that could be simplified with config templating, which may be best achieved by third-party tools at a layer above Gateway API.

Finally, for header manipulation (add or modify a header based on an existing header value in the response, which feels like the most direct way to implement this without touching the backend app itself), it feels like we might have some of the scaffolding to build from by combining type: RegularExpression from HTTPHeaderMatch and the add/set fields in HTTPRequestHeaderFilter if we were to extend the RegEx implementation from just matching to allow capture groups for setting a value, but then you're already in Custom conformance due to the variance in RegEx implementations, so I'm not sure how much value there is to spec that, although HTTPPathModifier could possibly benefit from a similar implementation. This last option feels beyond the scope of this current proposal (it could build on top of it though!) but may be worth exploring if there's sufficient interest, or first experiment by building something as an implementation-specific filter using the extensionRef field of HTTPRouteFilter?

I think there was some confusion on this thread, but after discussing in more detail at today's community meeting, it looks like Envoy provides a good example of this kind of capability. Essentially allowing users to intersperse variables in the responses using % special chars: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#custom-request-response-headers.

Nothing in our API for either request, or in this case response, header modification prevents that from working, but we should likely add a note to the spec that while this kind of functionality may be supported by the underlying implementation, it is unlikely to be portable.

+1 that we should have something in the implemented spec that says we're not excluding metacharacter use, but with a warning about portability. I think that can be left as an exercise for the implementer though.

candita · 2022-08-15T23:59:06Z

site-src/geps/gep-1323.md

+
+## Goals
+* Provide a way to modify HTTP response headers.
+* Reuse existing types as much as possible to reduce boilerplate code.


What if the admin wanted to add the same response header to all HTTPRoutes (e.g. the same HSTS response for all of them) - can you find a way for the admin to override response headers at a higher level?

I don't see how that's possible with the current structure of the API. Adding a response header to all HTTPRoutes is possible if the Gateway object had a way of modifying the responses that are handled by HTTPRoutes attached to it. From what I understand going through https://gateway-api.sigs.k8s.io/concepts/api-overview/#gateway, a Gateway is not responsible for actually modifying or forwarding requests/responses.
If we want to introduce this behavior, I'd guess that it's out of scope here and needs a GEP of it's own. cc: @robscott @shaneutt

That use case (admin wanting to force the addition of something) is one of the things we had in mind with Policy Attachment - I could see a Policy Resource that replicated the same structure used here in an override block, that could be attached to a Gateway resource. That would imply that these settings should Override any settings on the HTTPRoutes.

Yes, I agree, the Policy Attachment API is a good place for something like this. But I'm not sure if it falls under the scope of this proposal. If it does, then please let me know and I'll add a section about it. cc: @shaneutt @robscott

@aryan9600 if we're going to go straight to implementable with this, then I think adding a section for this and pointing to policy attachment sounds correct.

@youngnick added a section on Policy Attachment, please take a look, thanks.

Having a policy that overrides a filter creates an implicit policy for a filter in an HTTPRoute :) . It also means we now have two different ways to configure the same thing - through a policy and through a filter. I wonder if it is a good thing.

Perhaps another way is to have a policy that can override all filters rather than a specific one? So it is more general-purpose.

@pleshakov yes we have two different ways to configure the same thing, but they are meant to be used in a different context. The example in the Policy Attachment section describes a real world use case that highlights this.
I'm definitely okay with having a multi-purpose policy instead of this but please keep in mind that this is just a suggestion. As far as I understand, implementations are free to design their policies as they chose and like. In https://gateway-api.sigs.k8s.io/references/policy-attachment/, I can see policies like TimeoutPolicy and RetryPolicy being used throughout, although they are technically each a filter.

After discussing this at today's community meeting, it seems like we're all OK to leave policy attachment out of the scope of this GEP. As long as the GEP mentions that this may be further expanded by policy attachment in the future, I don't think we need to get into the details of what that would look like here.

youngnick

This is looking pretty great to me, just a few small things.

youngnick · 2022-08-18T05:07:36Z

site-src/geps/gep-1323.md

+
+## Goals
+* Provide a way to modify HTTP response headers.
+* Reuse existing types as much as possible to reduce boilerplate code.


That use case (admin wanting to force the addition of something) is one of the things we had in mind with Policy Attachment - I could see a Policy Resource that replicated the same structure used here in an override block, that could be attached to a Gateway resource. That would imply that these settings should Override any settings on the HTTPRoutes.

youngnick · 2022-08-18T05:13:54Z

site-src/geps/gep-1323.md

+Adding support for this unlocks a lot of real world use cases. Let’s review a couple of them:
+
+* For example, if a team has a frontend web app that along with two different versions of their backends exposed as Kubernetes services, and the frontend needs to know which backend it’s talking to, this can be easily achieved without any modifications to the application code.
+


It sounds like maybe you're describing something like the backend app setting a cookie, and then the Gateway implementation needing to do something with that value, @candita?

If that is correct, I know some proxies support setting a header value using the value from another, but I don't feel like that's common enough to be included in the API.

youngnick · 2022-08-18T05:22:08Z

site-src/geps/gep-1323.md

+}
+```
+
+Given the fact that this functionality is offered by only a few projects that currently implement Gateway API when using their own traffic routing CRDs, it’s better to support `ResponseHeaderModifier` as an _Extended_ feature, unlike `RequestHeaderModifier` which is a _Core_ feature. This will also not increase the difficulty of implementing Gateway API for any future ingress or service mesh.


I like this, we can use the same HTTPHeaderModifier struct to represent the actual config, but using the field that the struct appears in inside the Route can have different support, and that support can be documented on those fields.

Not sure if we need that in this GEP or not though.

Do you mean we could add a line in the HTTPRouteFilter API above the ResponseHeaderModifier field mentioning the support similar to this?

gateway-api/apis/v1beta1/httproute_types.go

Line 571 in bb549d0

// Support: Core

If yes, I didn't mention it in the proposal, because I thought it's implied that the support will be documented in the API docs. But if you think, we need to be explicit about it in the proposal, I'll add something accordingly.

Yes, I think that when this is implemented, any new fields should have their support explicitly documented.

candita · 2022-08-24T22:20:15Z

site-src/geps/gep-1323.md

+
+    // Default defines default policy configuration for the targeted resource.
+    // +optional
+    Default *HTTPHeaderModifier `json:"default,omitempty"`


I don't think Default is needed here. If so, how is the Default different than the Override? What happens if both are specified?

I think the example below describes the usage of Default and Override sufficiently. About both being specified, I don't think that makes any sense, so maybe I should add something like only one of Default and Override should be present in a Policy object?

https://gateway-api.sigs.k8s.io/references/policy-attachment/#policy-boilerplate does explicitly describe allowing both Default and Override to be set ("at least one"), although I think that often makes more sense when they are used for aggregate config objects and modify different values, such as the AcmeServicePolicy example in https://gateway-api.sigs.k8s.io/references/policy-attachment/#hierarchy

For this case, it could make sense to add a few headers by default, and only enforce adding one important one with an override.

candita · 2022-08-24T22:36:55Z

site-src/geps/gep-1323.md

+## Policy Attachment
+
+This feature can be further extended via [Policy Attachment](../references/policy-attachment.md). Implementations can introduce a new Policy, `ResponseHeaderPolicy`, which can be used to ensure that certain headers are present/absent in a response, regardless of the `responseHeaderModifier` field in the `HTTPRoute` associated with that response.
+


Maybe you don't need to hash all of this out in this GEP, and just a reference to future work could be included.

I don't understand what you mean by "a reference to future work"? Do you mean state something like "This API can be extended with Policy Attachment, which will be discussed in a future GEP"?

candita · 2022-08-24T22:39:31Z

site-src/geps/gep-1323.md

+
+`.spec.default` provides a way to modify headers of responses being returned from the resource specified in `.spec.targetRef`. This resource can be a part of the Gateway API like `Gateway` or a core Kubernetes resource like `Namespace`. Using a `Namespace` as the `.spec.targetRef` would result in any response being returned from any resource in that namespace having its headers modified according to the policy.
+`.spec.override` can be used to modify the headers of responses being returned from a resource that is already covered by `.spec.default` of a `ResponseHeaderPolicy`. Let's walk through an example:
+


This is a good example. If the community decides to keep these details within this GEP, then I'll offer a suggestion for a different way to implement a global policy with the possibility of override. The policy is set at the Gateway to apply to all *Routes globally as you described, but then each *Route could override the global default if needed, using the ResponseHeaderModifier mechanism you described earlier. The policy would then need to include some way to describe whether it is optional or not, i.e. whether it can be overridden or not. To me, this approach is clearer, but I still think it could be discussed in a different GEP.

If we allow a Route to override the override set by a Gateway using its ResponseHeaderModifier, how would admins set headers alongside the headers specified in the Route's ResponseHeaderModifier?
About, moving the Policy Attachment section to its own GEP, I'm neither for or against it, so some consensus from the maintainers on this would be helpful.
wdyt? @shaneutt @robscott @youngnick

I like this proposed approach and think it's sufficiently clear. Policy Attachment already has the structure for separating defaults and overrides - if a header was added or modified by an Default config a filter on the Route object should be able to further modify or remove it, but if added or modified by an Override config, a filter on the Route should not be able to modify or remove it. @candita does this fully address your proposed use case?

The main reason this section may warrant consideration as a separate GEP is that it contradicts the current warning in https://gateway-api.sigs.k8s.io/references/policy-attachment/#interaction-with-custom-route-filters against [using] multiple mechanisms for representing the same fields, although that does explicitly state:

Note that this guidance may change in the future as we gain a better understanding of how extension mechanisms of the Gateway API can interoperate.

@mikemorris having gone through the text, I understand that we disallow overlaps between Policy and custom Route filters. Since this proposed filter is not custom I feel like the current proposal is more or less okay.
I'll update it with your suggestion about Routes being able to modify Default config. @candita I now realize that I misread your initial comment 🤦 so yes, a Route should be able to modify the gloabal default

pleshakov · 2022-08-24T23:48:44Z

site-src/geps/gep-1323.md

+
+## Goals
+* Provide a way to modify HTTP response headers.
+* Reuse existing types as much as possible to reduce boilerplate code.


Having a policy that overrides a filter creates an implicit policy for a filter in an HTTPRoute :) . It also means we now have two different ways to configure the same thing - through a policy and through a filter. I wonder if it is a good thing.

Perhaps another way is to have a policy that can override all filters rather than a specific one? So it is more general-purpose.

pleshakov · 2022-08-24T23:52:46Z

site-src/geps/gep-1323.md

+        # set cookies for all requests being forwarded to this service
+        responseHeaderModifier:
+          set:
+            name: Set-Cookie


According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie, "To send multiple cookies, multiple Set-Cookie headers should be sent in the same response." Will it be possible to set multiple headers with the same name?

You can use add instead of set:

gateway-api/apis/v1beta1/httproute_types.go

Line 730 in 30f2983

Add []HTTPHeader `json:"add,omitempty"`

I think the validation will prevent it though - it will reject any attempt to set or add two headers with the same name

// +listType=map // +listMapKey=name

at the same time, it is probably will be very uncommon requirement

i'm not 100% sure, but i think that'll happen when you have the same header name (the key) present in both set and add. This conclusion of mine stems from https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules

map: X + Y performs a merge where the array positions of all keys in X are preserved but the values are overwritten by values in Y when the key sets of X and Y intersect. Elements in Y with non-intersecting keys are appended, retaining their partial order.

pleshakov · 2022-08-24T23:56:51Z

site-src/geps/gep-1323.md

+    // +listType=map
+    // +listMapKey=name
+    // +kubebuilder:validation:MaxItems=16
+    Add []HTTPHeader `json:"add,omitempty"`


does it make sense to always add headers? For example, what if the backend returns 500 response or the backend doesn't exist so that the proxy generates 500.

For example, in NGINX, it is possible to add headers for a subset of 2xx and 3xx codes only.

@pleshakov I think what you described is more like the "conditions" for running the filter. I think it can be implemented as a common option for all filters, i.e. check out the condition before running the filter.

@pleshakov you seem to be suggesting similar to #1326 (comment)
i agree with @tokers that this could be made more generic. furthermore, i believe it exists outside the scope of this GEP, as deciding how to conditional logic on filters is a GEP of it's own.

tokers · 2022-08-25T01:09:30Z

site-src/geps/gep-1323.md

+* Status: Implementable
+
+## TLDR
+Similar to how we have `RequestHeaderModifier` in `HTTPRouteFilter`, which lets users modify request headers before the request is forwarded to a backend (or a group of backends), it’d be helpful to have a `ResponseHeaderModifier` field which would let users modify response headers before they are forwarded to its destination.


Suggested change

Similar to how we have `RequestHeaderModifier` in `HTTPRouteFilter`, which lets users modify request headers before the request is forwarded to a backend (or a group of backends), it’d be helpful to have a `ResponseHeaderModifier` field which would let users modify response headers before they are forwarded to its destination.

Similar to how we have `RequestHeaderModifier` in `HTTPRouteFilter`, which lets users modify request headers before the request is forwarded to a backend (or a group of backends), it’d be helpful to have a `ResponseHeaderModifier` field which would let users modify response headers before they are returned to the client.

tokers · 2022-08-25T01:12:40Z

site-src/geps/gep-1323.md

+    // +listType=map
+    // +listMapKey=name
+    // +kubebuilder:validation:MaxItems=16
+    Add []HTTPHeader `json:"add,omitempty"`


@pleshakov I think what you described is more like the "conditions" for running the filter. I think it can be implemented as a common option for all filters, i.e. check out the condition before running the filter.

robscott · 2022-08-30T01:20:06Z

Thanks for all the work on this @aryan9600! We talked about this at today's community meeting. I left some follow up comments above, but I think the major themes are:

This GEP feels very close to ready to merge.
When it comes to variables in header names, we should mention that some underlying implementations like Envoy support interpolating variables into header values, but that is unlikely to be portable across implementations.
When it comes to policy attachment, we should mention it as a possible future extension point, but we don't need to cover it in detail in this GEP.

/ok-to-test

robscott · 2022-08-30T05:49:47Z

Thanks @aryan9600! This LGTM but will defer to someone else for the final approval.

/approve

k8s-ci-robot · 2022-08-30T05:50:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aryan9600, robscott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [robscott]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Sanskar Jaiswal <[email protected]>

youngnick · 2022-09-01T03:14:34Z

Yup, I think this is good to go.

/lgtm

k8s-ci-robot added kind/gep PRs related to Gateway Enhancement Proposal(GEP) cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 15, 2022

aryan9600 marked this pull request as draft August 15, 2022 15:47

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 15, 2022

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 15, 2022

k8s-ci-robot requested review from bowei and keithmattix August 15, 2022 15:47

shaneutt reviewed Aug 15, 2022

View reviewed changes

site-src/geps/gep-1323.md Outdated Show resolved Hide resolved

site-src/geps/gep-1323.md Show resolved Hide resolved

robscott reviewed Aug 15, 2022

View reviewed changes

site-src/geps/gep-1323.md Show resolved Hide resolved

site-src/geps/gep-1323.md Outdated Show resolved Hide resolved

site-src/geps/gep-1323.md Outdated Show resolved Hide resolved

site-src/geps/gep-1323.md Outdated Show resolved Hide resolved

candita reviewed Aug 15, 2022

View reviewed changes

aryan9600 requested review from robscott and shaneutt August 16, 2022 19:07

aryan9600 force-pushed the gep-1323 branch from 1ab7d92 to f5a7dbc Compare August 17, 2022 13:55

youngnick reviewed Aug 18, 2022

View reviewed changes

aryan9600 force-pushed the gep-1323 branch from 1b9aa31 to 0b3337e Compare August 23, 2022 06:48

aryan9600 marked this pull request as ready for review August 24, 2022 19:02

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2022

k8s-ci-robot requested a review from mikemorris August 24, 2022 19:02

candita reviewed Aug 24, 2022

View reviewed changes

pleshakov reviewed Aug 25, 2022

View reviewed changes

tokers reviewed Aug 25, 2022

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 30, 2022

aryan9600 force-pushed the gep-1323 branch from 0b3337e to 606dcb1 Compare August 30, 2022 04:57

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 30, 2022

add GEP-1323: Response Header Filter

2240a3d

Signed-off-by: Sanskar Jaiswal <[email protected]>

aryan9600 force-pushed the gep-1323 branch from 606dcb1 to 2240a3d Compare August 30, 2022 05:53

k8s-ci-robot assigned youngnick Sep 1, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2022

k8s-ci-robot merged commit fb07abb into kubernetes-sigs:main Sep 1, 2022

aryan9600 mentioned this pull request Sep 7, 2022

REQUEST: New membership for aryan9600 kubernetes/org#3664

Closed

9 tasks

candita mentioned this pull request Sep 22, 2023

REQUEST: New membership for candita kubernetes/org#4477

Closed

9 tasks

		Adding support for this unlocks a lot of real world use cases. Let’s review a couple of them:

		* For example, if a team has a frontend web app that along with two different versions of their backends exposed as Kubernetes services, and the frontend needs to know which backend it’s talking to, this can be easily achieved without any modifications to the application code.

+filters:
+   - type: ResponseHeaderModifier
+       responseHeaderModifier:
+         add:
+             name: number1or2
+             value: $someVariableSetAtTheBackEnd

		## Policy Attachment

		This feature can be further extended via [Policy Attachment](../references/policy-attachment.md). Implementations can introduce a new Policy, `ResponseHeaderPolicy`, which can be used to ensure that certain headers are present/absent in a response, regardless of the `responseHeaderModifier` field in the `HTTPRoute` associated with that response.


		`.spec.default` provides a way to modify headers of responses being returned from the resource specified in `.spec.targetRef`. This resource can be a part of the Gateway API like `Gateway` or a core Kubernetes resource like `Namespace`. Using a `Namespace` as the `.spec.targetRef` would result in any response being returned from any resource in that namespace having its headers modified according to the policy.
		`.spec.override` can be used to modify the headers of responses being returned from a resource that is already covered by `.spec.default` of a `ResponseHeaderPolicy`. Let's walk through an example:

	Similar to how we have `RequestHeaderModifier` in `HTTPRouteFilter`, which lets users modify request headers before the request is forwarded to a backend (or a group of backends), it’d be helpful to have a `ResponseHeaderModifier` field which would let users modify response headers before they are forwarded to its destination.
	Similar to how we have `RequestHeaderModifier` in `HTTPRouteFilter`, which lets users modify request headers before the request is forwarded to a backend (or a group of backends), it’d be helpful to have a `ResponseHeaderModifier` field which would let users modify response headers before they are returned to the client.

add GEP-1323: Response Header Filter #1326

add GEP-1323: Response Header Filter #1326

Conversation

aryan9600 commented Aug 15, 2022

k8s-ci-robot commented Aug 15, 2022

shaneutt left a comment

Choose a reason for hiding this comment

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

candita Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

aryan9600 Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

mikemorris Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aryan9600 Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youngnick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aryan9600 Aug 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aryan9600 Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

mikemorris Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikemorris Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robscott commented Aug 30, 2022

robscott commented Aug 30, 2022

k8s-ci-robot commented Aug 30, 2022

youngnick commented Sep 1, 2022

candita Aug 24, 2022 •

edited

Loading

aryan9600 Aug 25, 2022 •

edited

Loading

mikemorris Aug 25, 2022 •

edited

Loading

aryan9600 Aug 16, 2022 •

edited

Loading

aryan9600 Aug 20, 2022 •

edited

Loading

aryan9600 Aug 25, 2022 •

edited

Loading

mikemorris Aug 25, 2022 •

edited

Loading

mikemorris Aug 25, 2022 •

edited

Loading