Add GEP-1364 proposal

[youngnick]

What type of PR is this?
/kind gep

What this PR does / why we need it:
This PR adds an implementable proposal for GEP1364 (#1364).

It's a big GEP, for a very big change, that will definitely break conformance tests. But I think that it will set us up with clear rules for Conditions going forward.

It's quite opinionated, so I expect to need to defend my decisions here somewhat, don't be shy about objections.

Which issue(s) this PR fixes:
Updates #1364

Does this PR introduce a user-facing change?:

Status definitions have been updated across the API.

The release note will need something like the tl;dr in the GEP, once we all agree. 😄

[howardjohn]

Definitely a good direction

[howardjohn]

Currently the API supports partially-valid resources. I believe this is proposing a partially-valid resource would be Attached=false, but the config is actually 'active' (or, I would describe it, "attached").

To me this feels a bit odd. The old API sort of had this issue as well.

Thinking purely from a pedantic-correctness POV (which may not be ideal - sometimes simpler is better), I would expect there to be two conditions (I use bad names intentionally to avoid confusion): "ThisResourceWasApplied" - any part of the rule had some effect on the system, and "ThisRuleIsEntirelyValid" - no errors found in the rule at all.

Its possible my concern would be addressed with a different word than "Attached", rather than more conditions, as well

WDYT?

[howardjohn]

I guess this is somewhat addressed below.. but I am not sure its accurate? It seems we are defining a very small set of "failures that do not become unattached", but there are a lot more partially-accepted resources I think?

[youngnick]

The guideline I've been basing the "does something become unattached?" on is "Does this produce at least some configuration in the underlying data plane?". So I guess it sounds like I need to be more explicit with that guideline, but that aside from that, we're largely in agreement?

[howardjohn]

I think it makes sense to have "Is this attached" and "is this 100% valid" as two separate things.

Part of my confusion, I think, is I am not actually sure which fields result in "not attached". It seems most issues are partially accepted. Which is fine - Attached just becomes basically a pretty high level acknowledgement that the controller is handling the resource

[keithmattix]

Yeah I think I agree with @howardjohn's point here: "does this resource produce config" is distinct from "does this resource produce valid config". I'm sure some implementations don't allow invalid (e.g. duplicate route names or something) to be sent to the data plane, but it doesn't feel like that's a requirement for the Gateway API at this juncture.

[youngnick]

I guess I wrote this pooly - by "Does this resource produce config" I meant to read "Is this resource valid enough to produce some config when combined with the rest of the resource set".

The thing I'm thinking of particularly here is that for HTTPRoute, invalid references still produce data path config because the match will respond with 500s. Whereas for TCPRoute etc, if you don't have a service to route to, there's no way you can build a forwarding path - there's nothing to forward to! So that won't produce any data path config at all, and so, in the current design, would cause the resource to fail to be attached.

I don't think anything should be allowed to produce invalid config in the underlying data plane. If so, that should mean that the resource becomes unattached.

I guess this means that we need some more clarity about what attached means. I've been assuming that if we consider a resource to be "attached" to another, that:

• the resource is valid in itself
• the resource will produce at least some valid config in the underlying data plane. (note that "return a 500 for this route" is valid config).

And so, that having a "valid" condition is not really useful because if something is not valid, it won't be attached either.

[evankanderson]

Is it possible for something like an HTTPRoute to define two paths, one of which is "valid enough" config, while the other is garbage?

If so, are there expectations or requirements from a conformance PoV ("always apply all or none") that we need to specify?

[sunjayBhatia]

Is it possible for something like an HTTPRoute to define two paths, one of which is "valid enough" config, while the other is garbage?

Most cases of an invalid configuration in one route rule (with others being completely valid within the same HTTPRoute) seem to be misconfigurable only in the context of a reference to another resource and fall into the ResolvedRefs bucket (which seems stated below and possibly agreed upon that Attached: true be set with ResolvedRefs: false and return HTTP code 500 for the invalid rule).

One such example not covered is if you were to use a HTTPRequestRedirectFilter and HTTPURLRewrite in the list of filters for an individual HTTPRouteRule (ref). At the moment it doesn't appear we have prescribed any status code or other error handling for this error. This seems to fall into the bucket of things that cannot produce data plane configuration.

Going by the statements below, if this rule were the only one present Attached: false should be set, since it cannot result in any configuration (with another more specific error condition). If present with other valid rules, Attached: true should be set (with another more specific error condition again). I can definitely see the argument for setting Attached: false however in the latter case, since stated above, the Attached condition means The resource is syntatically and semantically valid, and internally consistent.

It seems that the Valid: true condition mentioned above ends up being somewhat equivalent to having for example for HTTPRoute ResolvedRefs: true and no error conditions (and vice versa, Valid: false equivalent to ResolvedRefs: false or some more specific error condition). Maybe this framing helps a little with determining possible merits of having another condition?

If so, are there expectations or requirements from a conformance PoV ("always apply all or none") that we need to specify?

Just doing a quick pass, I don't think I saw any existing conformance tests that cover this kind of thing so I agree that would be good to add.

[youngnick]

Avoiding "always apply all or none" is the reason this is tricky. There are mixed expectations about what happens if some config is present but bad, and I'm attempting to thread the needle here and make this as user-friendly as possible while not becoming really difficult to implement or understand.

[youngnick]

I added a section on partial validity to hopefully make this clearer.

[howardjohn]

This probably won't work for cloud providers (at least GKE is slower, not sure about others).

I am also not sure it provides value. What do we lose if we don't define anything here?

If we consider something like Knative, they certainly aren't going to be able to just wait X seconds; I'd expect the same with other users

[howardjohn]

And if we can't provide any guarantees about Readiness, then it seems like Ready becomes no different than Attached...

[youngnick]

We certainly can mandate Ready to be an extended condition, and say "If you supply a Ready condition, it has to work like this". Maybe doing that is a better place to start with Ready?

[howardjohn]

Do we have a feel for which implementations can and cannot support Ready properly?

[robscott]

Agree that cloud providers will likely have trouble implementing a Ready condition. Could we introduce a Reconciled condition that would be more broadly implementable?

[bowei]

+1 any distributed (e.g. geo distributed) implementation has a hard time with this (think: config was pushed to 95% of the sites around the world, is it ready?)

[keithmattix]

Missing href for the link here?

[youngnick]

It's all the way down at the bottom of the doc, with the other references. I thought it was better to have everything in the one place.

[keithmattix]

Suggested change

	are `type`, `status`, `season`, and `observedGeneration`.
	are `type`, `status`, `reason`, and `observedGeneration`.

[keithmattix]

Yeah I think I agree with @howardjohn's point here: "does this resource produce config" is distinct from "does this resource produce valid config". I'm sure some implementations don't allow invalid (e.g. duplicate route names or something) to be sent to the data plane, but it doesn't feel like that's a requirement for the Gateway API at this juncture.

[keithmattix]

This starts painting an observability problem IMO. Even if conditions are only meant to be human-readable, how does a human look at an HTTPRoute resource and know "oops I have a typo in my service backend"? Yes, 500s indicate something's wrong, but I would expect to find a condition pointing to the specific issue

[bowei]

We should allow room for error conditions in addition to attached in this case so the intformation can be conveyed.

[youngnick]

Yes, I guess I should make this clearer, that in this case, it's absolutely expected that there be some additional, negative-polarity error conditions that tell you that the backendRef doesn't exist. (This is a distinct case from the backend existing, but having zero endpoints, which would not generate a "backendRef doesn't exist" error, but would end up with the same behavior).

[evankanderson]

You might also get a BackendRefEmpty condition...

[robscott]

Thanks for all the work on this @youngnick! This is a great start

[robscott]

Would Attaching ever be valid as a condition? Maybe just verb or adjective is sufficient here?

[youngnick]

That sort of usage is specifically called out as not desirable in the API conventions, and I agree. Conditions are supposed to be level-triggered, not edge-triggered, so having a condition that says that something is happening is a bit weird. (Should Attaching transition to false once the attaching process is complete?)

[shaneutt]

For posterity, can we cite some sources for that assertion? It would probably actually be good as a part of this status effort to document somewhere in the code that we're avoiding this due to conventions, and then point to those conventions, as this will help future contributors to more quickly avoid pursuing such things (or give them a path to who they need to convince in the greater kubernetes community).

[youngnick]

This is almost a direct quote from the API conventions as it is, should I put a link in here to the exact line?

[youngnick]

The exact quote is as follows, I can just inline if it's better, but I was trying to avoid just pasting in the entire section:

---

Condition type names should describe the current observed state of the resource, rather than describing the current state transitions. This typically means that the name should be an adjective ("Ready", "OutOfDisk") or a past-tense verb ("Succeeded", "Failed") rather than a present-tense verb ("Deploying"). Intermediate states may be indicated by setting the status of the condition to Unknown.

For state transitions which take a long period of time (e.g. more than 1 minute), it is reasonable to treat the transition itself as an observed state. In these cases, the Condition (such as "Resizing") itself should not be transient, and should instead be signalled using the True/False/Unknown pattern. This allows other observers to determine the last update from the controller, whether successful or failed. In cases where the state transition is unable to complete and continued reconciliation is not feasible, the Reason and Message should be used to indicate that the transition failed.

---

I don't believe that we generally have any process that we want to handle in the way called out in the exception bullet point. Happy to be guided here on what you'd like to see.

[shaneutt]

I think a direct link to the language would be nice, but not a blocker.

[robscott]

My own personal preference would be for conditions with a positive polarity to represent the normal healthy state and always be present, and for conditions with a negative polarity to represent very specific error states and to only be present when true. I think we should do everything we can to avoid the presence of double negatives like "Detached == False", but recognize that there are some error states that are best represented with a negative condition. I just don't think those negative conditions should be present when false.

[bowei]

+1. As I mentioned above -- as long we document and check for it in conformance, this is the only guarantee that the conditions are meaningful programmatically, which is basically what we are looking for when we build for consistency.

[youngnick]

If we're going to have a mixed-polarity set of conditions, then I think we need to define:

• which Conditions are positive polarity (Probably Attached, Programmed, Ready, and Valid if we do it)
• which Conditions are positive polarity and should always be present (Probably Attached, Programmed, and Valid if we do it).
• which Conditions are core (currently, everything other than Ready)
• which Conditions are extended (only Ready).
• which Conditions are negative (all the error conditions).

[robscott]

Agree that cloud providers will likely have trouble implementing a Ready condition. Could we introduce a Reconciled condition that would be more broadly implementable?

[robscott]

I think I'd rather reserve Ready for "Ready at this moment" and instead use a different term to represent what we're actually describing - this config has been written to the data plane but we're not sure how long that will take to propagate. Previously Reconciled has been proposed to communicate this.

[youngnick]

Okay, I'll update this section to:

• add Ready as an Extended support condition, that, if present, indicates that the data plane is fully configured. But because it's not always present, we'll need Reconciled to indicate what we have here - that everything is okay and the data plane will be ready soon.

[bowei]

I think we should state that the conditions covered in the core API must be present -- but there are likely other conditions (e.g. implementation specific), that are optional.

[youngnick]

So, does this mean that all conditions that have Core conformance should always be present? Or does this mean that a set of positive-polarity summary endpoints we name should always be present?

The former could include any negative-polarity conditions we define as having Core conformance (which would clash with the "don't have double negatives" guidance we're looking for in other places.)

[youngnick]

I've guessed "summary conditions" here - meaning just the positive polarity ones.

[bowei]

+1. As I mentioned above -- as long we document and check for it in conformance, this is the only guarantee that the conditions are meaningful programmatically, which is basically what we are looking for when we build for consistency.

[bowei]

I don't know if this is necessarily true -- it just needs to be explicitly documented for and tested. I think the previous attempt by K8s to force one polarity just simply didn't work out because it didn't line up with people's intuition. We should really try to keep things intuitive as much as possible.

[youngnick]

It means that anything consuming the Conditions will need to have a hard-coded list of the positive polarity conditions. Not a deal breaker, but it's another obstacle in the way of building a generic condition-parsing library.

[bowei]

We should allow room for error conditions in addition to attached in this case so the intformation can be conveyed.

[bowei]

+1 any distributed (e.g. geo distributed) implementation has a hard time with this (think: config was pushed to 95% of the sites around the world, is it ready?)

[youngnick]

Okay, to summarize the comments around condition naming and polarity:

We're going to end up with four possible positive polarity summary conditions:

• ValidSpec or similar; there aren't any issues in the config of this resource on its own
• Attached; there aren't any issues with the way this resources attaches to others
• Programmed or similar; the resource has been processed and the data plane is in the process of being configured. It should be ready "soon", where "soon" is implementation-specific.
• Ready; an optional, Extended condition that is only added when the implementation can guarantee that all required configuration has been accepted by the data plane and is running. Because it's Extended, it will have conformance tests to guarantee this behavior.

Other error conditions may be added that are negative polarity - the only thing relevant about these Conditions is if they are status: true. If they are not status: true, they should not be present.

Three or four summary, positive-polarity conditions seems on the high side to me, but if everyone feels that the additional clarity is worth the overhead of managing more Conditions.

I'm kind of inclined to say that anything we would use a ValidSpec summary condition for should probably be in the validation rules - whether they are in the OpenAPI definitions or the validating webhook doesn't really matter. I don't think the cases in which ValidSpec is True and Attached is false are going to be very common, so I don't really see the utility of that condition as much, personally. But if we really want it, we can do it.

[howardjohn]

ValidSpec can have things not feasible to check in a webhook like temporary dependency issues (referenced service doesn't exist) or implementation specific issues (parent has FooGateway, which doesn't support the Bar field), so I think it holds weight.

If we do want to collapse we could have a single Valid field with reasons like NotValid, NotAttached, etc (naming needs work) possibly...?

Also re-iterating my comment on #1383 (comment) since I think I wasn't quite clear - I think making a Ready optional condition is actually harmful. It seems like the type of field only useful if its required, and I am not sure how feasible it is to require it.

[youngnick]

I guess in my mind, ValidSpec is always a requirement for Attached, so there's less added value in having the two bits of information. For me, in the cases you give:

• Service doesn't exist would cause a resource either to program 500s into the data plane and add a more specific condition for HTTPRoute (the already existing ResolvedRefs), or cause another Route type - that doesn't have an error code like HTTP - to become unattached.
• The FooGateway not supporting the Bar field would cause the HTTPRoute to fail to attach, and so Attached would be false anyway, with Reason and Message to indicate the reason.

To put this another way, for me, ValidSpec is always a requirement for Attached, so the marginal utility of having both seems low. Invalid objects can't attach, and the reason they can't is that they're invalid.

Is it just the naming that's the problem, or is it the pivoting the status around the idea of attachment?

[howardjohn]

I guess in my mind, ValidSpec is always a requirement for Attached, so there's less added value in having the two bits of information. For me, in the cases you give:

• Service doesn't exist would cause a resource either to program 500s into the data plane and add a more specific condition for HTTPRoute (the already existing ResolvedRefs), or cause another Route type - that doesn't have an error code like HTTP - to become unattached.

I think I am confused. I thought ValidSpec meant it was 100% valid. So we would make ValidSpec=false if there is a unknown service. I thought the proposal was to have only the conditions listed above, so how would ResolvedRefs be used?

• The FooGateway not supporting the Bar field would cause the HTTPRoute to fail to attach, and so Attached would be false anyway, with Reason and Message to indicate the reason.

Maybe.. unless the field is "fail open" and we just ignore it...? I am not sure we have a clear spec here on unsupported fields. It would be a bummer to add a filter and then suddenly have an outage because it wasn't supported.

[youngnick]

I think I am confused. I thought ValidSpec meant it was 100% valid. So we would make ValidSpec=false if there is a unknown service. I thought the proposal was to have only the conditions listed above, so how would ResolvedRefs be used?

Ah, I guess I've been including ResolvedRefs in the bucket of "specific error conditions" that we haven't specified in this GEP. If I've implied that only those conditions listed are to be used, I've made a mistake and need to update.

[youngnick]

To make my earlier response clearer (and I'm working on updating the text to include something like this), the idea behind Attached is that it's a summary condition that indicates whether or not there is a severe error that will prohibit any config at all from ending up in the data plane.

In the case you're talking about using Valid for, the intention is to have specific error conditions that will describe the error, and only be present when they are True. If there is an unknown Service in a backendRef, then the ResolvedRefs condition must be False (I'm leaving ResolvedRefs with its current polarity). That may also mean that the object becomes unattached - for resources that don't have a thing like sending a 500 error response like HTTPRoute. Basically, for HTTPRoute, because we can program an error response into the data plane, the set of things that will cause an unttachment is actually very small.

[costinm]

I think it is reasonable for the controller to do this - as an indication the resource is in use by the controller.

It becomes problematic if a resource may be processed by multiple controllers - for example the discussion
of HttpRoute applying to both mesh and Gateway can only work if the type has a prefix (MeshReady).

In general it would be good to indicate how to handle resources where multiple controllers operate on. While
it's good to know which controller mess with the resource - maybe the controller should skip this step if the
resource is not ignored ( a mesh controller will ignore any route that don't target both mesh and gateway as example)

[youngnick]

In most cases, the Conditions slice is in a separate struct that's part of a slice that's namespaced by controller name - so that controllers should be able to update only their own Conditions. I'd expect that mesh controllers should have a distinct controller name string, so this method should continue working.

[evankanderson]

@thockin has said that getting multiple controllers to cooperate on node health status was very subtle, so I'd be careful and maybe distill his experience if you need multiple controllers writing one status.

[robscott]

I think our practice of including controllerName for each distinct set of conditions should largely allow multiple controllers to cooperate on status for the same resource, but open to other potential improvements.

[evankanderson]

How do controllerName conditions roll up to Accepted?

Also, controllers probably need to be sure to use PATCH on the status sub-resource or do conditional PUTs and avoid extra writes if the conditions haven't changed.

[youngnick]

For some resources, all conditions are namespaced by controllerName. I agree we need to ensure that the rollup behavior is specified. I'm going to add a section about partial validity that should help.

[mikemorris]

For HTTPRoute as a specific example, the structure of RouteParentStatus appears well-suited to handle a separate list of conditions for each parentRef or controller, so a mesh or Service relationship for a route would have a separate set of conditions and Accepted status than those used for relaying the status of a concurrent parent relationship to a Gateway, with no need to collaborate on writing to a single status.

[youngnick]

Okay, I think I've captured all the outstanding comments in that update. Please take a look, our timebox for this last pass of reviews is in just under 48 hours, so that I can start working on the implementation.

[shaneutt]

Given the importance with timing of this one, I don't see anything in this GEP that's problematic enough that we couldn't merge this and if all else fails sort something out in the follow-up PR. As such I'm approving, but I will leave the LGTM to someone else as this change has a strong impact.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: keithmattix, shaneutt, youngnick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [shaneutt,youngnick]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[robscott]

Thanks @youngnick! These are great updates. A few nits and a question about naming but really like this direction.

[robscott]

I don't feel strongly about this, but any reason not just to use Accepted everywhere instead of Attached in most places and Accepted in this one place?

[skriss]

I had the same thought and am +1 to this, Accepted seems a little bit easier to understand for users IMO and can be consistently used across all resources.

[robscott]

Is Accepted distinct from Attached here? Would Accepted ever be true if ResolvedRefs was false?

[youngnick]

I hadn't updated, but with the change to Accepted everywhere, this is now correct, thanks.

[robscott]

Why would we set default values here? What happens when multiple controllers are implementing the same resource like a HTTPRoute? I would prefer relying on conformance tests to require controllers to set these conditions.

[youngnick]

I was intending to make it that the non-controllerName Conditions will be present, but I'll leave this out for now.

[robscott]

It seems like this paragraph is missing a sentence that says that a separate set of Error conditions will exist that will only be set when they are true.

[robscott]

Tiny nit: This sentence feels a bit too prescriptive, maybe "before programming the Route" instead of saying that it needs to be set to false?

[youngnick]

Yeah, that seems like a good idea.

[robscott]

Seeing it laid out this way makes me think that the smaller change would be to use Accepted everywhere since it would only represent a change to Gateway (and Listener within it).

[youngnick]

Yeah, agreed. I've migrated the proposal to this.

[robscott]

I don't understand this. What would empty resources look like? If they're not unattached what are they?

[robscott]

I think we'd had some conversations about making this required for conformant Gateway implementations. Probably out of scope for this GEP, just wanted to make sure it was intentionally excluded.

[youngnick]

Good point, I updated.

[howardjohn]

I am not expert here things seem reasonable to me

[howardjohn]

On some impls, provisioning takes about a minute. That may mean that some errors that are immediately known are masked. Or I suppose they aren't masked, but we need to wait the entire minute to know if its valid. The message field could say "all fields valid, waiting for deployment" perhaps, but that isn't in the machine parse-able section.

That may not warrant change, just a thought

[youngnick]

I worded this carefully, so as not to imply that the provisioning needs to have completed, just that it can be started. The completion of infra provisioning should be handled by Programmed or Ready, or both.

[evankanderson]

Conditions which appear and disappear from a resource can make status reporting UIs harder, so I'd recommend keeping the Condition set consistent.

[robscott]

I agree that we should have a set of permanent and reliable conditions that are always present, but I personally don't think it makes sense for error conditions to always be present, especially since that tends to lead to confusing double negative states.

[evankanderson]

@thockin has said that getting multiple controllers to cooperate on node health status was very subtle, so I'd be careful and maybe distill his experience if you need multiple controllers writing one status.

[evankanderson]

This will make UI display somewhat harder (green dots / checkmarks / exclamation points), but I realize it may align with human reading a bit more easily.

I don't have a strong opinion here, just want to highlight the cost elsewhere in the stack.

[evankanderson]

Is it possible for something like an HTTPRoute to define two paths, one of which is "valid enough" config, while the other is garbage?

If so, are there expectations or requirements from a conformance PoV ("always apply all or none") that we need to specify?

[evankanderson]

You might also get a BackendRefEmpty condition...

[sunjayBhatia]

Is it possible for something like an HTTPRoute to define two paths, one of which is "valid enough" config, while the other is garbage?

Most cases of an invalid configuration in one route rule (with others being completely valid within the same HTTPRoute) seem to be misconfigurable only in the context of a reference to another resource and fall into the ResolvedRefs bucket (which seems stated below and possibly agreed upon that Attached: true be set with ResolvedRefs: false and return HTTP code 500 for the invalid rule).

One such example not covered is if you were to use a HTTPRequestRedirectFilter and HTTPURLRewrite in the list of filters for an individual HTTPRouteRule (ref). At the moment it doesn't appear we have prescribed any status code or other error handling for this error. This seems to fall into the bucket of things that cannot produce data plane configuration.

Going by the statements below, if this rule were the only one present Attached: false should be set, since it cannot result in any configuration (with another more specific error condition). If present with other valid rules, Attached: true should be set (with another more specific error condition again). I can definitely see the argument for setting Attached: false however in the latter case, since stated above, the Attached condition means The resource is syntatically and semantically valid, and internally consistent.

It seems that the Valid: true condition mentioned above ends up being somewhat equivalent to having for example for HTTPRoute ResolvedRefs: true and no error conditions (and vice versa, Valid: false equivalent to ResolvedRefs: false or some more specific error condition). Maybe this framing helps a little with determining possible merits of having another condition?

If so, are there expectations or requirements from a conformance PoV ("always apply all or none") that we need to specify?

Just doing a quick pass, I don't think I saw any existing conformance tests that cover this kind of thing so I agree that would be good to add.

[youngnick]

Okay, updated again, this feels like it's getting closer.

Changelog this time:

• Changed references to Attached to Accepted. I think @robscott and @skriss are right here, this is probably clearer.
• Added a section on partial validity to explain this better.
• Added the example @sunjayBhatia gave to the examples.

[robscott]

Thanks @youngnick! This is a really well written GEP. Since previous generations of this PR have already been approved a couple times, I think we've got pretty clear consensus around this approach.

/lgtm

[mikemorris]

Quite late in doing a thorough review of this, but really happy with where it ended up - feels like a good step forwards in resolving some of the current ambiguity in status conditions. Thanks @youngnick!

[mikemorris]

I actually hadn't realized the uniqueness on type constraint - this should probably be documented more clearly within the Gateway API spec, as it has some bearing on how "multiple error" cases may be expressed, and explains the structure of RouteParentStatus, which will be increasingly relevant if we start expecting resources like HTTPRoute to bind to multiple parentRefs (for GAMMA and N/S concurrent usage or route delegation).

[mikemorris]

For HTTPRoute as a specific example, the structure of RouteParentStatus appears well-suited to handle a separate list of conditions for each parentRef or controller, so a mesh or Service relationship for a route would have a separate set of conditions and Accepted status than those used for relaying the status of a concurrent parent relationship to a Gateway, with no need to collaborate on writing to a single status.

[mikemorris]

With the update to use Accepted instead of Attached, GatewayClass is no longer an exception.

[mikemorris]

For both of these cases, the ResolvedRefs RouteConditionReason should be BackendNotFound

[mikemorris]

The ResolvedRefs RouteConditionReason should be RefNotPermitted

[mikemorris]

I think the ResolvedRefs RouteConditionReason in this case should be InvalidKind, because of the GKV of the reference is invalid. We may want to consider adding a condition specifically for unsupported ExtensionRefs, or note that the reference with an invalid kind should be included in the Condition message field.

[mikemorris]

Link appears broken due to a missing footnote reference.

[robscott]

@mikemorris these are great bits of feedback, do you have time to create a follow up PR or issue to address any that are straightforward to fix? Concerned that otherwise we'll lose track of these.