You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Webhook pods are marked ready before webhook is actually servicing requests, causing dependent pods to miss their mutations.
Steps To Reproduce
Install via helm, according to the docs (or using a terraform helm_release, in our case).
Watch logs to see when Serving webhook server is actually logged. Sometimes this takes several (~3.5) minutes but in the example below only a couple seconds.
When deploying subsequent application pods that are properly labeled, they are scheduled and start, but miss the mutation because no webhook is actually servicing requests yet (so they pass through without ENV vars, etc).
Expected behavior
The pod should not be marked ready until the webhook is actually usable, or you cannot (meaningfully) wait for the azwi deployment via helm --wait.
Logs
I0120 19:42:19.117325 1 main.go:88] entrypoint "msg"="initializing metrics backend" "backend"="prometheus"
I0120 19:42:19.117395 1 main.go:95] entrypoint "msg"="setting up manager" "userAgent"="azure-workload-identity/webhook/v0.15.0 (linux/amd64) 9e27154/2022-12-14-01:21"
I0120 19:42:19.795689 1 listener.go:44] controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8095"
I0120 19:42:19.795895 1 main.go:115] entrypoint "msg"="setting up cert rotation"
I0120 19:42:19.796043 1 main.go:147] entrypoint "msg"="starting manager"
NOT READY --> I0120 19:42:19.796301 1 internal.go:366] "msg"="Starting server" "addr"={"IP":"::","Port":9440,"Zone":""} "kind"="health probe"
I0120 19:42:19.796312 1 rotator.go:204] cert-rotation "msg"="starting cert rotator controller"
I0120 19:42:19.796375 1 internal.go:366] "msg"="Starting server" "addr"={"IP":"::","Port":8095,"Zone":""} "kind"="metrics" "path"="/metrics"
I0120 19:42:19.796423 1 controller.go:185] "msg"="Starting EventSource" "controller"="cert-rotator" "source"="&{{%!s(*v1.Secret=&{{ } { 0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc00044b4e0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0120 19:42:19.796451 1 controller.go:185] "msg"="Starting EventSource" "controller"="cert-rotator" "source"="&{{%!s(*unstructured.Unstructured=&{map[apiVersion:admissionregistration.k8s.io/v1 kind:MutatingWebhookConfiguration]}) %!s(*cache.informerCache=&{0xc00044b4e0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I0120 19:42:19.796463 1 controller.go:193] "msg"="Starting Controller" "controller"="cert-rotator"
I0120 19:42:19.896669 1 rotator.go:245] cert-rotation "msg"="refreshing CA and server certs"
I0120 19:42:19.896755 1 controller.go:227] "msg"="Starting workers" "controller"="cert-rotator" "worker count"=1
E0120 19:42:19.896848 1 rotator.go:674] cert-rotation "msg"="secret is not well-formed, cannot update webhook configurations" "error"="Cert secret is not well-formed, missing ca.crt"
I0120 19:42:21.046113 1 rotator.go:722] cert-rotation "msg"="Ensuring CA cert" "gvk"={"Group":"admissionregistration.k8s.io","Version":"v1","Kind":"MutatingWebhookConfiguration"} "name"="azure-wi-webhook-mutating-webhook-configuration"
I0120 19:42:21.065990 1 rotator.go:722] cert-rotation "msg"="Ensuring CA cert" "gvk"={"Group":"admissionregistration.k8s.io","Version":"v1","Kind":"MutatingWebhookConfiguration"} "name"="azure-wi-webhook-mutating-webhook-configuration"
E0120 19:42:21.427249 1 rotator.go:247] cert-rotation "msg"="could not refresh CA and server certs" "error"="Operation cannot be fulfilled on secrets \"azure-wi-webhook-server-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0120 19:42:21.445770 1 rotator.go:271] cert-rotation "msg"="no cert refresh needed"
(Sometimes this takes minutes...)
I0120 19:42:21.445841 1 rotator.go:757] cert-rotation "msg"="certs are ready in /certs"
I0120 19:42:21.445866 1 rotator.go:777] cert-rotation "msg"="CA certs are injected to webhooks"
I0120 19:42:21.445882 1 main.go:159] entrypoint "msg"="setting up webhook server"
I0120 19:42:21.445911 1 main.go:163] entrypoint "msg"="registering webhook to the webhook server"
I0120 19:42:21.445992 1 server.go:216] controller-runtime/webhook/webhooks "msg"="Starting webhook server"
I0120 19:42:21.446015 1 server.go:148] controller-runtime/webhook "msg"="Registering webhook" "path"="/mutate-v1-pod"
I0120 19:42:21.446177 1 certwatcher.go:131] controller-runtime/certwatcher "msg"="Updated current TLS certificate"
READY HERE! --> I0120 19:42:21.446276 1 server.go:270] controller-runtime/webhook "msg"="Serving webhook server" "host"="" "port"=9443
I0120 19:42:21.446302 1 certwatcher.go:85] controller-runtime/certwatcher "msg"="Starting certificate watcher"
Environment
Kubernetes version (use kubectl version):
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.24.6
Cloud provider or hardware configuration: AKS
Additional context
The cert resolve is sometimes fairly quick (such as this instance) but can ocassionally take several minutes.
We are using terraform for our helm deployment.
resource"helm_release""azwi-webhook" {
name="workload-identity-webhook"namespace=local.azwi_namespacechart="workload-identity-webhook"repository="https://azure.github.io/azure-workload-identity/charts"wait=truedepends_on=[
kubernetes_namespace.azwi,
kubernetes_service_account.app,
azuread_application_federated_identity_credential.azwi
]
set {
name ="azureTenantID"
value = data.azuread_client_config.current.tenant_id
}
}
The text was updated successfully, but these errors were encountered:
Describe the bug
Webhook pods are marked ready before webhook is actually servicing requests, causing dependent pods to miss their mutations.
Steps To Reproduce
Serving webhook server
is actually logged. Sometimes this takes several (~3.5) minutes but in the example below only a couple seconds.Expected behavior
The pod should not be marked ready until the webhook is actually usable, or you cannot (meaningfully) wait for the azwi deployment via
helm --wait
.Logs
Environment
kubectl version
):Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.24.6
Additional context
The cert resolve is sometimes fairly quick (such as this instance) but can ocassionally take several minutes.
We are using terraform for our helm deployment.
The text was updated successfully, but these errors were encountered: