Webhook timeouts settings make helm chart fail

[rahadiangg]

System that I use:

Kubernetes v1.27.13
Helm v3.14.4
Rancher v2.8.1 (chart rancher-2.8.1)
Rnacher Webhook v0.4.2 (chart rancher-webhook-103.0.1+up0.4.2)

Sometimes I got this error when doing helm upgrade command:

Error: UPGRADE FAILED: create: failed to create: Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "[https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=30s](https://rancher-webhook.cattle-system.svc/v1/webhook/mutation/secrets?timeout=30s)": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The default --timeout option from helm upgrade command is 5m (300 seconds). But the default mutatingwebhookconfigurations.admissionregistration.k8s.io for rancher (rancher.cattle.io) is 15s like below:

rancher.cattle.io Mutating Webhook

- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    caBundle: <hidden>
    service:
      name: rancher-webhook
      namespace: cattle-system
      path: /v1/webhook/mutation/secrets
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - DELETE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 15

That means rancher cancel the request because hit the timeout limit. Sometimes helm chart take longer when we have several helm release in one namespace, especially when one release have several history. the helm release history also stored in Kubernetes secret and I think this related with current case. I try to increase the timeout into 300, but the possible value is between 1 and 30.

The MutatingWebhookConfiguration "rancher.cattle.io" is invalid: webhooks[4].timeoutSeconds: Invalid value: 300: the timeout value must be between 1 and 30 seconds

The question is, it's there a another way to solve my case ?. Currently the options to solve this is delete that rancher.cattle.io Mutating Webhook. But in several time, that will automatically created again.