Unable to use Azure WI Proxy after automatic update to 1.1.0

[liamgib]

Describe the bug
It appears that overnight our cluster has automatically upgraded Azure WI to 1.1.0. Since this upgrade, deployments using the proxy sidecar are unable to authenticate.

We're getting the following error response from az login --identity

ERROR: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned http error: 400, reason: Bad Request

Why has this automatically upgraded? Was there a breaking change?

Steps To Reproduce
N/A
Possibly... use v1.0.0 with the proxy sidecars and then upgrade to 1.1.0

Expected behavior
Able to login to a federated managed identity with az login --identity

Logs
azwi-proxy-init

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
AZWI_PROXY_OUTPUT  tcp  --  anywhere             169.254.169.254      tcp dpt:http

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain AZWI_PROXY_OUTPUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             owner UID match 1501
AZWI_PROXY_REDIRECT  all  --  anywhere             anywhere            

Chain AZWI_PROXY_REDIRECT (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 8000

azwi-proxy

{"level":"info","timestamp":"2023-09-12T22:17:50.322015Z","logger":"proxy","caller":"/workspace/pkg/proxy/proxy.go:97$proxy.(*proxy).Run","message":"starting the proxy server","port":8000,"userAgent":"azure-workload-identity/proxy/v1.1.0 (linux/amd64) 656a033/2023-05-08-20:15"}
{"level":"info","timestamp":"2023-09-12T22:17:50.367875Z","logger":"proxy","caller":"/workspace/pkg/proxy/proxy.go:191$proxy.(*proxy).readyzHandler","message":"received readyz request","method":"GET","uri":"/readyz"

azure-wi-webhook-controller-manager

{"level":"info","timestamp":"2023-09-12T22:17:26.516130Z","logger":"entrypoint","caller":"/workspace/main.go:99$main.mainErr","message":"initializing metrics backend","backend":"prometheus"}
{"level":"info","timestamp":"2023-09-12T22:17:26.516257Z","logger":"entrypoint","caller":"/workspace/main.go:105$main.mainErr","message":"setting up manager","userAgent":"azure-workload-identity/webhook/v1.1.0 (linux/amd64) 656a033/2023-05-08-20:13"}
{"level":"info","timestamp":"2023-09-12T22:17:26.893995Z","logger":"controller-runtime.metrics","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/metrics/listener.go:44$metrics.NewListener","message":"Metrics server is starting to listen","addr":":8095"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894299Z","logger":"entrypoint","caller":"/workspace/main.go:191$main.setupProbeEndpoints","message":"added healthz and readyz check"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894325Z","logger":"entrypoint","caller":"/workspace/main.go:146$main.mainErr","message":"starting manager"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894448Z","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:369$manager.(*controllerManager).httpServe.func1","message":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8095"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894506Z","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:369$manager.(*controllerManager).httpServe.func1","message":"Starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894551Z","logger":"entrypoint","caller":"/workspace/main.go:162$main.setupWebhook","message":"registering webhook to the webhook server"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894659Z","logger":"controller-runtime.webhook","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/server.go:149$webhook.(*Server).Register","message":"Registering webhook","path":"/mutate-v1-pod"}
{"level":"info","timestamp":"2023-09-12T22:17:26.894732Z","logger":"controller-runtime.webhook.webhooks","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/server.go:217$webhook.(*Server).Start","message":"Starting webhook server"}
{"level":"info","timestamp":"2023-09-12T22:17:26.896383Z","logger":"controller-runtime.certwatcher","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/certwatcher/certwatcher.go:131$certwatcher.(*CertWatcher).ReadCertificate","message":"Updated current TLS certificate"}
{"level":"info","timestamp":"2023-09-12T22:17:26.896471Z","logger":"controller-runtime.webhook","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/server.go:271$webhook.(*Server).Start","message":"Serving webhook server","host":"","port":9443}
{"level":"info","timestamp":"2023-09-12T22:17:26.896579Z","logger":"controller-runtime.certwatcher","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/certwatcher/certwatcher.go:85$certwatcher.(*CertWatcher).Start","message":"Starting certificate watcher"}
{"level":"debug","timestamp":"2023-09-12T22:19:27.792388Z","logger":"controller-runtime.webhook.webhooks","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:96$admission.(*Webhook).ServeHTTP","message":"received request","webhook":"/mutate-v1-pod","UID":"0e1844e9-ffa1-4e38-a3d9-b3222f9c50f2","kind":"/v1, Kind=Pod","resource":{"group":"","version":"v1","resource":"pods"}}
{"level":"debug","timestamp":"2023-09-12T22:19:27.896804Z","logger":"controller-runtime.webhook.webhooks","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:143$admission.(*Webhook).writeAdmissionResponse","message":"wrote response","webhook":"/mutate-v1-pod","code":200,"reason":"","UID":"0e1844e9-ffa1-4e38-a3d9-b3222f9c50f2","allowed":true}
{"level":"debug","timestamp":"2023-09-12T22:19:27.903166Z","logger":"controller-runtime.webhook.webhooks","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:96$admission.(*Webhook).ServeHTTP","message":"received request","webhook":"/mutate-v1-pod","UID":"d61564ff-f8bb-4d4c-9996-c26907d08390","kind":"/v1, Kind=Pod","resource":{"group":"","version":"v1","resource":"pods"}}
{"level":"debug","timestamp":"2023-09-12T22:19:27.904410Z","logger":"controller-runtime.webhook.webhooks","caller":"/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:143$admission.(*Webhook).writeAdmissionResponse","message":"wrote response","webhook":"/mutate-v1-pod","code":200,"reason":"","UID":"d61564ff-f8bb-4d4c-9996-c26907d08390","allowed":true}

Environment

• Kubernetes version (use kubectl version):

Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.3
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
  azure-cni
• Others:

Additional context

[liamgib]

We have uninstalled the AKS Addon, and tried to install v1.1.0 via Helm but it had the same issue. After reverting back to v1.0.0 we're able to use az login --identity again.

[pdefreitas]

It might be related with Azure/azure-cli#26858.
The workaround provided there did the trick for me using the latest proxy.

az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID

Not ideal but it can mitigate the problem for now.