-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Image pull backoff due to race with image-pull-secret
Secret creation
#101
Comments
imagepullsecrets-patcher creates a secret in |
@sameersbn which component is responsible for patching Service Accounts with that secret? Is it also
No because images are pulled by kubelets, not by Pods. |
Yes imagesecret-patcher is the one that patches the service accounts |
I'm thinking it may be time to write our own imagepullsecret-patcher, because the way they iterate over namespaces on a schedule is slow due to the amount of GET requests which are sent constantly: https://github.com/titansoft-pte-ltd/imagepullsecret-patcher/blob/bdf0891920920d3e789a5b5bbf0ea041ad385746/main.go#L94-L130 We would be much better off having an informer watching Anyway, it seems like waiting long enough™ does eventually trigger a pull, but that's usually well over 6 min. |
image-pull-secret
Secret creationimage-pull-secret
Secret creation
could we try setting |
That would probably be counter productive, because the loop is already taking much longer than that to complete in prod. I made a quick test: created a ServiceAccount in my namespace, started a watch on it, and imagePullSecrets was only populated after 90 sec. It seems like the patcher is struggling with the amount of namespaces we have in prod. |
Closing because images are now public, and the aforementioned patcher was removed from our standard installation. |
Problem
It seems like the creation of the
image-pull-secret
Secret and/or its addition to new Service Accounts takes quite some time after the creation of a namespace or a ServiceAccount object. In my experiments, I observed delays of up to 90 seconds.This is causing a fair amount of timeouts in tests (
Error waiting for resource ...
) due to the following error (example):(another example I've seen was the
gcr.io/triggermesh-private/awseventbridgetarget:v1.7.0
image)On top of that, Kubernetes doesn't immediately propagate changes to ServiceAccounts'
imagePullSecrets
, and keeps trying to pull private images without credentials (at least for some time which I couldn't determine) if those credentials weren't referenced when the first pull attempt occurred.Solution
The difficulty here is that each integration may potentially be using its own Service Account (I'm thinking of multi-tenant sources, especially), so we can't make generic assumptions, such as always checking that the
default
Service Accounthas some
imagePullSecrets`.Instead, it's probably best to plug some specific logic to each test that requires image pull secrets, e.g. "watch this specific Service Account and wait until it references some imagePullSecret".
Not super portable, but those tests were designed to run against our own infra anyway.
We could also look into Knative's image cache. If I remember correctly, it is possible to make it work with private repos too, given the right credentials are passed to the serving controller.
The text was updated successfully, but these errors were encountered: