Declarative update of configmaps and secrets from the contents of files

[sijnc]

Currently AFAIK you can only create a secret from a file using:

kubectl create secret generic <name> --from-file=<key>=<file>

The drawback is this, If a change is made to the file you need to delete and then recreate the secret:

kubectl delete secret <name>
kubectl create secret generic <name> --from-file=<key>=<file>

This is awkward because to update the other items (service, deployment, pv) in the same app you just run:

kubectl apply -f <filename>

I realize that I can create a script to base64 encode the contents of the file and inject it into a secret spec file and then use create/apply/delete as with other API objects but this just feels awkward because the create/apply/delete workflow is so clean.

For context this is a redis config file that contains a password and I'm mounting it with a volumeMount in a deployment:

- name: redis-conf-secret
  readOnly: true
  mountPath: /etc/redis

[eparis]

so you are hoping for kubectl apply secret generic <name> --from-file=<key>=<file> ?

[sijnc]

@eparis,

Something like this:

kubectl apply -f my-secret.yaml

my-secret.yaml

apiVersion: v1
    kind: Secret
    metadata:
      name: my-secret
    dataFromFile:
      key: redis.conf

When kubectl create or apply is run, kubectl would load and base64 encode the file on the client where kubectl is running before making its API calls.

Not really asking for a new feature here, the workflow just seems awkward but maybe this is something that should be in external tooling or a script or maybe there's a better way. I'm keeping my redis.conf in the same folder as the svc, pv, pvc, dm yaml files (seems to make sense) but I'm forced to delete and create the secret to change it when creating from a file vs just using apply.

I also haven't explored the side effects on the pod of create/delete on the secret while the pod is running so that may be another issue.

[bgrant0607]

See also #19575, #22059

[bgrant0607]

@sijnc The API can't read your local file, so what you wrote won't work.

Do you want the secret updated, or to create a new secret? Do you want all consumers of the secret to get the updated value simultaneously?

[bgrant0607]

cc @kubernetes/kubectl @pmorie

[bgrant0607]

You may be able to do:

kubectl create secret generic <name> --from-file=<key>=<file> --dry-run | kubectl apply -f -

[sijnc]

@bgrant0607 In my second comment I talk about kubectl loading the local file when running:

kubectl create | apply -f my-secret.yaml

When it encounters dataFromFile kubectl base64 encodes it before inserting it into the spec and submitting it to the API.

WTR Kubernetes updating the secret, it would probably be confusing if calling kubectl apply did not update the secret and it was consumed as a volume but how that happens, roll the pods or just update the file on the pod I haven't given much thought. I was more pointing out that it would be nice if the create/apply/delete workflow worked with secrets.

[pmorie]

@sijnc good news, @ingvagabund has made a PR today to make extend the secret volume and make it behave like the configmap volume: atomic updates when the secret updates, and control over filenames and subdirectories of keys in the volume

[pmorie]

@sijnc oops; #25285

[sijnc]

@pmorie #25285 adds KeyToPath and also atomic updates when the secret is updated. If you look at my first two comments they address using a secret with the create/apply/delete commands in kubectl.

I always found the base64 encoding of values in secret yaml files awkward and wondered why you can't just reference a path and kubectl could do the encoding before calling the kubernetes API.

It looks like the reasoning behind this is that the secret spec is an API object so adding dataFromFile to a spec that is ultimately consumed inside the cluster where the file is NOT present explains this and the decision to add the --from-file=<file> flag to kubectl was a better option.

Since this is a config file that is deployed as a secret I'll continue to keep the config file separate from the secret yaml file and just script the extra commands to make this work.

Thanks for sharing the info about atomic secret updates, that's good to know.

[pmorie]

@sijnc Yep, it was meant to be informational -- since the secret volume plugin doesn't update at all right now, it is a piece needed to give you what you want :)

I need to play around and see if I can make a recommendation to you on how to update the secret. In the meantime, do check out the issue the @bgrant0607 linked (#22059) -- it would allow you to submit Secret descriptors with strings instead of base64 encoded byte arrays, which would effectively allow you to use kubectl replace (and perhaps kubectl apply, I will need to familiarize myself more with apply).

@liggitt any comment? ^

[sijnc]

@pmorie #22059 is a nice way of getting around the base64 legacy issue and allow raw strings, the way it's done could also be applied to a file by adding a second descriptor file:path.

Value is a string:

"username": "string:myuser"

Value is a file:

"config-file": "file:path/to/file.conf"

All good but you still have the issue where this needs to be handled on the client with the file and converted to string or base64 before submitting to the kubernetes API. The API vs kubectl issue wasn't fully apparent to me until @bgrant0607 pointed it out.

Adding special client side cases in the API spec are probably a bad idea but in the end I was just looking for a way to have a config file stored from a secret yaml that referenced the local config file but it's starting to look like the config file should be stored in a repo which can be pulled from the cluster, like the GitRepoVolumeSource with some type of authentication since this conf file contains a password.

Thanks for looking at this, let me know if you have any other thoughts.

[liggitt]

#22059 works server side as well. The API should not perform transformations/conversions that depend on state outside the API content (like an external file reference), in my opinion.

[sijnc]

@liggitt I agree and what I was suggesting happen client side would need to pollute the secret spec with an additional value that's not required in the API.

[bgrant0607]

Note that apply has the additional issue that secrets are captured by the config annotation: #23564.

[bgrant0607]

@sijnc It sounds like what you're looking for is client-side preprocessing. Some people in the community are doing that using various tools: jinja, jsonnet, etc. It would be easy to write a Makefile to translate from the raw secret data to a yaml file containing the K8s Secret, and then invoke kubectl apply.

[bgrant0607]

I expanded this to include configmaps, so that I could close #30337

[bgrant0607]

How I see this fitting into the big picture is covered by my uber doc: https://goo.gl/T66ZcD

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale
/lifecycle frozen

[hofnarwillie]

This worked for me:

kubectl create secret generic <name> --from-file=<key>=<file> --dry-run -o json | kubectl apply -f -

[bgrant0607]

See also:
https://github.com/kubernetes-sigs/kustomize

[nyetwurk]

This worked for me:

kubectl create secret generic <name> --from-file=<key>=<file> --dry-run -o json | kubectl apply -f -

I got this warning:

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

[fOO223Fr]

@nyetwurk you got this warning because when you first created the resource you didn't use --save-config flag refer here for detailed understanding of three way comparison which kubectl apply does

[nyetwurk]

@fOO223Fr Thanks!

[rdxmb]

The drawback is this, If a change is made to the file you need to delete and then recreate the secret:

kubectl delete secret <name>
kubectl create secret generic <name> --from-file=<key>=<file>

This seems also to be the case when not creating from file but creating from a CI-Pipeline like:

kubectl delete secret docker-registry
kubectl create secret docker-registry $DOCKER_REGISTRY --docker-server=$DOCKER_REGISTRY --docker-username=$DOCKER_REGISTRY_PULL_USER --docker-password=$DOCKER_REGISTRY_PULL_PASSWORD

[rikatz]

@sijnc is this issue still something? I guess during the discussion folks got a lot of different approaches :)

If not, I guess this can be closed right?