Kubernetes Service Accounts secrets created at a fast pace

[kot0dama]

Steps to reproduce

1. Deploy postgresql-k8s charm channel=14/stable, revision=158
2. Let it run for a while (see following comments by the team responsible for running the application itself, if any)

Expected behavior

The k8s cluster is not overwhelmed with a high amount of secrets.

Actual behavior

A number of secrets are created and never cleaned up. These secrets are Service Account Tokens, named like model-exec-token-XXXXX where XXXXX is a random hexadecimal character.

Versions

Operating system: Ubuntu bionic
Juju CLI: 2.9.49
Juju agent: 2.9.49
Charm revision: 158
charmed-kubernetes: 1.21.14
Juju debug log: not available when reporting this bug

Additional context

This was detected at the IS level (k8s cluster operator), we are not the owners of the applications running in that k8s cluster so I will ask the owners of the application to reply to this bug report.

We found out quite a lot of secrets were created, when prometheus alerted about disk space issues. This was probably caused by the cardinality of some kube-state-metrics resulting of the secrets/SA creation pace.

Top 10 label names with value count
Name	Count
secret	221034

Top 10 series count by metric names
Name	Count
kube_secret_info	221070

It would seem Kubernetes is able to purge unused SA tokens starting cluster version 1.29, but then the pace at which these are created is worrying. Per the prometheus history, about 12000 such secrets were created in about 10 days, which amounts for almost one service account/secret per minute.

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4750

[dragomirp]

Hi, @kot0dama, can you confirm the juju version (2.9)? I don't think the charm itself should be creating secrets on Juju 2.9, so we should check if Patroni does so.

[taurus-forever]

Dear @kot0dama , is it still reproducible / topical? Tnx!

[kot0dama]

Hi, @kot0dama, can you confirm the juju version (2.9)? I don't think the charm itself should be creating secrets on Juju 2.9, so we should check if Patroni does so.

Yes, the juju version was 2.9.49

Dear @kot0dama , is it still reproducible / topical? Tnx!

As we are merely hosting the application running in our k8s cluster, I wouldn't know for sure. Afaik, the whole namespace has been removed by the team responsible for running it.
@kian99 any thoughts on this one please?

[kian99]

Hi,

The environment was cleaned up so unfortunately I don't have any more info to offer on this.