[BUG] VaultException: Configured token cannot be verified. It is most likely expired or invalid.

[ichilton]

Description

Had a working Salt / Vault setup using the old integration.

Started to get deprecation warnings so have been attempting to switch to the new vault extension.

Have been trying for hours and I can't get past:

Error running 'vault.read_secret': Failed to read secret! VaultException: Configured token cannot be verified. It is most likely expired or invalid.

I get this if I try issuing from the master or supply locally on the client.

The token is definitely valid (tested it direct on vault and tried different ones). Even tried the root token.

If I change the URL, it fails with a different error, so definitely seems to be attempting to connect to valult.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

/etc/salt/master.d/vault.conf:

vault:
  auth:
    method: token
    token: xxx
  server:
    url: https://vault.myinstance.com
  cache:
    backend: disk
  issue:
    type: token
    token:
      params:
        explicit_max_ttl: 30  # Tokens will be valid for 30s
        num_uses: 10          # Tokens will be limited to 10 uses
  policies:
    assign:
      - 'salt/minions'
      - 'salt/minion/{minion}'

Please be as specific as possible and give set-up details.

• on-prem machine
• [ X ] VM (Virtualbox, KVM, etc. please specify) - KVM/Ganeti
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• [ X ] onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

salt-call vault.read_secret salt/minion/myminion.domain.com/something

Expected behavior

Should return the secret.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Extensions:
 saltext.vault: 1.0.0

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-21-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm

[lkubb]

To verify the token, the client issues a lookup request to the server and checks the status code of the response. If it is not 200, it interprets this as the token being invalid.

It would be very interesting to see what the response is exactly. Unfortunately, there is no logging at that line. Can you add it and report back what it says? You should find the relevant file at /opt/saltstack/salt/extras-3.10/saltext/vault/utils/vault/factory.py:

diff --git a/src/saltext/vault/utils/vault/factory.py b/src/saltext/vault/utils/vault/factory.py
index 9978efb..ccad692 100644
--- a/src/saltext/vault/utils/vault/factory.py
+++ b/src/saltext/vault/utils/vault/factory.py
@@ -621,6 +621,7 @@ def _fetch_token(config, opts, token_cache, unwrap_client, force_local=False, em
                 # lookup and verify raw token
                 token_info = unwrap_client.token_lookup(embedded_token, raw=True)
                 if token_info.status_code != 200:
+                    log.error(f"Token lookup failed! status: {token_info.status_code} text: {token_info.text}")
                     raise VaultException(
                         "Configured token cannot be verified. It is most likely expired or invalid."
                     )

You might need to restart the daemon for the change to take effect.

[lkubb]

@ichilton Since I did not hear back from you, did you find the cause? If not, you can try the new release, which at least logs the API response to aid in debugging.

[ichilton]

Hi @lkubb,

Sorry, I completely missed these replies!

I just came back to this in the hope that the problem might have gone away / been fixed, but seeing the same.

I upgraded to v1.1.1 as you suggested and I see the same error, but with an indication that it's getting a 503 response.

root@salt-master:~# salt-pip  list | grep vault
saltext.vault      1.0.0

root@salt-master:~# salt-pip install --upgrade saltext.vault
Requirement already satisfied: saltext.vault in /opt/saltstack/salt/extras-3.10 (1.0.0)
Collecting saltext.vault
  Downloading saltext.vault-1.1.1-py2.py3-none-any.whl.metadata (5.4 kB)
[snip requirement already satisfied stuff]
Downloading saltext.vault-1.1.1-py2.py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 kB 5.9 MB/s eta 0:00:00
Installing collected packages: saltext.vault
  Attempting uninstall: saltext.vault
    Found existing installation: saltext.vault 1.0.0
    Uninstalling saltext.vault-1.0.0:
      Successfully uninstalled saltext.vault-1.0.0
Successfully installed saltext.vault-1.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.2
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
root@salt-master:~# salt-pip list | grep vault
saltext.vault      1.1.1
root@salt-master:~# systemctl restart salt-master
root@salt-master:~# systemctl restart salt-minion
root@salt-master:~# salt-call vault.read_secret salt/global/test
[ERROR   ] Token lookup failed! status code: 502
Error running 'vault.read_secret': Failed to read secret! VaultException: Configured token cannot be verified. It is most likely expired or invalid.

Any ideas?

Thanks,

Ian

[ichilton]

Hi,

Ok, tracked down the 503 error - i've got nginx in front of vault, for SSL termination and as a proxy.

I had this:

  location / {
    proxy_pass http://localhost:8200/;
  }

Vault is only listening on v4:

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_disable	= 1
}

So I was getting this in the nginx error.log:

2024/07/30 20:43:56 [error] 3350#3350: *162 connect() failed (111: Connection refused) while connecting to upstream, client: <redacted>, server: _, request: "GET /v1/auth/token/lookup-self HTTP/1.1", upstream: "http://[::1]:8200/v1/auth/token/lookup-self", host: "<redacted>"

So I fixed it with:

  location / {
    proxy_pass http://127.0.0.1:8200/;
  }

Now I get a 403 error...

root@salt-master:~# salt-call vault.read_secret salt/global/test
[ERROR   ] Token lookup failed! status code: 403
Error running 'vault.read_secret': Failed to read secret! VaultException: Configured token cannot be verified. It is most likely expired or invalid.

<redacted> - - [30/Jul/2024:20:45:31 +0100] "GET /v1/auth/token/lookup-self HTTP/1.1" 403 80 "-" "python-requests/2.31.0"

Any ideas? - or is there a way of seeing why vault would be doing that?

If I do vault token lookup <token from /etc/salt/master.d/vault.conf>, it works fine and doesn't expire until a future date.

Thanks,

Ian

[ichilton]

Hi @lkubb,

If I do a request to that URL from the salt-master, it does succeed.

VAULT_TOKEN environment var is set with the value from:

vault:
  auth:
    method: token
    token: <TOKEN>

root@salt-master:~# curl -I --request GET --header "X-Vault-Token: $VAULT_TOKEN" https://vault.fqdn.com/v1/auth/token/lookup-self
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Tue, 30 Jul 2024 20:18:16 GMT
Content-Type: application/json
Content-Length: 685
Connection: keep-alive
Cache-Control: no-store
Strict-Transport-Security: max-age=31536000; includeSubDomains

You're not doing a HTTP HEAD request are you?

root@salt-master:~# curl -I --header "X-Vault-Token: $VAULT_TOKEN" https://vault.fqdn.com/v1/auth/token/lookup-self
HTTP/1.1 403 Forbidden

Thanks,

Ian

[ichilton]

tcpdump'ing port 8200 on the vault server, I can see the raw HTTP requests hitting vault.

I can see that the token used in the token-lookup request is not the token from the config file (interestingly it does have the same first few characters - but then is otherwise different).

I was then thinking it must be the token for the minion that vault issues by calling auth/token/create with the configured token.

However, there is no sign of any such call to vault - the only call is the /v1/auth/token/lookup call in question.

So where is it getting the token from??

Thanks,

Ian

[lkubb]

@ichilton

Sorry, I completely missed these replies!

No worries!

Ok, tracked down the 503 error

Awesome :)

Now I get a 403 error... Any ideas? - or is there a way of seeing why vault would be doing that?

Permission denied on this endpoint means the token the minion uses is invalid for that server.

I can see that the token used in the token-lookup request is not the token from the config file.

The token lookup is only done when using local configuration, not when a minion receives a freshly issued token from the master. I would guess this is where the issue can be found:

Since you're using salt-call (even on the master node), you're calling the minion, which apparently has a local configuration itself. Just remove the vault configuration, especially the outdated token, from /etc/salt/minion* configuration files (or replace it with a valid one).

Edit:
To verify the master token itself, not the minion one, you can run the following:

salt-run salt.cmd vault.query get auth/token/lookup-self

[ichilton]

Soooo, I just found some old vault config hidden in a file under /etc/salt/master.d, which explains the incorrect token!

but now I get this:

root@salt-master:~# salt-call vault.read_secret salt/global/test
[ERROR   ] Failed to get Vault connection from master! An error was returned: VaultInvocationError: child policies must be subset of parent
Error running 'vault.read_secret': Failed to read secret! CommandExecutionError: {'error': 'VaultInvocationError: child policies must be subset of parent'}

Config is now:

vault:
  auth:
    method: token
    token: "token"
  server:
    url: https://vault.fqdn.com
  cache:
    backend: disk
  issue:
    type: token
    token:
      params:
        explicit_max_ttl: 300  # Tokens will be valid for 300s
        num_uses: 100          # Tokens will be limited to 100 uses
  policies:
    assign:
      - salt/minions
      - salt/minion/{minion}

Now where am I going wrong? :)

Thanks,

Ian

[lkubb]

https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#prerequisites

When using token issuance, you need to ensure your master is allowed to issue roles policies different from itself (or add all roles policies you want to assign to minions to it). There are two ways to do that:

1. Give it sudo capabilities on auth/token/create (strongly discouraged)
2. Create a token role

[ichilton]

Thank you - will read up and try again later!

[lkubb]

@ichilton In case it's still relevant, I added a more detailed description of Token Roles and their purpose to the docs: https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#credential-issuance

Since the cause of the original issue turned out to be external and the new logging helped the diagnosis, I'm going to close this. Feel free to ask further questions in here or in a discussion. :) Thanks!