[BUG] Vault as external pillar is all or nothing

[jtraub91]

Description

When using vault as external pillar, if a particular minion does not have access to all secrets being integrated, the pillar rendering will fail upon the first secret it encounters that it does not have access to, and thus fail to render pillars even for which the minion does have access to.

Setup

1. Install vault
2. Start a dev vault server

vault server -dev -dev-listen-address 0.0.0.0:8200

3. Install salt-master and salt minion on different servers
4. Configure the salt-master with vault / external pillar

# /etc/salt/master.d/vault.conf
vault:
  url: http://<vault-url>
  verify: False
  auth:
    method: token
    token: <vault-token>
  policies:
    - salt/{minion}

# /etc/salt/master.d/ext_pillar.conf
ext_pillar:
  - vault:
      conf: path=secret/mysql
      nesting_key: vault.mysql
  - vault:
      conf: path=secret/postgres
      nesting_key: vault.postgres

# /etc/salt/master.d/peer_run.conf
peer_run:
  .*:
    - vault.generate_token

and on minions

# /etc/salt/minion.d/vault.conf
vault:
  verify: False

Also make sure secrets exist in vault at the above referenced locations, i.e. secret/mysql and secret/postgres, as well as policies that grant access to each secret path, respectively, which will be applied to minions templated by minion id. E.g.

# salt/minion0
path "secret/data/mysql" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

path "secret/metadata/mysql" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

# salt/minion1
path "secret/data/postgres" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

path "secret/metadata/postgres" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

Steps to Reproduce the behavior
Verify that each minion can see their respective secrets, but not others

# salt \* vault.read_secret secret/mysql
minion0:
    ----------
    password:
        yolofam
minion1:
    ERROR: Failed to read secret! HTTPError: 403 Client Error: Forbidden for url: http://192.241.214.57:8200/v1/secret/data/mysql

# salt \* vault.read_secret secret/postgres
minion0:
    ERROR: Failed to read secret! HTTPError: 403 Client Error: Forbidden for url: http://192.241.214.57:8200/v1/secret/data/postgres
minion1:
    ----------
    password:
        cowabunga

But with external pillar it's all or nothing

# salt \* pillar.items
minion1:
    ----------
    vault.mysql:
        ----------
    vault.postgres:
        ----------
minion0:
    ----------
    vault.mysql:
        ----------
        data:
            ----------
            password:
                yolofam
        metadata:
            ----------
            created_time:
                2022-03-21T19:30:29.169216688Z
            custom_metadata:
                None
            deletion_time:
            destroyed:
                False
            version:
                1
    vault.postgres:
        ----------

Notice that if the order is switched in the ext_pillar configuration, i.e.

ext_pillar:
  - vault:
      conf: path=secret/postgres
      nesting_key: vault.postgres
  - vault:
      conf: path=secret/mysql
      nesting_key: vault.mysql

then we get different results

~# salt \* pillar.items
minion0:
    ----------
    vault.mysql:
        ----------
    vault.postgres:
        ----------
minion1:
    ----------
    vault.mysql:
        ----------
    vault.postgres:
        ----------
        data:
            ----------
            password:
                cowabunga
        metadata:
            ----------
            created_time:
                2022-03-21T19:30:43.418454867Z
            custom_metadata:
                None
            deletion_time:
            destroyed:
                False
            version:
                1

suggesting that the external vault pillar rendering occurs until first failure.

Expected behavior
I would expect the pillar rendering to continue over forbidden paths to still obtain configured secret paths for which the minion does have access to. As-is, external vault pillar doesn't seem practical except for use cases where all minions have policy access to all configured external pillar secret paths.

Versions Report

# salt --versions-report
Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Nov 26 2021, 20:14:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-97-generic
        system: Linux
       version: Ubuntu 20.04 focal

[OrangeDog]

All pillar backends work the same way. A single syntax error in an sls will fail pillar compilation for that minion.

[MKLeb]

I was able to reproduce this, and have a fix for it. PR will be up once I get the time to write tests for it.