Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] x509_v2: ca_server cannot handle all minions #65469

Closed
tqre-mc opened this issue Oct 26, 2023 · 2 comments
Closed

[BUG] x509_v2: ca_server cannot handle all minions #65469

tqre-mc opened this issue Oct 26, 2023 · 2 comments
Labels
Bug broken, incorrect, or confusing behavior Duplicate Duplicate of another issue or PR - will be closed

Comments

@tqre-mc
Copy link

tqre-mc commented Oct 26, 2023

Description
CA server becomes unavailable when handling ~20 minions.

Salt-master gives the following error:

----------
          ID: minion_certificate
    Function: x509.certificate_managed
        Name: /etc/pki/<hostname from grains>.crt
      Result: False
     Comment: ca_server did not respond. Salt master must permit peers to call the sign_remote_certificate function.
     Started: 12:05:07.995662
    Duration: 15773.609 ms

Setup
The x509 minion configuration and master setup is identical to the documentation, with the exception that the certificate file names are based on hostnames in the state file:
(https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.x509_v2.html#x509-setup)

minion_certificate:
  x509.certificate_managed:
    - name: /etc/pki/{{ grains['host'] }}.crt
    - ca_server: test-ca
    - signing_policy: prod
    - private_key: /etc/pki/{{ grains['host'] }}.key
    - CN: {{ grains['host'] }}
    - O: "Company"
    - OU: "Test environment"
    - C: "Country"
    - ST: "State"
    - L: "City"
    - subjectAltName: "DNS:{{ grains['host'] }}.ourdomain.local"
    - days_valid: 360
    - days_remaining: 30

Please be as specific as possible and give set-up details.

  • VM/KVM
  • onedir packaging

Steps to Reproduce the behavior
Set up x509_v2 CA server + 20 minions to where certificates are distributed.

On the test-ca server I get the following errors:

Oct 26 12:05:23 test-ca salt-minion[544]: [ERROR   ] Timeout encountered while sending {'cmd': '_return', 'id': 'test-ca', 'success': True, 'return': {'data': {'signing_cert': b'<REDACTED>', 'O': 'Company', 'OU': 'Test environment', 'C': 'Country', 'ST': 'State', 'L': 'City', 'basicConstraints': 'critical CA:false', 'keyUsage': 'critical keyEncipherment digitalSignature', 'extendedKeyUsage': 'serverAuth, clientAuth', 'subjectKeyIdentifier': 'hash', 'authorityKeyIdentifier': 'keyid:always,issuer', 'days_valid': 360, 'copypath': '/etc/pki/issued_certs/', 'prepend_cn': True}, 'errors': []}, 'retcode': 0, 'jid': '20231026090508369687', 'fun': 'x509.sign_remote_certificate', 'fun_args': ['test', {}, True], 'user': 'root', '_stamp': '2023-10-26T09:05:08.640470', 'nonce': 'e994b334c150499bacbfc9a8ef85823f'} request
Oct 26 12:05:23 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950cb93d60> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:23 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:23 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:23 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:23 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:23 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:23 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:23 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:23 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:23 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:23 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:23 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in ZMQStream callback
Oct 26 12:05:23 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:23 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:23 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:23 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:23 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:23 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in zmqstream callback
Oct 26 12:05:23 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:23 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:23 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:23 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:23 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:23 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:23 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:23 test-ca salt-minion[544]: [ERROR   ] Exception in callback (<zmq.Socket(zmq.REQ) at 0x7f950e7bb3a0>, <function wrap.<locals>.null_wrapper at 0x7f950e3148b0>)
Oct 26 12:05:23 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 890, in start
Oct 26 12:05:23 test-ca salt-minion[544]:     handler_func(fd_obj, events)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:23 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:23 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:23 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:23 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:23 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:23 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:23 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:23 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950e333d30> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950cf60e80> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950eb09900> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950cb92cb0> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in ZMQStream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in zmqstream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Exception in callback (<zmq.Socket(zmq.REQ) at 0x7f950e7bb3a0>, <function wrap.<locals>.null_wrapper at 0x7f950e3148b0>)
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 890, in start
Oct 26 12:05:24 test-ca salt-minion[544]:     handler_func(fd_obj, events)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950cb92dd0> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in ZMQStream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in zmqstream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Exception in callback (<zmq.Socket(zmq.REQ) at 0x7f950e7bb3a0>, <function wrap.<locals>.null_wrapper at 0x7f950e3148b0>)
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 890, in start
Oct 26 12:05:24 test-ca salt-minion[544]:     handler_func(fd_obj, events)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f950cbea290> exception was never retrieved: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2727, in handle_event
Oct 26 12:05:24 test-ca salt-minion[544]:     yield _minion.req_channel.send(
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.throw(*exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     ret = yield _do_transfer()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     value = future.result()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Oct 26 12:05:24 test-ca salt-minion[544]:     raise_exc_info(self._exc_info)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "<string>", line 4, in raise_exc_info
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Oct 26 12:05:24 test-ca salt-minion[544]:     yielded = self.gen.send(value)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 284, in _do_transfer
Oct 26 12:05:24 test-ca salt-minion[544]:     data = self.auth.crypticle.loads(data, raw, nonce=nonce)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1531, in loads
Oct 26 12:05:24 test-ca salt-minion[544]:     raise SaltClientError("Nonce verification error")
Oct 26 12:05:24 test-ca salt-minion[544]: salt.exceptions.SaltClientError: Nonce verification error
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in ZMQStream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Uncaught exception in zmqstream callback
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'
Oct 26 12:05:24 test-ca salt-minion[544]: [ERROR   ] Exception in callback (<zmq.Socket(zmq.REQ) at 0x7f950e7bb3a0>, <function wrap.<locals>.null_wrapper at 0x7f950e3148b0>)
Oct 26 12:05:24 test-ca salt-minion[544]: Traceback (most recent call last):
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 890, in start
Oct 26 12:05:24 test-ca salt-minion[544]:     handler_func(fd_obj, events)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 577, in _handle_events
Oct 26 12:05:24 test-ca salt-minion[544]:     self._handle_recv()
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 606, in _handle_recv
Oct 26 12:05:24 test-ca salt-minion[544]:     self._run_callback(callback, msg)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 556, in _run_callback
Oct 26 12:05:24 test-ca salt-minion[544]:     callback(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Oct 26 12:05:24 test-ca salt-minion[544]:     return fn(*args, **kwargs)
Oct 26 12:05:24 test-ca salt-minion[544]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 628, in handle_reply
Oct 26 12:05:24 test-ca salt-minion[544]:     future.set_result(data)
Oct 26 12:05:24 test-ca salt-minion[544]: AttributeError: 'NoneType' object has no attribute 'set_result'

Expected behavior
All minions get their certificates without errors

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
SALT-MASTER:
Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-87-generic
        system: Linux
       version: Ubuntu 22.04.3 jammy

SALT-MINIONS
Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-84-generic
        system: Linux
       version: Ubuntu 22.04.3 jammy
@tqre-mc tqre-mc added Bug broken, incorrect, or confusing behavior needs-triage labels Oct 26, 2023
@lkubb
Copy link
Contributor

lkubb commented Oct 26, 2023

This doesn't seem to be an issue with the x509_v2 modules, but rather the transport, possibly a variant of #65114 caused by high load.

@dwoz
Copy link
Contributor

dwoz commented Dec 18, 2023

Duplicate of #54114 and should be resolved in 3006.5

@dwoz dwoz closed this as completed Dec 18, 2023
@dwoz dwoz added Duplicate Duplicate of another issue or PR - will be closed and removed needs-triage labels Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug broken, incorrect, or confusing behavior Duplicate Duplicate of another issue or PR - will be closed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dwoz @lkubb @tqre-mc