[BUG] salt-ssh error reporting/handling of remote errors/misbehavior is optimistic to a fault

[lkubb]

Description
There are several situations in which salt-ssh deviates from regular salt-call regarding error reporting/handling. This is intended as a collective issue/overview since they are related.

1. state.* wrappers treat an exception during the state run on the target as success.
2. When calling execution modules on the target using the FunctionWrapper, remote errors/exceptions are generally swallowed. The function seemingly returns the error dict now.
   .a This results in undefined behavior in wrapper modules and rendering templates which call non-wrapped modules that raise an exception.
   .b This also results in mine functions that cause remote exceptions to happily pass along the error dict as the function's return without reporting a non-zero exitcode.
3. When calling execution modules on the target using the FunctionWrapper, local deserialization errors are generally swallowed. The function seemingly returns the error dict now.
   .a see 2.a
   .b see 2.b
   .c This might be related to salt-ssh return exit code 0 if salt-call fails to run #50727 if still relevant.
4. When calling execution modules on the target using the FunctionWrapper and the remote does not return a dict containing "return", an empty dict is returned instead.
   .a see 2.a
   .b see 2.b

Setup
irrelevant

Steps to Reproduce the behavior
See linked issues.

(1) Is a bit hard to reproduce since it requires Salt crashing on the remote, but can be easily inferred from the code:

salt/salt/client/ssh/wrapper/state.py

Lines 723 to 739 in b0bf7eb

	stdout, stderr, _ = single.cmd_block()
	# Clean up our tar
	try:
	os.remove(trans_tar)
	except OSError:
	pass
	# Read in the JSON data and return the data structure
	try:
	return salt.utils.json.loads(stdout)
	except Exception as e: # pylint: disable=broad-except
	log.error("JSON Render failed for: %s\n%s", stdout, stderr)
	log.error(str(e))
	# If for some reason the json load fails, return the stdout
	return stdout

(2.a) #52452 (comment)
(2.b)

salt/salt/client/ssh/__init__.py

Line 1282 in b0bf7eb

ret = salt.utils.json.dumps({"local": {"return": result}})

(2/3/4)

salt/salt/client/ssh/wrapper/__init__.py

Lines 129 to 141 in b0bf7eb

	try:
	ret = salt.utils.json.loads(stdout)
	if len(ret) < 2 and "local" in ret:
	ret = ret["local"]
	ret = ret.get("return", {})
	except ValueError:
	ret = {
	"_error": "Failed to return clean data",
	"stderr": stderr,
	"stdout": stdout,
	"retcode": retcode,
	}
	return ret

(2/3/4) might come into play for any wfunc call

salt/salt/client/ssh/__init__.py

Lines 1137 to 1142 in b0bf7eb

	opts_pkg = pre_wrapper["test.opts_pkg"]() # pylint: disable=E1102
	if "_error" in opts_pkg:
	# Refresh failed
	retcode = opts_pkg["retcode"]
	ret = salt.utils.json.dumps({"local": opts_pkg})
	return ret, retcode

Expected behavior
Behave as similarly as possible to salt-call. Treat exceptions as exceptions, report them as such and especially do not pass off a truthy value as the return of a function that clearly failed.

Versions Report
Still present in current master.

Additional context
Related:
QubesOS/qubes-issues#8146 (1)
#50727 (maybe 3)
#52452 (2.a)

#64514 (already fixed yesterday)