Is there any way to avoid run_in_executor during the connection process?

[KoBOLL]

I don't want to use any configuration from system or user files, so I set config=None, and fill SSHClientConnectionOptions myself when calling asyncssh.create_connection().
However, there is a line inside asyncssh.connect() that calls run_in_executor:

new_options = cast(SSHClientConnectionOptions, await _run_in_executor(
        loop, SSHClientConnectionOptions, options, config=config,
        host=host, port=port, tunnel=tunnel, family=family,
        local_addr=local_addr, **kwargs)).

Is there any way to avoid executing part of the code in another thread?
It seems to me, that in my case it is not optimal.

[ronf]

The config file handling was changed to run in an executor to accommodate the "Match Exec" feature. This makes sure that the asyncio event loop keeps running for other connections if the executable run in the "Match Exec" blocks for any reason. The executor allows the new thread to block while everything else keeps on running. When the config parsing completes, the async task for the new connection then runs in the main event loop, and the thread used for the executor is returned to the pool, lasting only as long as it took to complete reading the config.

What's your concern here? In theory, the executor could be avoided if the config file didn't have a "Match Exec" in it, but that would required scanning the config file twice, which might be more expensive than the overhead of running the existing parsing in its own thread. In my experience, the overhead of using the executor is quite low, especially compared to some of the other operations like the cryptographic signing and encryption.

[KoBOLL]

Thanks for the detailed reply.

I'm writing a scanner to test vulnerabilities on a local network. So I need to apply the same config to different targets multiple times. Moreover, my config is not stored in a file or even in openssh format (as you correctly pointed out I don't use "Match Exec"), but is loaded from a database (one time) and SSHClientConnectionOptions values are set based on it.

I'm a bit concerned that every time a connection is established, I have no way to avoid processing the already configured SSHClientConnectionOptions in the thread. Assuming a large network is being scanned, there will be a lot of connections.

I realize the overhead is probably low, maybe it's just my perfectionism :)

[ronf]

You can definitely get a benefit from loading options in advance, particularly when it comes to reading any keys you might be using over and over. The same goes for other things like known_hosts. Even if functions like connect() create a new options object from the old one, things like loading keys are not repeated in such a case.

In the latest code in the "develop" branch, there's a new construct() method on the options objects which handles the run_in_executor(). It's possible I could be more selective there about what parts of the processing are run in the executor vs. the main thread, perhaps limiting it to just processing a config file. So, if config was passed as None, perhaps the run_in_executor() could be skipped. However, I'd want to see some evidence that there would be a noticeable performance gain by doing that before reworking this.

One of the things that makes this challenging is that run_in_executor() must be called from an async function, but all of the methods in the Options object (with the except of this new construct() method) are non-async. So, a fair amount of refactoring would be required to do some of the work in the main thread and other parts of it in an executor thread. In fact, most of the work is done in a prepare() method called from the constructor, so it's not simple to add async functions there.

My suggestion to you would be to focus on loading the keys in advance, or perhaps even loading up all the shared options into an Options object which can then be passed as via the "options" argument to all of the calls to connect(). Additional config like the destination host & port will be filled in by the connect() itself, but the bulk of the config settings should just be copied over as-is. If you don't need to read anything from an OpenSSH config file, setting the "config" argument explicitly to None will also help to speed things up slightly.