Need ability to add custom plugin API namespace that's directly connected to a backend implementation

[jcortell68]

Feature Description:

The following documentation describes how a Theia extension can extend the plugin API with new API namespaces.
https://github.com/eclipse-theia/theia/blob/master/packages/plugin-ext/doc/how-to-add-new-plugin-namespace.md

And it works great. But...it's limited in that the new API can't be directly supported by the Theia extensions' backend. The API connects the plugin running in the plugin host node process to the Theia extension's frontend. And if the API ultimately is about driving logic in the frontend, then that works out perfectly. BTW, this configuration is the only one shown in the diagram here.

But imagine the Theia extension having a backend service that it wants to expose to plugins. It should be able to define an API namespace that connects the plugin code to the backend service in the Theia extension. Today, you can't make that direct connection. You would have to go through an intermediary in the frontend. That is inefficient on multiple fronts.

Basically, a Theia extension's backend should be able to bind a MainPluginApiProvider and have it work in the same way it does when the frontend binds that. But MainPluginApiProvider is only injected by Theia frontend code (in @theia/plugin-ext).

Here is an example of a Theia app that contributes a working API namespace wired to the frontend, but then fails to contribute API namespace from the backend. See commit message and comments in the code. In particular, look at the diff between the head commit and the previous one.

git clone -b api_namespace_backend https://github.com/jcortell68/theiasandbox
cd theiasandbox
yarn
yarn start:electron

What are you looking for? Notice the following doesn't show up in stdout when you launch the app.

HelloWorldBakendMainPluginApiProvider.initialize() called

That absence proves that the Theia extension has no chance of contributing an API namespace implemented in its backend. You can see that the normal (wired to frontend) API namespace works fine by invoking the Hello World command that the extension contributes. The message string you see was obtained by exercising the custom plugin API.

[tsmaeder]

@jcortell68 Please correct me if I'm paraphrasing this issue incorrectly, but you're asking for a way to extend the Theia back-end at install/run time, as opposed to the compile-time extensibility we have now, correct?
One interesting aspect in this is life-cycle: the plugin host process lives inside a frontend-connection scope, so if there is a back-end-scoped plugin host, it will have a totally different life-cycle from the regular plug-in engine. In the current system, the back-end is not really involved with plugins, it just mediates the connection between the front-end and the plugin host.
Unfortunately, there is the notion of a "built-in" API namespace, so we would have to do some clean-up if we wanted to reuse the plugin-api mechanism for the back-end API use case. I did a proof of concept a while back: #8180.
Technically, I think we could just run an plugin host process for the back-end and have a "main"-side client running in the back-end process There is nothing in principle that says the "main"-side needs to be in a browser. I'd run the plugins out of process for the same reasons of isolation as in the current case. Probably, this would require quite a bit of refactoring to remove certain assumptions that are built into the code right now. Also, we'd have to figure out how to deploy plugins to this back-end host and whether there are declarative contributions, etc., etc. What I'm getting at is that this looks like a bunch of work ;-)
Personally, I think having run-time extensibility for the back end would be a great addition to our story. Since this seems a major undertaking, the next step would be to bring the project to the attention of the wider community and then to find some adopters willing to fund the work (nudge nudge, wink wink, knowhatimean ;-))

[jcortell68]

but you're asking for a way to extend the Theia back-end at install/run time, as opposed to the compile-time extensibility we have now, correct?

A way to extend and/or utilize the backend of a Theia app via something that can be deployed and installed at runtime, same as how Theia supports VSIX files. We're not asking for any specific backend-to-backend API from Theia, though. We want to use this new capability to utilize/extend product specific backend services. I.e., I as a Theia product developer will develop a Theia extension that contributes the backend-to-backend API namespaces. Whether Theia the platform at some point wants to add any backend-to-backend API, that's up to you guys.

so if there is a back-end-scoped plugin host, it will have a totally different life-cycle from the regular plug-in engine.

And that would be perfectly fine :-)

Unfortunately, there is the notion of a "built-in" API namespace, so we would have to do some clean-up if we wanted to reuse the plugin-api mechanism for the back-end API use case. I did a proof of concept a while back: #8180.

Yep. Basically, if you want any plugin API in your product, you get the Theia API in the mix. I.e., if I wanted to build a Theia product that supports only product-specific plugins that have access only to product-specific API, that's not possible today. The plugin mechanism and the vscode/theia extension API are bundled into the same Theia extension (@theia/plugin-ext). So indeed, if you end up using the same infrastructure to support backend-to-backend plugins, then that would have to be broken up.

Technically, I think we could just run an plugin host process for the back-end and have a "main"-side client running in the back-end process There is nothing in principle that says the "main"-side needs to be in a browser. I'd run the plugins out of process for the same reasons of isolation as in the current case.

That's what I was hoping for. That we could use the same programming model for adding backend-to-backed API namespaces. I.e., that this guidance would equally apply for backend-to-backend API, at least mostly.

As for having a dedicated plugin host process for all this, there I wonder if we couldn't avoid that and save apps the added runtime overhead of another Node.js process (apps that make use of this new capability, that is). At least theoretically, a single Node.js process should be able to communicate with both the browser and with Theia's backend Node.js via RPC. It's doing the former today; we just need it to do the latter as well. Naturally, we'd need separate lifecycles hooks for each.

[martin-fleck-at]

Closed with #13138