Python virtual env support

[simark]

This issue is to track support of virtualenv when writing Python code. The scope of the feature (at least in my mind) is to provide a way for the user to specify the virtual environment they intend to run their code in. This will influence how the code is analyzed by the language server.

See the original discussion here:

4ba4e65#r30632208

cc @AlexTugarev @marechal-p

[simark]

Replying to @AlexTugarev's post in the thread linked above:

to decouple the setting of the VIRTUAL_ENV env var from the PythonContribution. though we definitely need a mechanism to restart the contribution/LS after changing the virtualenv. how this is to be triggered remains a bit unclear. though we might have a component to trigger restarts. let's think about that.

This is how I see it: the user will have some UI to change the currently selected virtual environment one way or another. Maybe it will be as simple as a preference, or more complete UI like the C++ build configurations. Changing the selected virtual environment should trigger a restart of the language server, where the new language server will be spawned with a new value of VIRTUAL_ENV. There is an example of language server being restarted in #2907. I still need to confirm that the same strategy would work here.

About the language server being launched in a container thing, I am not sure what it changes. If your backend contributes a different python contribution that spawns the python language server differently (in a container), then it will just need to honor the virtualEnv parameter that is sent by the frontend, and inject the VIRTUAL_ENV environment variable in the container. Or maybe I don't understand what you mean.

[simark]

So as I feared, let's say the Python language server was created initially with the /services/languages/python?virtualenv=/someplace, calling .restart() on the language client after the virtualenv preference was changed to /someotherplace won't initiate a new channel to be created with the path /services/languages/python?virtualenv=/someotherplace :(

If we want to continue on this path, we would need to figure out how to completely tear down the rpc channel and create a new one on a different path.

An alternative path would instead be to delay the start of the language server when opening the /services/languages/python channel and pass the parameters in the first json-rpc message, which would be handled specially by the backend.

[AlexTugarev]

Thanks @simark for the detailed explanation. As far as I understand your requirement, it goes well with what I was thinking about the situation, where the pyls/python is started on a remote machine.

Could you please test your requirement with the following concept?

1. changes to the VIRTUAL_ENV Env Var can be made via a terminal session or a slick UI (TBD)
   (while a python project form source might include a script to prepare the virtualenv, I'm not sure how this is supposed to work in any UI with an empty project.)
2. when starting the PythonContribution we can have a strategy to get and inject the value of VIRTUAL_ENV.
   a. if pyls is directly started as child process, we want to prepend that to the environment in order to be picked up by jedi.
   b. if started remotely, it might even depend on the integration, so we maybe still need to inject this, but differently; or maybe there is nothing to do at all.
3. there is a restart mechanism for the python LS (which maybe is useful for LanguageContributions in general.)
4. there is an event fired when VIRTUAL_ENV value has changed.
   (this is quite tricky for the remote session, but not impossible.)
5. the python LS will be restarted on change of the VIRTUAL_ENV value.

In general, I think we can accomplish that without touching the language client. Managing of environments can be done in completely different ways, therefore I prefer to separate it from the languages service.

[AlexTugarev]

An alternative path would instead be to delay the start of the language server ...

That would be exactly the effect, if we would have an python environment manager in place, which would read the VIRTUAL_ENV value and inject it once called in PythonContribution.start.

[simark]

Could you please test your requirement with the following concept?

I'm not 100% sure we are totally in sync, may it's me missing the point...

1. changes to the VIRTUAL_ENV Env Var can be made via a terminal session or a slick UI (TBD)
   (while a python project form source might include a script to prepare the virtualenv, I'm not sure how this is supposed to work in any UI with an empty project.)

I am not sure I understand how, if as a user I set VIRTUAL_ENV in a terminal session, it could be picked up by Theia. Unless that terminal session is the one from which I also start the backend? But I don't think that's what you meant.

2. when starting the PythonContribution we can have a strategy to get and inject the value of VIRTUAL_ENV.
   a. if pyls is directly started as child process, we want to prepend that to the environment in order to be picked up by jedi.
   b. if started remotely, it might even depend on the integration, so we maybe still need to inject this, but differently; or maybe there is nothing to do at all.

Well, if the user wishes to use a virtualenv, you will have to find a way to set that VIRTUAL_ENV variable even if the process runs remotely. Also, if the process runs remotely, it will need to somehow have access to the same filesystem, with the project at the same path as where Theia runs? Otherwise we would need some kind of path translation mechanism I guess.

3. there is a restart mechanism for the python LS (which maybe is useful for LanguageContributions in general.)

There is already a way to restart the language server (not very well tested though). Calling BaseLanguageClientContribution.restart should end up restarting the language server.

4. there is an event fired when VIRTUAL_ENV value has changed.
   (this is quite tricky for the remote session, but not impossible.)

The way I see it, the change in the virtual env to use always comes from the user (the virtualenv path is not something the project sources dictate, it's something the user decides). To the change in the virtual env to use will come from Theia, either by the user changing a preference or choosing a path in a pre-defined list.

5. the python LS will be restarted on change of the VIRTUAL_ENV value.

I see it the other way around: when the user decides to use a different virtual env, we restart the language server with a different VIRTUAL_ENV env var value (we don't react to something changing the value of the VIRTUAL_ENV env var.

[adrienrenaud]

I would love to see this feature. Support for Conda environment would also be great!

[tsmaeder]

We no longer implement any LSP functionality. Closing.