Voila process is leaking threads #896

roman-kouzmenko · 2021-05-26T13:40:19Z

Each time a notebook is accessed, voila is adding 3 threads that don't go away even as kernels are culled, eventually depleting all server resources.

Is anyone observing the same / has a fix?

roman-kouzmenko · 2021-05-27T21:23:09Z

Debugging this a bit, it seems these threads are client channels that might not be disposed of properly, below is the output of lsof|grep [VOILA_PID]|grep TCP:

voila     146700                jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700                jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700                jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700                jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165411         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165412         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165413         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165414         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165415         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165415         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165415         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165415         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165417         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165418         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165419         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165419         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165419         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165419         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165518         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165519         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165520         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165520         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165520         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165520         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)

clydebw · 2021-05-28T01:37:57Z

@roman-kouzmenko I think you're right in that these are client channels not getting cleaned up.

This problem of leaking threads was pointed out to me this morning, and I spent some time today trying to track this down as well. It seems two threads get created here:

voila/voila/handler.py

Line 166 in da39b76

kernel_id = await ensure_async(self.kernel_manager.start_kernel(

Although I think these stick around for the app's entire lifecycle, and so are not a problem. Three threads seem to get added each time the notebook is accessed here though, and these don't seem to be cleaned up:

voila/voila/handler.py

Line 179 in da39b76

await ensure_async(self.executor.kc.start_channels())

I'm going to keep looking, but any help from someone more familiar with the codebase would be much appreciated.

roman-kouzmenko · 2021-06-02T11:22:50Z

FYI, I mitigated the leak on kubernetes by:

creating a liveness probe counting the number of leaked threads and marking the voila container as bad above a certain threshold
increasing the grace termination period to a large number to avoid shutting active connections immediately
adding a preStop hook to count active kernels and wait on them to finish. If they don't terminate before the grace period in 2), they will get terminated automatically

dou-du · 2021-07-06T16:30:22Z

We are running Voila app with dokku. The CPUs are keeping eating up by Voila. We have to restart our dokku server every few days. I believe it is caused by this problem. Could anyone fix this issue? Thanks.

ltalirz · 2021-07-07T13:15:58Z

This is a longstanding issue, see the very related issues
#209
#479
#849

There starts to be some work on it, though, see PRs linked in #849 (comment)

giovannipizzi · 2021-08-25T11:04:43Z

I would like to report the same issue.
We are running voila inside docker.
After a few hours of the website being online, If I grep all Kernel started and Kernel shutdown from the log file, each kernel is shut down, either by the web socket being closed, or by culling.
However, if I do ps -eLf | grep voil[a] | wc -l I get >400 threads. The voila process (only 1) is constantly at 50% CPU usage, and the number of threads keeps increasing at every new connection to the website (that, with so many leaked threads, is unresponsive or very slow).

The only solution for us is currently to restart the docker container, but we are now at the point where we need to do it twice a day :-(

I wanted to inquire if there is any work being done to address this issue, and if you already have a timeline for solving the issue: we are relying on voila a lot, but this issue is making our service unreliable and we might need to change technology. (I hope this is not needed, as we like a lot the work being done on voila! :-) )

choldgraf · 2021-09-07T21:34:43Z

Just a quick thought here - I wonder why this is not something that the Voila Gallery has run into: https://voila-gallery.org/ (repo here: https://github.com/voila-gallery/gallery)

That is (I believe) a persistent JupyterHub serving Voila dashboards via Jupyter servers. If they are not running into this "runaway threads" issue, then perhaps there is a workaround?

I guess another question is whether this is an issue with Voila itself, or dependent on something deeper in Jupyter machinery.

edit: ahh I might be mistaken, I think the voila gallery may use BinderHub, in which case it's probably shutting down the whole pod after inactivity

dou-du · 2021-09-07T21:45:55Z

Just a quick thought here - I wonder why this is not something that the Voila Gallery has run into: https://voila-gallery.org/ (repo here: https://github.com/voila-gallery/gallery)

That is (I believe) a persistent JupyterHub serving Voila dashboards via Jupyter servers. If they are not running into this "runaway threads" issue, then perhaps there is a workaround?

I guess another question is whether this is an issue with Voila itself, or dependent on something deeper in Jupyter machinery.

Voila can be runned in two way. One is from the Jupyter server extension by calling "/voila/render/" in the URL. In the dokku
server, we use the voila program from the terminal (voila --template=osscar --enable_nbextensions=True mynotebook.ipynb). I think the problem comes from calling the voila program in the terminal.

choldgraf · 2021-09-07T21:47:48Z

Interesting - so this doesn't occur when running Voila via an already-running Jupyter Server, but it does occur when you run it via the command line?

dou-du · 2021-09-07T21:53:08Z

Interesting - so this doesn't occur when running Voila via an already-running Jupyter Server, but it does occur when you run it via the command line?

I am not sure about the voila Jupyter server extension. But I am calling the server extension a lot everyday and I did not experience any CPU overload related to voila. Our dokku apps problem is definitely from the voila program, since we
call it in the "Procfile" file.

jtpio · 2021-09-09T09:31:55Z

edit: ahh I might be mistaken, I think the voila gallery may use BinderHub, in which case it's probably shutting down the whole pod after inactivity

Yes that's right. It used to be a JupyterHub running on a single instance, but switched to using BinderHub as a backend so the gallery can be deployed as a static site on GitHub Pages: voila-dashboards/tljh-voila-gallery#83

martinRenou · 2021-09-17T14:13:56Z

Hopefully this is fixed now in Voila 0.2.14. Feel free to re-open the issue if you can still reproduce.

dou-du · 2021-09-27T14:02:46Z

We tested the Voila 0.2.14 for one app on our dokku server, the new Voila has a much better performance.
However, it seems the problem still exists. The usage of CPU increases from 0% to about 5% in one week.
Older version of Voila will make it to 120% in one week. Here is the data shown in the figure. One can
clearly see a slight linear increment of the CPU usage.

martinRenou · 2021-09-28T11:20:51Z

Do you have kernel culling enabled? If so what is your culling config?

dou-du · 2021-09-28T11:52:43Z

Do you have kernel culling enabled? If so what is your culling config?

Here is my voila configuration:

{
"VoilaConfiguration": {
"enable_nbextensions": true,
"template": "osscar",
"strip_sources": true
},
"VoilaExecutePreprocessor": {
"timeout": 180
},
"NotebookApp": {
"shutdown_no_activity_timeout": 60
},
"MappingKernelManager": {
"cull_idle_timeout": 900,
"cull_interval": 60,
"cull_busy": true
}
}

martinRenou · 2021-09-28T11:55:28Z

Would setting "cull_connected": true be possible for you? It might help in case users keep the tab open and never reboot their machine.

martinRenou · 2021-09-28T11:56:13Z

It's definitely possible we missed another issue on Voila though.

martinRenou mentioned this issue Sep 17, 2021

Cleanup executor resources after execution #969

Merged

martinRenou closed this as completed Sep 17, 2021

bsyouness mentioned this issue Jun 22, 2023

Do not clean up kernel resources after execution #1334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Voila process is leaking threads #896

Voila process is leaking threads #896

roman-kouzmenko commented May 26, 2021 •

edited

Loading

roman-kouzmenko commented May 27, 2021 •

edited

Loading

clydebw commented May 28, 2021 •

edited

Loading

roman-kouzmenko commented Jun 2, 2021 •

edited

Loading

dou-du commented Jul 6, 2021

ltalirz commented Jul 7, 2021

giovannipizzi commented Aug 25, 2021

choldgraf commented Sep 7, 2021 •

edited

Loading

dou-du commented Sep 7, 2021

choldgraf commented Sep 7, 2021

dou-du commented Sep 7, 2021 •

edited

Loading

jtpio commented Sep 9, 2021

martinRenou commented Sep 17, 2021

dou-du commented Sep 27, 2021

martinRenou commented Sep 28, 2021

dou-du commented Sep 28, 2021

martinRenou commented Sep 28, 2021

martinRenou commented Sep 28, 2021

Voila process is leaking threads #896

Voila process is leaking threads #896

Comments

roman-kouzmenko commented May 26, 2021 • edited Loading

roman-kouzmenko commented May 27, 2021 • edited Loading

clydebw commented May 28, 2021 • edited Loading

roman-kouzmenko commented Jun 2, 2021 • edited Loading

dou-du commented Jul 6, 2021

ltalirz commented Jul 7, 2021

giovannipizzi commented Aug 25, 2021

choldgraf commented Sep 7, 2021 • edited Loading

dou-du commented Sep 7, 2021

choldgraf commented Sep 7, 2021

dou-du commented Sep 7, 2021 • edited Loading

jtpio commented Sep 9, 2021

martinRenou commented Sep 17, 2021

dou-du commented Sep 27, 2021

martinRenou commented Sep 28, 2021

dou-du commented Sep 28, 2021

martinRenou commented Sep 28, 2021

martinRenou commented Sep 28, 2021

roman-kouzmenko commented May 26, 2021 •

edited

Loading

roman-kouzmenko commented May 27, 2021 •

edited

Loading

clydebw commented May 28, 2021 •

edited

Loading

roman-kouzmenko commented Jun 2, 2021 •

edited

Loading

choldgraf commented Sep 7, 2021 •

edited

Loading

dou-du commented Sep 7, 2021 •

edited

Loading