Configuring automatic shutdown and kernel culling #306

cfecherolle · 2019-07-15T09:08:40Z

Hello,

First of all, thank you for the amazing tool that is Voila, very useful and well-maintained! :)

I came here today because I have some issues regarding configuration for a standalone notebook launch.

I'm trying to edit my Voila config to set kernel culling (with MappingKernelManager.cull_idle_timeout and MappingKernelManager.cull_interval) and NotebookApp.shutdown_no_activity_timeout when there is no activity on my server.

Unfortunately, I don't understand which config file is used, I've tried running the voila command using --Voila.log_level=DEBUG but it only displays tried folders, not the config file(s) which are actually used. The docs are sparse about this topic and I tried computing info from jupyter, notebook, and voila docs but it still doesn't work.

Theoretically, I could setup a voila.json file with content such as:

{
  "NotebookApp": {
    "nbserver_extensions": {
      "voila.server_extension": true
    },
    "shutdown_no_activity_timeout": 3600
  },
  "MappingKernelManager": {
    "cull_idle_timeout": 1200,
    "cull_interval": 120
  }
}

And it would set things up just fine.

I tried various places and config files:

Creating ~/.jupyter/voila.json
Editing /usr/local/etc/jupyter/jupyter_notebook_config.d/voila.json
Editing some jupyter global config files (jupyter_notebook_config.py) in various places
To no avail.

Are these timeout and culling parameters supposed to log something when used with Voila? Did you manage to make them work? It has become frustrating to randomly create and edit config files without knowing what to set.

Thanks in advance for your help!

Have a nice day,
Cécile

The text was updated successfully, but these errors were encountered:

jtpio · 2019-07-15T09:34:09Z

Thanks @cfecherolle for this detailed report!

What does the command jupyter --paths return? (https://jupyter.readthedocs.io/en/latest/projects/jupyter-directories.html)

Voila should be able to pick up the configuration file located at ~/.jupyter/voila.json by default.
Alternatively you can try putting the voila.json file in the current working directory (where voila is started) as it should also be picked up.

When culling is enabled, there will be some logs showing when a kernel is culled to due inactivity:

[Voila] Kernel started: b8a274d4-6460-435d-90c9-40f96344b8de
[Voila] Culling kernels with idle durations > 10 seconds at 10 second intervals ...
...
[Voila] WARNING | Culling 'idle' kernel 'python3' (b8a274d4-6460-435d-90c9-40f96344b8de) with 0 connections due to 17 seconds of inactivity.

cfecherolle · 2019-07-15T10:06:19Z

Thank you for your quick reply!

jupyter --paths returns the following:

config:
    /home/jovyan/.jupyter
    /usr/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/jovyan/.local/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/jovyan/.local/share/jupyter/runtime

You were right about ~/.jupyter/voila.json being used! My mistake was, as I'm running voila in a docker container, I was starting a new voila process on a different port while being already connected to the container (to test the config), but it wasn't working.
To take the config changes into account I have to restart the whole container but well, why not. As long as it works.

Apart from this unrelated Docker stuff, I managed to make Voila use the right voila.json file thanks to your info but now I get:

HTTPServerRequest(protocol='http', host='localhost:8899', method='GET', uri='/', version='HTTP/1.1', remote_ip='172.17.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'notary'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'db_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'data_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.7/dist-packages/voila/handler.py", line 47, in get
    model = self.contents_manager.get(path=notebook_path)
  File "/usr/local/lib/python3.7/dist-packages/jupyter_server/services/contents/filemanager.py", line 387, in get
    model = self._notebook_model(path, content=content)
  File "/usr/local/lib/python3.7/dist-packages/jupyter_server/services/contents/filemanager.py", line 346, in _notebook_model
    self.mark_trusted_cells(nb, path)
  File "/usr/local/lib/python3.7/dist-packages/jupyter_server/services/contents/manager.py", line 503, in mark_trusted_cells
    trusted = self.notary.check_signature(nb)
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/usr/local/lib/python3.7/dist-packages/jupyter_server/services/contents/manager.py", line 65, in _notary_default
    return sign.NotebookNotary(parent=self)
  File "/usr/local/lib/python3.7/dist-packages/nbformat/sign.py", line 392, in __init__
    self.store = self.store_factory()
  File "/usr/local/lib/python3.7/dist-packages/nbformat/sign.py", line 340, in factory
    return SQLiteSignatureStore(self.db_file)
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/usr/local/lib/python3.7/dist-packages/nbformat/sign.py", line 351, in _db_file_default
    if not self.data_dir:
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/usr/local/lib/python3.7/dist-packages/nbformat/sign.py", line 327, in _data_dir_default
    app.initialize(argv=[])
  File "</usr/local/lib/python3.7/dist-packages/decorator.py:decorator-gen-6>", line 2, in initialize
  File "/usr/local/lib/python3.7/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/jupyter_core/application.py", line 243, in initialize
    self.migrate_config()
  File "/usr/local/lib/python3.7/dist-packages/jupyter_core/application.py", line 168, in migrate_config
    migrate()
  File "/usr/local/lib/python3.7/dist-packages/jupyter_core/migrate.py", line 247, in migrate
    with open(os.path.join(env['jupyter_config'], 'migrated'), 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter/migrated'

What is this migrated directory (non-existing on my machine) and what does it have to do with the whole thing? I'm not sure. 🤔

jtpio · 2019-07-15T10:29:50Z

The Jupyter paths look good.

To take the config changes into account I have to restart the whole container but well, why not. As long as it works.

Yes that's a bit cumbersome, but that's because the files are loaded at startup and I suppose voila is started as PID 1 in the container.

What is this migrated directory (non-existing on my machine) and what does it have to do with the whole thing?

Is the Docker image you are using based on an existing one from the jupyter docker stacks? It looks more like a permission issue.

Also is voila used as standalone app or as a server extension?

cfecherolle · 2019-07-15T10:50:34Z

I see! Yes indeed, Voila is started as PID 1.
The image I'm using is custom-made so no, it is not based on an existing Jupyter docker stack image.

I created a ~/.jupyter/migrated empty directory then restarted the container, and I no longer have the "PermissionError" from above: my notebook is running and the kernel culling message shows up as expected. Making some progress! 🎉

Even without knowing the role of this directory, I'll just add the mkdir command to the image initialization to bypass the error.

Culling seems fine now, but I still don't know how to make the server shutdown after some inactivity, no logs about this specific property and no sign of auto-shutdown when I test it. Does Voila support this in standalone/Tornado mode?

EDIT: I'm using Voila in standalone mode.

jtpio · 2019-07-15T11:20:54Z

I believe the migrated file was added to handle the migration from .ipython to .jupyter: jupyter/jupyter_core#25

In your Docker image, is voila started from a specific user or from the jovyan user? The base-notebook Dockerfile should have some hints on how to fix permissions.

shutdown_no_activity_timeout would work when using voila as a server extension, but not in standalone mode at the moment.
There is however ongoing work to make voila as a Jupyter Server ExtensionApp, which should add support for this option.

cfecherolle · 2019-07-15T12:51:04Z

Voila is started from the jovyan user 😄

Ok, we were pretty much set on standalone mode because we want to keep things as light as possible, so we will try and build some sort of inactivity purge with a Shell script wrapper to start Voila, maybe something like that.

I'll keep an eye on the ExtensionApp roadmap to see how it goes, but we might as well find a workaround for now 😉

Thanks for your help!

timkpaine · 2019-07-15T13:00:48Z

+1 for @Zsailer

Zsailer · 2019-07-25T16:04:48Z

Yes, once the ExtensionApp PR lands, shutdown_no_activity_timeout will work in standalone mode (since this PR actually makes Voila a server extension always).

jtpio · 2019-10-02T10:45:21Z

Closing as answered.

@cfecherolle if you have any more question, feel free to post a comment here or open a new issue!

cfecherolle · 2019-10-02T12:24:13Z

Thank you ! Le mer. 2 oct. 2019 à 12:45, Jeremy Tuloup <notifications@github.com> a écrit :

…

Closed #306 <#306>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#306?email_source=notifications&email_token=AAYFZSFFC7WODCGM2I5HSJ3QMR3ULA5CNFSM4IDUWEDKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOT67YOLQ#event-2680129326>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYFZSBPHFZ4MZK36O33LDLQMR3ULANCNFSM4IDUWEDA> .

jtpio closed this as completed Oct 2, 2019

jtpio mentioned this issue Nov 15, 2019

Close preview tab in Jupyterlab doesn't shut down corresponding kernel #479

Open

jtpio mentioned this issue Jan 27, 2020

Add documentation about culling idle kernels #527

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuring automatic shutdown and kernel culling #306

Configuring automatic shutdown and kernel culling #306

cfecherolle commented Jul 15, 2019 •

edited

Loading

jtpio commented Jul 15, 2019

cfecherolle commented Jul 15, 2019

jtpio commented Jul 15, 2019

cfecherolle commented Jul 15, 2019 •

edited

Loading

jtpio commented Jul 15, 2019 •

edited

Loading

cfecherolle commented Jul 15, 2019

timkpaine commented Jul 15, 2019 •

edited

Loading

Zsailer commented Jul 25, 2019

jtpio commented Oct 2, 2019

cfecherolle commented Oct 2, 2019 via email

Configuring automatic shutdown and kernel culling #306

Configuring automatic shutdown and kernel culling #306

Comments

cfecherolle commented Jul 15, 2019 • edited Loading

jtpio commented Jul 15, 2019

cfecherolle commented Jul 15, 2019

jtpio commented Jul 15, 2019

cfecherolle commented Jul 15, 2019 • edited Loading

jtpio commented Jul 15, 2019 • edited Loading

cfecherolle commented Jul 15, 2019

timkpaine commented Jul 15, 2019 • edited Loading

Zsailer commented Jul 25, 2019

jtpio commented Oct 2, 2019

cfecherolle commented Oct 2, 2019 via email

cfecherolle commented Jul 15, 2019 •

edited

Loading

cfecherolle commented Jul 15, 2019 •

edited

Loading

jtpio commented Jul 15, 2019 •

edited

Loading

timkpaine commented Jul 15, 2019 •

edited

Loading