Refactor template system to be easily extensible and voila compatible #1056

maartenbreddels · 2019-06-28T09:54:30Z

This is the followup of what was discussed in #1028, planned for voila in voila-dashboards/voila#208.

The basic idea is that templates are a directory in either:
$prefix/nbconvert/templates/<template_name> where $prefix is determined by jupyter_paths.
On OSX it would be:

	/Users/maartenbreddels/Library/Jupyter/nbconvert/templates/default
	/Users/maartenbreddels/miniconda3/envs/voila/share/jupyter/nbconvert/templates/default
	/usr/local/share/jupyter/nbconvert/templates/default
	/usr/share/jupyter/nbconvert/templates/default

With this PR, #1005 would simply be a directory share/jupyter/nbconvert/templates/lab and now the old full.tpl is a template called 'notebook'.

In combination with voila-dashboards/voila#208 we can make all nbconvert template compatible by only overriding the {% block ipywidgets %}, and instead of inlining CSS, we can request it from the server.

Loose ends

the latex templates clutter the default directory a bit, maybe we can put these in different template directories. `
The meaning ofTemplateExporter.template_file changed a bit, it's cannot be an absolute path anymore, it now refers to the filename in the search path, resolved by jinja2 only. (also the reason unit tests fail).
We discussed using index.j2.html instead of filenames for templates, I think without the .j2 is looks cleaner.
Some systems may want to list template (eg voila), filtered by what kind of exporter is supported, we could list that in the conf.json file.
the html exporter should maybe use the notebook template by default, since now it's using the old 'basic.tpl' template.

MSeal · 2019-06-28T16:56:43Z

@t-makaro should probably help review this as he's been more involved on the template side of things recently.

Depends on jupyter/nbconvert#1056 Depends on voila-dashboards/voila#208

MSeal · 2019-07-26T16:49:18Z

I need to do some testing this weekend to help make sure everything is in working order, but it looks like all the changes were what we discussed. I'm not sure if we agreed for this change to be in 6.0 release or in 5.x path, if you remember?

The test suite is quite angry and will need some attention for the new patterns.

MSeal · 2019-07-26T16:49:41Z

the latex templates clutter the default directory a bit, maybe we can put these in different template directories. `

Can also be a follow-up PR to avoid making this any larger

MSeal · 2019-07-26T16:50:47Z

Some systems may want to list template (eg voila), filtered by what kind of exporter is supported, we could list that in the conf.json file.

Probably needs a separate issue to discuss. We may want this to be a dynamic calculation.

MSeal · 2019-07-26T16:54:08Z

the html exporter should maybe use the notebook template by default, since now it's using the old 'basic.tpl' template.

Need to look at the differences here, but I think there was consensus to have exports be the classic notebook format by default and lab format to be an opt-in pattern, if that's what's being referenced by this question.

MSeal · 2019-07-28T17:49:31Z

I need to do some testing this weekend to help make sure everything is in working order, but it looks like all the changes were what we discussed. I'm not sure if we agreed for this change to be in 6.0 release or in 5.x path, if you remember?

The test suite is quite angry and will need some attention for the new patterns.

Missed the 6.0 tag on the PR makes sense. Let me know if you need any help with figuring out tests later on.

maartenbreddels · 2019-07-31T12:46:00Z

~~I changed the target branch to 6.x, but CI does not run on this, can this be enabled? (works now)~~

Need to look at the differences here, but I think there was consensus to have exports be the classic notebook format by default and lab format to be an opt-in pattern, if that's what's being referenced by this question.

I've changed it, such that lab is a template: $ nbconvert --template lab ...

The old system of using a template file still exists, but now is available as --template-file=,

maartenbreddels · 2019-07-31T12:51:12Z

I've seem to have hit a bug in traitlets however, like voila, I want this to work:

$ voila     --template gridstack --theme dark basics.ipynb
$ nbconvert --template gridstack --theme dark --to html --execute basics.ipynb

Such that arguments --template and --theme have the same meaning.

However, it seems the HTMLExporters does not pick this up, the alias --theme gets translated to SlidesExporter.theme (!) (from a debug session):

(Pdb) self.config  # <-- this is HTMLExporter.config
{'NbConvertBase': {'display_data_priority': ['application/vnd.jupyter.widget-state+json', 'application/vnd.jupyter.widget-view+json', 'application/javascript', 'text/html', 'text/markdown', 'image/svg+xml', 'text/latex', 'image/png', 'image/jpeg', 'text/plain']}, 'CSSHTMLHeaderPreprocessor': {'enabled': True}, 'HighlightMagicsPreprocessor': {'enabled': True}, 'RegexRemovePreprocessor': {'enabled': True}, 'TagRemovePreprocessor': {'enabled': True}, 'NbConvertApp': {'export_format': 'html'}, 'TemplateExporter': {'template_name': 'gridstack'}, 'SlidesExporter': {'theme': 'dark'}, 'ExecutePreprocessor': {'enabled': True}}

Maybe I am missing something, if not we should open an issue for traitlets.

SylvainCorlay · 2019-07-31T14:26:07Z

nbconvert/exporters/tests/test_html.py

@@ -74,7 +74,7 @@ def test_prompt_number(self):
                }
            }
        )
-        exporter = HTMLExporter(config=no_prompt_conf, template_file='full')    
+        exporter = HTMLExporter(config=no_prompt_conf, template_name='notebook')


Maybe the name of the template should be classic instead of notebook

For now 'default' is the classic one, it may have been a good idea to have the default html template be a 'nothing' template, without css and all
The classic template would just add some css to the template, have some opinion on the dom structure etc. This way the lab template would not inherit from the classical template.

SylvainCorlay · 2019-07-31T14:34:05Z

share/jupyter/nbconvert/templates/reveal/index.html

@@ -1,5 +1,5 @@
-{%- extends 'basic.tpl' -%}


does default stand for the default html template?

yes, I think we can make reveal actually a normal template, just like we want with voila, with just some extra resources. So you would do --template reveal.

maartenbreddels · 2019-07-31T14:55:25Z

I don't understand check-manifest, I added:

+include share/jupyter/nbconvert/templates/default/static/style.css
+include share/jupyter/nbconvert/templates/lab/static
+include share/jupyter/nbconvert/templates/lab/static/index.css
+include share/jupyter/nbconvert/templates/lab/static/theme-dark.css

To MANIFEST.in, but it still gave a warning, so FTM I'm ignoring that directory.

t-makaro · 2019-07-31T16:50:22Z

I will take a deeper look at this (hopefully) tomorrow, but I do have a few things on my mind that I'd like to put out there.

Do in-memory templates still work? (Given that template_file and template_name are too different things, I believe the answer is yes.)
Can we have templates of the same name for different file types? Perhaps by having multiple index files within a directory.

template_name/
    index.j2.html
    index.j2.markdown
    index.j2.py

(looking at the code I believe the answer is yes).
3. I do think that template files should have .j2 within them.
4. Does template_path still add the cwd to the search path by default (the the user's ability to change it)? Again, I believe the answer is yes.
5. Can we add some sort of check to make absolute paths still work?
Currently, this line ensures that an absolute path is added to the template search paths if needed. I suppose that template_file can be used with an absolute path?
6. Can index.j2.* files still reference other *.j2.* files inside the same directory? Again, I believe the answer is yes.

@maartenbreddels

However, it seems the HTMLExporters does not pick this up, the alias --theme gets translated to SlidesExporter.theme (!) (from a debug session):

Perhaps related to #822?

maartenbreddels · 2019-08-01T07:33:43Z

Do in-memory templates still work? (Given that template_file and template_name are too different things, I believe the answer is yes.)

yes, convered in test

2. Can we have templates of the same name for different file types? Perhaps by having multiple index files within a directory.

yes

I do think that template files should have .j2 within them.

I'm 50/50 on it, maybe other may express their opinion, I'll leave it like this for the moment, we can always do a rename at the end.

Does template_path still add the cwd to the search path by default (the the user's ability to change it)? Again, I believe the answer is yes.

yes

Can we add some sort of check to make absolute paths still work?

covered in tests

Can index.j2.* files still reference other *.j2.* files inside the same directory? Again, I believe the answer is yes.

yes, in the same, and other directories.

maartenbreddels · 2019-08-01T07:37:28Z

After discussion with @SylvainCorlay, the plan is to:
move templates to their separate directory, e.g.:

templates/
          latex/
          markdown/
          python/
          classic/
          lab/

in conf.json, each template can list the mimetype it outputs to (so you can still have multiple mimetypes/formats in 1 template directory, e.g. latex+html). This makes it easier to list only templates of a specific mimetype.
Common template may still go into a directory (say base, or skeleton).

maartenbreddels · 2019-08-07T11:41:35Z

the suggested change are in, I think this is ready for review.

t-makaro · 2019-08-07T16:33:46Z

Why not rename article.tex to index.tex instead of creating a new file? I believe that the other files inside the latex template directory should be renamed to have .tex extensions instead of .tplx too. Also, the report.tex file could be it's own template.

t-makaro · 2019-08-07T16:50:32Z

In number #1076, I made it possible to use style_*.tplx as their own top level template (making article.tplx redundant). This means in nbconvert 5.6 one can do one of the following:

Jupyter nbconvert --to pdf notebook.ipynb --template=article.tplx
Jupyter nbconvert --to pdf notebook.ipynb --template=style_jupyter.tplx
Jupyter nbconvert --to pdf notebook.ipynb --template=style_ipython.tplx

where the first 2 are equivalent (and the extension can in theory be dropped).

My understand as that after this PR I would do:

Jupyter nbconvert --to pdf notebook.ipynb --template=latex
Jupyter nbconvert --to pdf notebook.ipynb --template-file=style_jupyter.tplx
Jupyter nbconvert --to pdf notebook.ipynb --template-file=style_ipython.tplx

This is where my concern is. It may seem arbitrary to a user to have to use --template-file. If I do:

Jupyter nbconvert --to pdf notebook.ipynb --template=style_jupyter.tplx

will it work? If not, I would refactor these latex templates a fair bit to make these style_*.tplx templates top level. Though I would be happy to do that in a separate PR.

Though I would like if we could detect if a user passed in a template_name with a file extension and use that directly as the template_file then.

MSeal · 2019-09-04T23:06:57Z

Ok rebasing 6.x with master added some conflict files unfortunately.

maartenbreddels · 2019-10-14T14:05:44Z

squash and rebased.
Note: this should be split in 2 commits before merking to give @SylvainCorlay credits for the lab template, but this was impossible to rebase otherwise.

MSeal · 2019-10-15T05:03:22Z

@maartenbreddels If you wish you can coauthor the rebase commit by adding https://help.github.com/en/articles/creating-a-commit-with-multiple-authors

I'm good to merge either way, I'll check on this tomorrow and merge it then either way. Thanks a ton for getting this prepped.

maartenbreddels · 2019-10-15T15:28:45Z

I added two last commits that were needed to make integration with voila smoother, which will make a review easier. I'll squash it all before committing.

@SylvainCorlay do you want to take another look?

The more visible change is in null.tpl/null.j2.
Before we only had thebody block, this was difficult to change, and needed lots of copy pasting. By having body subdivided in body_header/body_loop/body_footer, it's much easier to add something to the header and footer, and not the loop, or visa versa.

MSeal · 2019-10-15T15:52:58Z

I can do the squash on merge if you want, or we can leave a few minor commits hanging around.

Co-authored-by: Sylvain Corlay <sylvain.corlay@gmail.com>

MSeal · 2019-10-17T03:27:56Z

I'm gonna merge this -- we can make changes later if need be :D

choldgraf · 2019-10-17T06:32:57Z

✨✨✨✨✨💪💪💪💪🤩🤩🤩🤩🤩

maartenbreddels · 2019-10-17T08:15:24Z

Awesome!
Managed to get the following templates to work with nbconvert and voila

And the voila templates contains almost no code, except for the spinner in the lab template:

https://github.com/voila-dashboards/voila/pull/208/files

Fixes jupyter#1430 Many users are reporting that nbconvert 6.x has permission errors when queried or converting files. These seem to arise when nbconvert is installed as root, then used as a user. We hypothesize that the code removed in this commit, which attempts to create template directories throughout the jupyter path hierarchy, succeeds in creating template directories when running as root. Note that it creates the template directories with permission 700. Those permissions as root mean that regular users won’t be able to read these directories, and those are the sort of permission errors we are seeing reported. This directory creation was originally introduced in jupyter#1028, where I think it was limited to a single user configuration directory (based on the PR discussion). In jupyter#1056, this code appears to have been copied over and applied to the entire Jupyter directory hierarchy.

* Do not attempt to create additional template paths. Fixes #1430 Many users are reporting that nbconvert 6.x has permission errors when queried or converting files. These seem to arise when nbconvert is installed as root, then used as a user. We hypothesize that the code removed in this commit, which attempts to create template directories throughout the jupyter path hierarchy, succeeds in creating template directories when running as root. Note that it creates the template directories with permission 700. Those permissions as root mean that regular users won’t be able to read these directories, and those are the sort of permission errors we are seeing reported. This directory creation was originally introduced in #1028, where I think it was limited to a single user configuration directory (based on the PR discussion). In #1056, this code appears to have been copied over and applied to the entire Jupyter directory hierarchy. * Filter additional template paths to existing paths to be consistent.

bdklahn · 2022-09-01T16:46:20Z

Is this issue related, at all, to why there is no "to gridstack"-like entry in the export menu item?
I have been using the Gridstack editor, successfully (with pop-out browser tab rendering), but wondered why there might not be a corresponding entry in the export menu.
I wondered if it were the Gridstack folks who would/could/should add that, but . . .
voila-dashboards/voila-gridstack#180 (comment)

I've been able to get the Voila-DejaVu-like (static page) behavior using the following config settings, in a script.

    c = Config()
    c.TemplateExporter.exclude_input = True
    c.TemplateExporter.exclude_output_prompt = True
    c.TemplateExporter.exclude_input_prompt = True
    c.ExecutePreprocessor.enabled=True
    c.HTMLExporter.template_name = 'gridstack'

Barring any potentially-related fundamental issues, possibly like this, I would think adding a Gridstack render export could be pretty straight-forward(?).

I apologize if this is too unrelated to this and should be in it's own issue.

MSeal added this to the 6.0 milestone Jun 28, 2019

MSeal requested review from t-makaro and MSeal June 28, 2019 16:56

maartenbreddels added a commit to maartenbreddels/voila-material that referenced this pull request Jul 24, 2019

Make it work with the voila/nbconver template refactor

0170ef4

Depends on jupyter/nbconvert#1056 Depends on voila-dashboards/voila#208

maartenbreddels added a commit to maartenbreddels/voila-material that referenced this pull request Jul 24, 2019

Make it work with the voila/nbconver template refactor

bf70dc3

Depends on jupyter/nbconvert#1056 Depends on voila-dashboards/voila#208

maartenbreddels mentioned this pull request Jul 24, 2019

Make it work with the voila/nbconver template refactor voila-dashboards/voila-material#2

Closed

willingc added the status:work-in-process Do not merge label Jul 29, 2019

maartenbreddels changed the title ~~POC: Refactor template system to be easily extensible and voila compatible~~ Refactor template system to be easily extensible and voila compatible Jul 31, 2019

maartenbreddels changed the base branch from master to 6.x July 31, 2019 12:41

SylvainCorlay reviewed Jul 31, 2019

View reviewed changes

maartenbreddels force-pushed the refactor_templates branch from e44311d to 2fc375d Compare July 31, 2019 14:54

maartenbreddels force-pushed the refactor_templates branch from 2fc375d to d6946f4 Compare July 31, 2019 15:04

This was referenced Aug 7, 2019

Incremental rendering of output and widgets voila-dashboards/voila#142

Closed

Convergences w/ Jupyter Book voila-dashboards/voila#339

Open

maartenbreddels force-pushed the refactor_templates branch from cc487ef to 621eb9d Compare October 14, 2019 14:04

maartenbreddels force-pushed the refactor_templates branch from 4b2009f to 6df9090 Compare October 14, 2019 14:45

MSeal approved these changes Oct 15, 2019

View reviewed changes

maartenbreddels force-pushed the refactor_templates branch from 6df9090 to d41a2f8 Compare October 15, 2019 15:23

maartenbreddels mentioned this pull request Oct 15, 2019

convert to nbconvert and voila compatible template voila-dashboards/voila-material#8

Merged

Refactor template system to be easily extensible and voila compatible

f4c7dae

Co-authored-by: Sylvain Corlay <sylvain.corlay@gmail.com>

maartenbreddels force-pushed the refactor_templates branch from d41a2f8 to f4c7dae Compare October 15, 2019 17:26

MSeal merged commit cfb47b5 into jupyter:6.x Oct 17, 2019

choldgraf mentioned this pull request Oct 17, 2019

Document recent updates to the templating process #1119

Open

mgeier mentioned this pull request Oct 26, 2019

Sphinx parallel errors #1126

Closed

This was referenced Nov 2, 2019

Update template name changed in nbconvert development version spatialaudio/nbsphinx#344

Merged

add a timeout error so that it shows the contents of the cell which timed out #1129

Closed

chrisjsewell mentioned this pull request Nov 9, 2019

support nbconvert version 6 when released chrisjsewell/ipypublish#108

Open

maartenbreddels mentioned this pull request Dec 13, 2019

Fix jquery and element var #1152

Merged

maartenbreddels mentioned this pull request Jan 28, 2020

VoilaExecutePreprocessor.allow_errors always True voila-dashboards/voila#519

Open

chrisjsewell mentioned this pull request Sep 25, 2020

Team sync on two-month development plan executablebooks/meta#149

Closed

25 tasks

kandersolar mentioned this pull request Oct 6, 2020

Fix/issue 205 NREL/rdtools#207

Merged

5 tasks

jasongrout mentioned this pull request Sep 17, 2021

Permission denied problem with nbconvert #1430

Closed

jasongrout mentioned this pull request Sep 17, 2021

Do not attempt to create additional template paths. #1646

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor template system to be easily extensible and voila compatible #1056

Refactor template system to be easily extensible and voila compatible #1056

maartenbreddels commented Jun 28, 2019

MSeal commented Jun 28, 2019

MSeal commented Jul 26, 2019 •

edited

Loading

MSeal commented Jul 26, 2019

MSeal commented Jul 26, 2019

MSeal commented Jul 26, 2019

MSeal commented Jul 28, 2019

maartenbreddels commented Jul 31, 2019 •

edited

Loading

maartenbreddels commented Jul 31, 2019

SylvainCorlay Jul 31, 2019

maartenbreddels Jul 31, 2019

SylvainCorlay Jul 31, 2019

maartenbreddels Jul 31, 2019

maartenbreddels commented Jul 31, 2019

t-makaro commented Jul 31, 2019

maartenbreddels commented Aug 1, 2019

maartenbreddels commented Aug 1, 2019

maartenbreddels commented Aug 7, 2019

t-makaro commented Aug 7, 2019

t-makaro commented Aug 7, 2019 •

edited

Loading

MSeal commented Sep 4, 2019

maartenbreddels commented Oct 14, 2019

MSeal commented Oct 15, 2019

maartenbreddels commented Oct 15, 2019

MSeal commented Oct 15, 2019

MSeal commented Oct 17, 2019

choldgraf commented Oct 17, 2019

maartenbreddels commented Oct 17, 2019

bdklahn commented Sep 1, 2022

Refactor template system to be easily extensible and voila compatible #1056

Refactor template system to be easily extensible and voila compatible #1056

Conversation

maartenbreddels commented Jun 28, 2019

Loose ends

MSeal commented Jun 28, 2019

MSeal commented Jul 26, 2019 • edited Loading

MSeal commented Jul 26, 2019

MSeal commented Jul 26, 2019

MSeal commented Jul 26, 2019

MSeal commented Jul 28, 2019

maartenbreddels commented Jul 31, 2019 • edited Loading

maartenbreddels commented Jul 31, 2019

SylvainCorlay Jul 31, 2019

Choose a reason for hiding this comment

maartenbreddels Jul 31, 2019

Choose a reason for hiding this comment

SylvainCorlay Jul 31, 2019

Choose a reason for hiding this comment

maartenbreddels Jul 31, 2019

Choose a reason for hiding this comment

maartenbreddels commented Jul 31, 2019

t-makaro commented Jul 31, 2019

maartenbreddels commented Aug 1, 2019

maartenbreddels commented Aug 1, 2019

maartenbreddels commented Aug 7, 2019

t-makaro commented Aug 7, 2019

t-makaro commented Aug 7, 2019 • edited Loading

MSeal commented Sep 4, 2019

maartenbreddels commented Oct 14, 2019

MSeal commented Oct 15, 2019

maartenbreddels commented Oct 15, 2019

MSeal commented Oct 15, 2019

MSeal commented Oct 17, 2019

choldgraf commented Oct 17, 2019

maartenbreddels commented Oct 17, 2019

bdklahn commented Sep 1, 2022

MSeal commented Jul 26, 2019 •

edited

Loading

maartenbreddels commented Jul 31, 2019 •

edited

Loading

t-makaro commented Aug 7, 2019 •

edited

Loading