Use of nbconvert TemplateExporter to exclude input/output and/or the related in/out prompts

[keluc]

In order to use this for every notebook at once I'm still trying some things out (see #15) for my earlier comments)

As suggested there, I'v added some nbsphinx options and used the global setting in conf.py

def setup(app):
  ...
    #hide or display input, output,....
    app.add_config_value('nbsphinx_exclude_output', False, rebuild='env')
    app.add_config_value('nbsphinx_exclude_input', False, rebuild='env')
    app.add_config_value('nbsphinx_exclude_input_prompt', False, rebuild='env')
    app.add_config_value('nbsphinx_exclude_output_prompt', False, rebuild='env')
    app.add_config_value('nbsphinx_exclude_markdown', False, rebuild='env')
    app.add_config_value('nbsphinx_exclude_code_cell', False, rebuild='env')

which is used then in the config setting of Exporter(nbconvert.RSTExporter)

class Exporter(nbconvert.RSTExporter):
...
    def __init__(self, execute='auto', kernel_name='', execute_arguments=[],
                 allow_errors=False, timeout=30, codecell_lexer='none',
                 exclude_output = False, exclude_input = False, 
                 exclude_input_prompt = False, exclude_output_prompt = False, 
                 exclude_markdown = False, exclude_code_cell = False):
       ...
        self._exclude_output = exclude_output
        self._exclude_input = exclude_input
        self._exclude_input_prompt = exclude_input_prompt
        self._exclude_output_prompt = exclude_output_prompt
        self._exclude_markdown = exclude_markdown
        self._exclude_code_cell = exclude_code_cell 
        
        super(Exporter, self).__init__(
            template_file='nbsphinx-rst.tpl', extra_loaders=[loader],
            config=traitlets.config.Config(
                {'HighlightMagicsPreprocessor': {'enabled': True},
                'TemplateExporter':{
                "exclude_output": exclude_output,
                "exclude_input": exclude_input,
                "exclude_input_prompt": exclude_input_prompt,
                "exclude_output_prompt": exclude_output_prompt,
                "exclude_markdown": exclude_markdown,
                "exclude_code_cell": exclude_code_cell}}),
            filters={
                'convert_pandoc': convert_pandoc,
                'markdown2rst': markdown2rst,
                'get_empty_lines': _get_empty_lines,
                'extract_toctree': _extract_toctree,
                'get_output_type': _get_output_type,
                'json_dumps': json.dumps,
            })

This works fine but not for every output prompt.
I tried to solve it but apparently in class NbOutput(rst.Directive) I don't seem to be able to get hold of the "exclude_output_prompt" setting.
With a global variable I can tackle this but still trying to figure out how to do it correctly without using this 'dirty hack':

class NbOutput(rst.Directive):
 ...
    def run(self):
           ....
            global nbsphinx_args_output_prompt
            if nbsphinx_args_output_prompt:
                text = ''
            else:
                text = 'Out [{}]:'.format(execution_count)

See: changes
Any Idea how I can pass env.config.nbsphinx_exclude_output_prompt to the run method of the class NbOutput?

[mgeier]

You can try to access it with self.state.document.settings.env.config.nbsphinx_exclude_output_prompt.
There is probably a shorter way to get it ...

[keluc]

This does indeed work and is sufficient to hide the output prompt. So with this we can hide input code, input and output prompts. If ok for you, I'll be happy to create a pull request and an addition to the documentation (e.g. with a notebook 'Options2HideInformation.ipynb' explaining how to use these options)

[mgeier]

I'm not very enthusiastic about hiding stuff, but several people already asked for something like that, so I guess I'll have to accept it if you make a nice PR.

[keluc]

I understand your remark but it's how you look at it.
If your audience are other technicians than you want to show also the code, but if your audience are reports to non-technicians, to management, ... than the code is not of interest to the reader. What I'd also like to add is to have the possibility to include a hyperlink to a 'mirror' page where in- and output is shown (or a show/hide button on every page to show the input). Finally the source code option gives you a also a sort of 'audit trail'

[mgeier]

IMHO if you don't want to show the code, Jupyter notebook is the wrong tool for you.

There are tons of tools for automated document generation and many different templating languages available for you to choose from, you don't have to use Jupyter.

But of course, every tool can be abused, and to a certain degree that's OK.

And if those nbsphinx_exclude_* options help some people in doing what they think they want, I'm willing to include them, as I said above.

[rddaz2013]

first...nbsphinx is an nice tool and great work...!

IMHO if you don't want to show the code, Jupyter notebook is the wrong tool for you.

There are some situations where the source code in the report or presentation is not desired - but for the editing must be present.

The fine thing is with nbsphinx I have code for generating charts and normal text together in one document. But for the paper version I do not need every line of the source code.

Is there now a way for nbsphinx to 'disappear' the source code but to print the output?

[mgeier]

@rddaz2013 The situation here is unchanged. I still think it's a bad thing, nobody has shown me an example that would change my mind, but I'm still open to accepting a PR nevertheless. Nobody has created such a PR yet.

I think in the meantime nbconvert supports "tag-based filtering", but I don't know if and how this works together with nbsphinx.

[rddaz2013]

From the point of view of the idea of Jupyter, 'suppressing' the source code is certainly not a good reason to do it. I am completely on your side @mgeier

But for the 'Old World' who want a report without calculation and just want to see the graphs on a printable paper - it would be incredibly practical.

Practically because I have only one workflow to produce the document with source or without.

[DavidPowell]

Is there any update on this? Being able to hide code and output prompts is vital for my use case.

@rddaz2013 There are plenty of valid use cases for hiding input that do not violate the idea of Jupyter. For example, I may generate tikz schematic figures with ipython-tikz, but I do not want the tikz source code appearing in my output.

[keluc]

A quick update from my end. I used my simple solution in a project that lasted about 10 months for about 15 fully automated runs. I encountered no issues. I didn't come up with a PR due to my current workload and also a little bit because of the given remarks. I think that the ideal solution would be both options : hiding input all together, only for a certain (set of) pages and tag-based filtering. I was thinking of finding a solution for all these options at once. So I have to dig in the tag-based filtering first (hopefully after may). PS. an example in favor of this possibility: see also what people in the R-world are doing with R-markdown, R-notebooks, R-bookdown,... In addition to that the reticulate package recently gives one the possibility to use Python code in those solutions (not optimal yet imho). In Python you have pweave but it misses the overall functionality of sphinx

[mapio]

Any progress on this? I agree with @DavidPowell that this would be very useful!

[mgeier]

Tag-based filtering works if nbconvert 5.3 or newer is installed (see #125). Other than that, I don't know of any news.

[mapio]

Sorry for being unclear. I know (and use) the feature of nbconvert. What I was asking is if there is a plan to somehow integrate such functionality with nbsphinx (that is, a way of configuring nbsphinx to respect tag driven exclusion/inclusion of cells, inputs and/or outputs).

[mgeier]

Sorry, I was unclear: tag-based filtering should also work in nbsphinx (given that the installed nbconvert version supports it).
If it doesn't work for you, please create a new issue.

[mapio]

I know how to use it invoking jupyter nbconvert from the command line. Where can I find the documentation on how to use it within nbsphinx?

[DavidPowell]

@mgeier I'd like to challenge your view that this proposal represents a misuse of Jupyter notebooks.
After all, the Jupyter devs created the highly flexible nbconvert tool specifically to allow unforseen use-cases. The only thing it doesn't do is create a good structure of multiple linked documents with table of contents etc. Sphinx naturally gives us this structure, so I really appreciate that nbsphinx was created to bridge these two tools.

Regarding your earlier comment

There are tons of tools for automated document generation and many different templating languages available for you to choose from, you don't have to use Jupyter.

I am not aware of any tool other than Jupyter that gives us the ability to generate rich output with Python (or any other language), inspect the results directly inline, iteratively improve them with quick feedback, and then store the results in the same document as the input. The only alternative I can think of is a python script with a whole bunch of rich outputs stored as separate files.

To me, Jupyter notebooks seem like the ideal tool for authoring html documents with rich content, I can easily include matplotlib/holoviews/bohek plots, tikz schematics, ipython widgets (which can work well in static output as long as they don't need a server backend), latex equations, videos etc.

Back to the technical issues, currently the only way for nbsphinx to pass options to nbconvert is through modifications to nbsphinx code such as this one by @keluc. I did try running nbconvert --create-config and editing the resulting jupyter-nbconvert-config.py, but it seems that the values in this config file are ignored when a config option is explicitly provided in the code.

Anyway, the proposed modifications do look more or less ready to go, so if there is a consensus on this approach then I'm willing to go ahead and create a pull request.

My only hesitation is that it might be better to allow an nbconvert configuration object to be passed by the user (potentially overriding some of the values set by nbsphinx), allowing the many nbconvert options to be utilised without requiring nbsphinx to support them all.

Pros of a user-specified nbconvert configuration:

• Maximum flexibility, including for user-written filters, preprocessors etc.
• Minimises the code changes required to nbsphinx

Cons:

• Might be a bit unfriendly for the most common use cases proposed, as it needs in-depth understanding of all the nbconvert options (this might be mitigated by a clear example)
• Output prompts are currently manually added, thus they are unaffected by nbconvert filtering options. (Could this be refactored somehow?)

[mgeier]

Sorry @mapio, I was totally wrong, tag-based filtering does not work in nbsphinx, because there seem to be no pre-defined tags available (or are there?). I was confusing this with the pre-defined tag named raises-exception, which works, see #125.

Thanks @DavidPowell for your comment. I agree with most of it. I agree that the Jupyter notebook is a great tool for interactively exploring stuff and sharing this stuff (including rich media) with the world.
I think it is not a great tool for hiding stuff from the world.

But as much as I think that hiding is bad, I'm willing to accept a PR for this issue here, as I already mentioned above.

if there is a consensus on this approach then I'm willing to go ahead and create a pull request.

I don't know about consensus, but it looks fine to me.

it might be better to allow an nbconvert configuration object to be passed by the user (potentially overriding some of the values set by nbsphinx), allowing the many nbconvert options to be utilised without requiring nbsphinx to support them all.

I'm not generally against that, but I'm also not sure if that's a good idea.
It might work for some options, but other options will simply be ignored, because nbsphinx does some things differently from nbconvert.

Especially I wouldn't count on the options related to TemplateExporter, because on the long run I want to get rid of its use in nbsphinx (see #36).

I don't have a problem with mirroring some of the nbconvert options in nbsphinx if people are requesting them.

Output prompts are currently manually added, thus they are unaffected by nbconvert filtering options. (Could this be refactored somehow?)

I'm open to any suggestions for refactorings or other improvements!

[DavidPowell]

Given comments by @mgeier on the long-term plans to remove nbconvert, the most logical approach is to just merge @keluc's changes to the latest master. I have created such a PR, after verifying that this code works for my specific examples.

[mgeier]

@DavidPowell Thanks for the PR. Just for clarification: I'm not planning to remove nbconvert completely, only the use of the TemplateExporter and the conversion from Markdown to reST (see #36).

[DavidPowell]

To everyone interested in this issue, on the PR (see #185) I am proposing to only implement hiding code cell input, input prompts and output prompts.

If you have a use case which isn't covered by these features, please comment here or in the PR.

[OverLordGoldDragon]

So the "workarounds" discussed here aren't for an out-of-box nbsphinx? Is there anything we can put in conf.py for a direct install to hide In [] and Out []?

[OverLordGoldDragon]

Why not just

div.prompt {display: none;}

style.css? Or is this Issue about something else? Seems to work perfectly for me.