Incremental rendering of output and widgets

[jeffyjefflabs]

Would it be possible to render output and widgets cell by cell instead of waiting until the whole notebook has executed? Our particular use case:

1. Build an input form using widgets
2. Use something like ipython_blocking to pause until the user submits the form or input validation succeeds
3. Perform analysis based on input parameters and generate output visualizations

See also #125; currently the execution timeout also prevents this from working.

I can think of other use cases as well -- status messages during long-running operations, visualizations that update as analysis progresses, etc.

[SylvainCorlay]

@jeffyjefflabs absolutely. There are three open PRs by @maartenbreddels implementing different ways to achieve that. This will definitely be in the next releasd.

[choldgraf]

cc @somedave who considered this a blocker for their setup as well!

[gedankenstuecke]

I just wanted to chime in to say that this is a blocker for our deployment too! Looking forward for that release!

[vidartf]

There are three open PRs by @maartenbreddels [...]

Xref #60, #71, #133. At the current time, only #133 remains open.

[vidartf]

Looking at this, I think I would prefer the option of a two-phase system along these lines:

1. A template is rendered immediately.
2. JS in the template hits an API that triggers the notebook execution. This API might return / generate more content from another template.

It is still not clear to me whether it would be better for the kernel to start up during step 1 or step 2.

Advantages to this approach:

• Do complete initialization in an immediate step, allowing scripts to trigger on page load completion.
• Allows the JS to receive the notebook output in JSON format, and/or just simply rely on Comms for sending widget data.
• Possibly allow JS to react to kernel messages such that we could support things like progress bars.

[vidartf]

Ping @maartenbreddels :)

[vidartf]

Note: I think I can achieve what I want by adding a new API entry point (via a separate server extension) that preloads a page before calling the voila entry point. I could then have the template yield the inner content only, if combined with this PR. Ideally this would be more integrated with Voila, but I'm happy to work around it if need be, to avoid stalling on this issue.

[maartenbreddels]

I think we can do this without a separate entry point, as #133 shows.
My preference is to do this after #208 and jupyter/nbconvert#1056 have landed.

However, after working on #1056 I am also not convinced anymore that we should have this in nbconvert, since it would require a large refactor of nbconvert, and is more important for voila.

I'll see if I can pick up #133 again, and include per cell rendering.

[pbadenski]

+1 this would really help us with authentication flows for some of the notebooks

[maartenbreddels]

I think we should close this issue, we have implemented the progressive render. Progressive rendering of widgets is technically not possible (since we don't know which widget/comm creation/open events we missed). I still like @vidartf's idea of loading a page, and via an API render the notebook client side. I think we should open a new issue for that if we want that.