Question / Idea

[tam203]

@SylvainCorlay Theo from the Met Office Informatics lab here. Great to talk earlier.

I had a idea that I've not fully processed but thought would be fun to bounce off you.

Do you remember my concern over notebooks with some initial heavy lifting / loading and that this would be repeated for each page view/request?

Do you think that it would be theoretically (at least) possible to execute the notebook, capture the output but but at this point fork the process and return/use the fork as the kernel for the output/JS widgets. The original is "kept" and each new request just forks the original "first pass" kernel/environment. The advantage if this is possible is that any heavy lifting is only done once. Additionally due to (and my understanding is poor here so this might be nonsense) the nature of process forking only copying memory on modification, there could be a potential memory footprint saved if the code had large objects but that the underlying data isn't changed (just different views taken).

No idea if this is even possible but would like your thoughts.

Cheers,

Theo.

[niallrobinson]

👋 from me too

[SylvainCorlay]

Hey @niallrobinson @tam203 thanks for dropping by.

Yes, it sounds like a reasonable approach. Another one would be to require all the heavy-lifting part done only once to be easily recovered from a bunch of pickle / serialized files that hold all the outcome of the computation and can be used to bootstrap things in a more efficient way, but that would require the application to be written in a specific way for that purpose...

[tam203]

As you say it would require users to take a certain approach but a nice style might be:

try:
   data =  load('my.pickle')
except IOError:
   # do stuff to make data
   pickle(data, 'my.pickle')

[maartenbreddels]

I talked to @minrk about this idea, and I think it would be useful in general to be able to fork/clone a kernel after some point. However, this should not go in voila, I guess it should be something implemented in ipykernel, and added to the specs accordingly. Voila could then make use of this, but I don't see this as something that would go into voila itself.