Recompute after cell update

[turiot]

Hello, following our exchange on reddit, I think it's a very hard task to detect variables updates, for instance :

    var liste = new ArrayList<String>();
    var x = "x";
    liste.add(x);
    liste.get(0).toUpperCase();
    
    void method(Point point) {
       point.x = 5;
    }
    var p = new Point(10, 20);
    method(p);  
    
    method in a jar not analyzed by jshell

Using a "deepHashCode" to compare vars after snippet execution can be a solution, but with size limitations (huge collections or arrays).

Another option is to recompute the whole notebook, excepting snippets which previous executions was time consuming, asking advice to the user.
This solution overlaps the restore of a saved notebook.

Maybe these options can coexist and be chosen by the user, according to his usage profile (java learning, api testing, data science, ...)

[cyrilou242]

Hey,

I'm not sure to understand the example. Is this something that is not working as expected in the current implementation?

If not I'll move this to discussions.

Using a "deepHashCode" to compare vars after snippet execution can be a solution, but with size limitations (huge collections or arrays).
Another option is to recompute the whole notebook, excepting snippets which previous executions was time consuming, asking advice to the user.
This solution overlaps the restore of a saved notebook.
Maybe these options can coexist and be chosen by the user, according to his usage profile (java learning, api testing, data science, ...)

[turiot]

Sorry, discussions was actually the place to post.

[cyrilou242]

moved.

[cyrilou242]

Using a "deepHashCode" to compare vars after snippet execution can be a solution, but with size limitations (huge collections or arrays).

When I designed the engine I concluded using hash codes would not work.
I'm curious if you have a good algorithm in mind.

Eg

var l = new ArrayList(4, 4);
var x = l.remove(1);

At this point x = 4 and the list is [4]

Now let's say I change the second line to

var x = l.remove(0);

If I run this line, x is not changed, but my list is now empty.
How would I know with hashes I need to re-run the first line to be in a correct state?

The other option I considered was to watch for hash changes and deduce what should be updated, but then:

var l = new ArrayList(4, 4);
var x = l.add(1);

Change line 2 to l.add(2);
run l.add(2) because it has changed.
run line 1 because it was detected that l has changed.
re-run line 2 because l line 1 was run.
--> I end up executing two times line 2, and this quickly gets out of control when there are more than 2 lines of code.

Another option is to recompute the whole notebook, excepting snippets which previous executions was time consuming, asking advice to the user.
This solution overlaps the restore of a saved notebook.
Maybe these options can coexist and be chosen by the user, according to his usage profile (java learning, api testing, data science, ...)

Yes I think I'll go with this.
I think by default the server will re-run everything and let users put magic comments to cache some lines (eg loading csv data).
Stats of re-run will be maintained. As soon as the time to re-run becomes >> time to build a dependency graph, then I'll switch to the dependency graph mode.
Stats when using the dependency graph will also be maintained and the server will be able to chose what is best.

[turiot]

A hashCode could be the MD5 checksum of the serialized object (by kryo etc) and a stream.

Isn't it enough to recompute only the snippets below the updated one ?

I agree with you, a notebook is more than a spreadsheet, more a program elaborating in a shell like fashion; most part of users should understand its complexity in terms of order of execution and operating time; so pining the previous long snippet execution as 'recompute only on demand' could be a acceptable compromise.

[cyrilou242]

Oh got it thanks.
Nice idea.

But then I'd have to save a state at each statement I think.

Eg:

1. var l = new ArrayList<>();
2. l.add(1);
3. l.add(2);
4. l.add(3);

if line 3. is edited --> I need to get back to the state just before.
more generally the algo would be:

if statement n is different from before --> load state from statement n-1

• manage imports separately

Am I getting your point right?

With a persistent map this may be ok. But performing the reflection to know what object needs to be serialized based on a static AST parsing would be complex.
Other issue I guess will be with everything volatile or connections-like that are not easy to ser/deser.

But this make me think I could consider snapshotting the full jshell process. I don't know much about this but there are JVM snapshotting technologies, I'll have a look.

[cyrilou242]

https://danheidinga.github.io/Everyone_wants_fast_startup/

[turiot]

I am afraid no one tackled the objects tracking problem before.
Serialization for comparing may also be tricky, as JEE use proxies.
JVMTI/JPDA seems very difficult.
And corner cases may still exists : off heap data structures, unsafe cheats and so on, reuse of a variable in another non related step (x,y,z,t).

A lisp like save image is original but what about sockets, volume and versioning ?

So IMHO a visual dependency graph helper may be a solution, and an alternative cells pining as in seco (cells that must be reevaluated at startup) is the simplest comprehensive choice.

Don't miss the point that a notebook is a presentation tool :

• a developer won't use it, preferring a simple shell to test a syntax or an api;
• a teacher will use it, but he is skilled and won't manipulate big data;
• a data scientist will also use this tool, but he will process by steps (with intermediate files or db), marking visually bootstrap steps, and done steps could be enough.

Maybe you could report your efforts on confort features :

• libraries management (maven repo)
• manifold like preprocessing, for example to add exponentiation or operator overloading
• collect more renderers (plot results, images)
• add viewers for files or db
• integrate other script languages

Nonetheless thanks again for this pure java tool.