Bokeh update causing "_pending_writes should be non-None"

[nitrocalcite]

Possibly a regression of #344.

I have a Bokeh figure with an image, colorbar, and colormapper. I want to interactively update the bounds on the colorbar. When I use Bokeh to serve the figure statically, this works great with color_mapper.update(low=low, high=high). However, if I embed this figure in a panel with pn.pane.Bokeh(p), this command produces a RuntimeError from Bokeh, identical to that in #344 (stack trace & MRE below):

RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

I expected that as Panel is based off of Bokeh, I should be able to interactively update Bokeh plots.

This error seems to be related to pushing the change to the client browser. Even with the RuntimeError, refreshing the browser will yield the desired changes. The RuntimeError will not occur if the change does nothing; for example, trying to set the same parameter values twice will only cause an error on the first attempt.

I discovered this originally on panel 0.9.5 & bokeh 2.0.1 but have since reproduced on panel 0.9.6 & bokeh 2.1.0, running on Ubuntu 16.04 LTS.

Please let me know if there is a workaround, as short of swallowing the exception and forcing the client to refresh, I don't know any approaches. Thanks!

Here's my MRE:

import panel as pn
import numpy as np
from bokeh.plotting import figure
from bokeh.models import ColorBar, ColumnDataSource, LinearColorMapper, AdaptiveTicker

image = np.random.randn(128, 128)

low, high = image.min(), image.max()

def _get_src_dict_from_image(image):
    return dict(image=[image], x=[0], y=[0], dw=[image.shape[1]], dh=[image.shape[0]])

# create bokeh figure: one image w/colorbar and associated mapper
p = figure(tooltips=[("x", "$x"), ("y", "$y"), ("value", "@image")])
src = ColumnDataSource(data=_get_src_dict_from_image(image))
color_mapper = LinearColorMapper(palette="Viridis256", low=low, high=high)
p.image("image", source=src, x='x', y='y', dw='dw', dh='dh', level="image",
                color_mapper=color_mapper)
color_bar = ColorBar(color_mapper=color_mapper, ticker=AdaptiveTicker(),
                    label_standoff=12, border_line_color=None, location=(0, 0))
p.add_layout(color_bar, 'right')

pn.pane.Bokeh(p).show(threaded=True)  # give to Panel
color_mapper.update(high=100)   # this line throws the error
color_mapper.update(high=100)   # this line will not throw an error

Producing this stack trace:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-da8b0df4fb70> in <module>
----> 1 color_mapper.update(high=100)

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/has_props.py in update(self, **kwargs)
    368         '''
    369         for k,v in kwargs.items():
--> 370             setattr(self, k, v)
    371
    372     def update_from_json(self, json_attributes, models=None, setter=None):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/has_props.py in __setattr__(self, name, value)
    272
    273         if name in props or (descriptor is not None and descriptor.fset is not None):
--> 274             super().__setattr__(name, value)
    275         else:
    276             matches, text = difflib.get_close_matches(name.lower(), props), "similar"

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/property/descriptors.py in __set__(self, obj, value, setter)
    537             raise RuntimeError("%s.%s is a readonly property" % (obj.__class__.__name__, self.name))
    538
--> 539         self._internal_set(obj, value, setter=setter)
    540
    541     def __delete__(self, obj):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/property/descriptors.py in _internal_set(self, obj, value, hint, setter)
    761
    762         old = self.__get__(obj, obj.__class__)
--> 763         self._real_set(obj, old, value, hint=hint, setter=setter)
    764
    765     def _real_set(self, obj, old, value, hint=None, setter=None):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/property/descriptors.py in _real_set(self, obj, old, value, hint, setter)
    830
    831         # for notification purposes, "old" should be the logical old
--> 832         self._trigger(obj, old, value, hint=hint, setter=setter)
    833
    834     # called when a container is mutated "behind our back" and

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/core/property/descriptors.py in _trigger(self, obj, old, value, hint, setter)
    907         '''
    908         if hasattr(obj, 'trigger'):
--> 909             obj.trigger(self.name, old, value, hint, setter)
    910
    911

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/model.py in trigger(self, attr, old, new, hint, setter)
    659                     self._document._invalidate_all_models()
    660         # chain up to invoke callbacks
--> 661         super().trigger(attr, old, new, hint=hint, setter=setter)
    662
    663     def _attach_document(self, doc):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/util/callback_manager.py in trigger(self, attr, old, new, hint, setter)
    155                     callback(attr, old, new)
    156         if hasattr(self, '_document') and self._document is not None:
--> 157             self._document._notify_change(self, attr, old, new, hint, setter, invoke)
    158         else:
    159             invoke()

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/document.py in _notify_change(self, model, attr, old, new, hint, setter, callback_invoker)
   1040
   1041         event = ModelChangedEvent(self, model, attr, old, new, serializable_new, hint, setter, callback_invoker)
-> 1042         self._trigger_on_change(event)
   1043
   1044     def _push_all_models_freeze(self):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/document.py in _trigger_on_change(self, event)
   1135             for cb in self._callbacks.values():
   1136                 cb(event)
-> 1137         self._with_self_as_curdoc(invoke_callbacks)
   1138
   1139     def _with_self_as_curdoc(self, f):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/document.py in _with_self_as_curdoc(self, f)
   1148             else:
   1149                 set_curdoc(self)
-> 1150             return f()
   1151         finally:
   1152             set_curdoc(old_doc)

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/document.py in invoke_callbacks()
   1134         def invoke_callbacks():
   1135             for cb in self._callbacks.values():
-> 1136                 cb(event)
   1137         self._with_self_as_curdoc(invoke_callbacks)
   1138

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/document.py in <lambda>(event)
    702     def on_change_dispatch_to(self, receiver):
    703         if not receiver in self._callbacks:
--> 704             self._callbacks[receiver] = lambda event: event.dispatch(receiver)
    705
    706     def on_session_destroyed(self, *callbacks):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/events.py in dispatch(self, receiver)
    267
    268         '''
--> 269         super().dispatch(receiver)
    270         if hasattr(receiver, '_document_model_changed'):
    271             receiver._document_model_changed(self)

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/document/events.py in dispatch(self, receiver)
    122         super().dispatch(receiver)
    123         if hasattr(receiver, '_document_patched'):
--> 124             receiver._document_patched(self)
    125
    126     def generate(self, references, buffers):

~/miniconda3/envs/PyX/lib/python3.7/site-packages/bokeh/server/session.py in _document_patched(self, event)
    216
    217         if self._pending_writes is None:
--> 218             raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")
    219
    220         # TODO (havocp): our "change sync" protocol is flawed because if both

RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

[nitrocalcite]

I should add that this error is not specific to the color_mapper.update call. Attempting to replace the colormapper or similar updates causes identical errors. However, updating the plot via the data source (src in the above MRE) works fine.

[nitrocalcite]

Potential workaround: calling color_mapper.update from Bokeh itself seems to bypass this issue. I have it registered as a p.on_event(Tap, func) callback and this issue disappears

[philippjfr]

In Bokeh you would also not be allowed to update the model directly, you would usually have to wrap the change in a callback, e.g. by using Document.add_next_tick_callback. Since we don't currently expose this in Panel I don't yet have a great suggestions here. I suspect we will have to expose that functionality in Panel too.

[xavArtley]

Indeed there is no easy way.
you need to retrieve the bokeh document(s) of the server (using pn.state_servers you get the list of servers and documents associated)

The you create a callback to pass to the add_next_tick_callback of the document:

[philippjfr]

There is definitely an easier way to get at the Document at least since the model itself will reference it:

color_mapper.document.add_next_tick_callback(lambda: color_mapper.update(high=100))

[philippjfr]

What I could imagine offering is a model_update utility which appropriately schedules the Model property update(s). Something like:

pn.io.server.update(color_mapper, high=100)

[xavArtley]

There is definitely an easier way to get at the Document at least since the model itself will reference it:

color_mapper.document.add_next_tick_callback(lambda: color_mapper.update(high=100))

Interesting I didn't know models referenced their documents

However it references only the last one created?
If based on a server update it should update all documents hold by the server?

[philippjfr]

However it references only the last one created?

Correct that's why you should (almost) never share a Bokeh model across sessions, they are not designed to support this use caase.

If based on a server update it should update all documents hold by the server?

That depends if you really want to share one model across sessions (which seems very rare) you will indeed have to find some way to do this, but the far more common scenario is that each session should create its own model and respond independently.

[philippjfr]

Weirdly I can't reproduce this using Panel 0.9.7, which I can't really explain. Can anyone still reproduce this? That said I'm going to add an API to schedule a next tick callback from Panel anyway.