Live plotting

[alexcjohnson]

I have a bunch of cleanup still to do, to put in automatic labeling, automatic dependent vars, etc... but the overall system works. In the future I expect it won't be too hard to do things like pop these plots out into their own windows, but at the moment they're inline in the notebook.

• When you make a Plot, you specify how often to update it (default once a second)
• It creates an invisible widget in the cell that runs a Javascript setInterval timer
• Javascript sends a message back to Python periodically, which syncs the data and redraws the plot.
  • This means the notebook is still unblocked during plotting, though I've noticed that autocomplete slows down or sometimes doesn't work right while we're in a plotting loop.
  • Note that the way this is implemented, it depends on in-place modifications of the data arrays. You can use this with anything that matplotlib knows how to plot, like regular lists and numpy arrays, as long as you modify them in place, rather than reassigning to them.
• The sync functions return a boolean:
  • True means we should keep updating (Actually, anything but exactly False behaves this way, so functions that don't know about this and either return nothing (None) or return the data or something will keep us updating until told to stop.
  • False means the data is complete and we can stop updating.
  • For now I just keep updating as long as the sweep is continuing, but if we access this data remotely (or even from some other process that doesn't know about our DataServer) it could certainly look at the file on disk and keep updating until the file has not changed in a sufficiently long time.
• When all the sync functions (in case you're plotting data from several DataSets together, for example) return False, we send a message back to the invisible widget that stops the Javascript timer loop.
• you can also manually stop updates by calling plot.update_widget.halt() (perhaps I'll alias that to plot.halt()?)

[akhmerov]

A technological question: why do you use a widgetarea rather than using IPython.display.Javascript, or any analogous way to hook to the jupyter display system?

Also why do you implement the refresh loop in js rather than a thread/process?

Next, right now if I understand you redraw the plot completely, which renders plot widgets useless (the axes are reset on every refresh). An alternative would be to update the data.

Finally, as far as labels go, matplotlib now has an API for understanding labeled data: http://matplotlib.org/devdocs/users/whats_new.html#working-with-labeled-data-like-pandas-dataframes; it might be the best to use it.

[alexcjohnson]

A technological question: why do you use a widgetarea rather than using IPython.display.Javascript, or any analogous way to hook to the jupyter display system?

Maybe there's a better way, but this seemed like a standard and easy way to get bidirectional communication between python and javascript. It also generalizes easily to periodically updating anything displayed in the notebook.

Also why do you implement the refresh loop in js rather than a thread/process?

I didn't see an easy way to communicate from another python thread/process than the main one to the notebook. Also this way I don't have to start any new threads/processes, the javascript event loop is already there, and since the notebook page is the ultimate consumer of these refresh events that seemed like the natural choice.

Next, right now if I understand you redraw the plot completely, which renders plot widgets useless (the axes are reset on every refresh). An alternative would be to update the data.

Finally, as far as labels go, matplotlib now has an API for understanding labeled data: http://matplotlib.org/devdocs/users/whats_new.html#working-with-labeled-data-like-pandas-dataframes; it might be the best to use it.

Thanks, I'll take a look at both of these! I'm not a particularly experienced matplotlib user and find their documentation pretty horrible... particularly as relates to the interplay with jupyter. So anything you see me doing wrong / suboptimally by all means let me know!

[AdriaanRol]

@alexcjohnson
I don't have a self contained example as I have made the live plotting integrated in our measurement control but the following snippet should get it to work quite easily.

What it does is open a new window (not a cell in the notebook) in which we have our live plotting. I actually prefer this to having it within the notebook cell as it allows us to make best use of our multi-monitor setup.

import pyqtgraph as pg
import pyqtgraph.multiprocess as pgmp

# The following is within the init to ensure we don't have a memory leak by 
# repeatedly creating plot objects
        # starting the process for the pyqtgraph plotting
        # You do not want a new process to be created every time you start a run
        pg.mkQApp()
        self.proc = pgmp.QtProcess()  # pyqtgraph multiprocessing
        self.rpg = self.proc._import('pyqtgraph')

    def initialize_plot_monitor(self):
        self.win = self.rpg.GraphicsWindow(title='Plot monitor of %s' % self.name)
        self.win.resize(1000, 600)
        self.curves = []
        xlabels = self.column_names[0:len(self.sweep_function_names)]
        ylabels = self.column_names[len(self.sweep_function_names):]
        for xlab in xlabels:
            for ylab in ylabels:
                p = self.win.addPlot()
                p.setLabel('bottom', xlab)
                p.setLabel('left', ylab)
                c = p.plot(symbol='o')
                # symbol options are  o, s(quare), t(riangle), d(iamond), +
                self.curves.append(c)
            self.win.nextRow()
        return self.win, self.curves

    # We call this between every data point (super fast) 
    def update_plotmon(self):
        i = 0
        nr_sweep_funcs = len(self.sweep_function_names)
        for x_ind in range(nr_sweep_funcs):
            for y_ind in range(len(self.detector_function.value_names)):
                x = self.dset[:, x_ind]
                y = self.dset[:, nr_sweep_funcs+y_ind]
                self.curves[i].setData(x, y)
                i += 1

[akhmerov]

I just noticed that some of the warnings seem to be flushed to the stderr of the notebook server:

[W 13:00:31.464 NotebookApp] Saving untrusted notebook .../Qcodes/Qcodes example.ipynb
WARNING:root:negative delay -3.7e-05 sec
WARNING:root:negative delay -0.000109 sec
WARNING:root:negative delay -0.000158 sec
WARNING:root:negative delay -0.003438 sec
WARNING:root:negative delay -0.000206 sec
WARNING:root:negative delay -0.000897 sec
WARNING:root:negative delay -0.000968 sec
WARNING:root:negative delay -0.003585 sec
WARNING:root:negative delay -0.00056 sec
WARNING:root:negative delay -0.0002 sec
WARNING:root:negative delay -0.00145 sec
WARNING:root:negative delay -0.002627 sec
WARNING:root:negative delay -0.005242 sec
[I 13:18:31.341 NotebookApp] Saving file at .../Qcodes/Qcodes example.ipynb

Do you observe the same? (This is just after running the example notebook).

[alexcjohnson]

Yes, I see those errors (though not as much on my mac as on windows... I wonder if there's a time.time/time.clock issue I should sort out) - not quite sure when those are coming in, but they mean that somewhere in the measurement loop, something happening during the delay time is taking more time than it should.

BTW @akhmerov "update the data" turned out to be a much bigger headache than I expected - works easily for lines, but mpl somehow only knows how to autoscale to a heatmap for the first draw, not after that. But I think I finally have it sorted out.

[akhmerov]

I wonder if there's a time.time/time.clock issue I should sort out

I rather meant that under normal conditions user code shouldn't appear in the notebook server logs.

[alexcjohnson]

user code shouldn't appear in the notebook server logs.

True - these are warnings from a subprocess though, so they don't naturally come back to the notebook. Do you know of an easy way to push it back there? I suppose I could do something like have sys.stdout and sys.stderr in the subprocess dump into a multiprocessing.Queue, that gets periodically checked via an UpdateWidget tied to the cell where you originally ran the loop... that would be nice.

[alexcjohnson]

@AdriaanRol thanks for the snippet - I'll try that out this afternoon.

I actually prefer this to having it within the notebook cell as it allows us to make best use of our multi-monitor setup.

Yes, though it's nice to have both options - but perhaps we can have pyqtgraph dump its output into the notebook when you want that (though I guess you'd lose its interactivity at that point). For plots that work in a browser tab I was imagining getting the same effect by popping the output into a new window (which is a little bit of work to make them still live updating, but shouldn't be too hard).

Also, can I ask you about your workflow a bit? So far we've just been talking about the actual running of the experiment. But what do you do when you want to present data at a group meeting, or in a paper? Do you use the pyqtgraph graphs directly? Do you remake them in matplotlib or something else? One of the really nice things about the old Igor workflow is that the plots you take data in are already good enough to present at group meetings, and good enough to publish with just minor tweaks - and also lets you do better work than if you have to use poorer quality graphics just to get the right speed and interactivity. I'm not saying pyqtgraph is poorer quality, I haven't played with it yet, just want to understand what you actually do. (that said, it's suspicious to me that on pyqtgraph's homepage they imply you should use matplotlib if you want "very nice publication-quality graphics"...)

[MerlinSmiles]

@alexcjohnson @AdriaanRol
you might want to check out this quick discussion for inline pyqtgraph.
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/pyqtgraph/Nu921kIkeFk/LicfatT-3pwJ

[AdriaanRol]

Also, can I ask you about your workflow a bit?

In general we have a live-plotting monitor which provides decent looking but very simple graphs with high interactivity and speed (previously we used labview for this which IMO is not the best option). Once the experiment finishes we have an experiment specific analysis function we call that performs a default analysis and creates a nice looking figure.

I think the advantage of pyqtgraph is that it is really easy to create highly interactive and decent looking monitors (quite similar to the labview advantages now that I write them down).

I do very much like the IGOR workflow you describe with respect to the live plotting. However I would argue that matplotlib based plotting is not the way to go for that, for one it does not provide the speed to allow for decent interactivity (or I am missing something) and secondly the command interface is not the nicest. Maybe your plotly actually fits exactly in that sweet spot but I still have not used it (mostly because of the online server based plotting).

I think that developing a complete graphing package might be a bit out of scope for this project at this stage but who knows.

[alexcjohnson]

However I would argue that matplotlib based plotting is not the way to go for that

I agree - it seems to me that the update speed (the "live-ness") of the plot is OK with matplotlib, but the interactivity is pretty limited, confusing, and slow. And despite it being "the standard" python plotting library, even in my limited use of it I've already found it to be buggy and the interface quite heavy and nonintuitive.

I think everyone is going to want different things from their plotting... inline vs separate window, different kinds of interactivity needs, different emphasis on looks, and we're not going to be able to get it all from one package, at least not at the beginning. So what I'm going to try to do is implement the basics using several packages, in a way that each user can customize it in (as much as possible) any way that each package allows.

For matplotlib I think I'm there at least for inline plotting (haven't pushed the last bits yet... will soon), now I'm looking at pyqtgraph, after that I'll do plotly - which can now be used completely locally, no server connection required - and I expect can match or improve on pretty much everything matplotlib can do but I'm not sure yet about pyqtgraph in terms of speed and the range of interactivity.

I think that developing a complete graphing package might be a bit out of scope for this project at this stage but who knows.

Uh, yeah. I'd imagine that each of the three packages above has more than an order of magnitude more work put into it than we can devote to the whole of qcodes! Plotly has the advantage that I can add to it if we need something, but that's definitely a last resort, as it'll likely be a lot of work for something that's right on the edge of our scope.

[eendebakpt]

Matplotlib is fine for standard plotting, but for interactive or live plotting I find it insufficient. I have used pyqtgraph succesfully and it is good for live plotting. The main author of pyqtgraph has started a new package vispy (see http://vispy.org/). Vispy is fast and allows for usage of OpenGL, but it requires much more effort from the user.

Going for several options and letting the user choose seems like a good choice to me.

[alexcjohnson]

FYI as we were talking about licenses earlier (cc @alan-geller ), pyqtgraph requires Qt which is on LGPL, or requires commercial purchase... I think this is fine, our interface to pyqtgraph can still be MIT, and anyone who wants to use qcodes+pyqtgraph in the dark has options.

[alexcjohnson]

alright all you pyqtgraph fans, have a look. It was a royal pain getting pyqtgraph installed on my mac (for example: the latest version of Qt, 5.5, seems to be simply incompatible with python, though I think this only applies to macs, unless you rebuild it from source... but 5.4 works fine) but it's indeed faster and more interactive than the matplotlib version. I implemented them almost the same way, updates initiated by a javascript timer, though there are some obvious ways to speed it up later on:

• sync only changes to the data, not the whole DataSet
  • this would speed up the matplotlib version too.
• have the DataServer push data directly to the QtProcess, rather than having the main process drive this loop with a javascript timer. I didn't start out this way for two reasons:
  • it was easier to make this way, especially to debug the rather quirky and fragile pyqtgraph when the main process is controlling it
  • I quite like the fact that when you're live plotting, you have the live data available in the main process for inspection / analysis. Of course we could still do this separately but concurrently with the live plotting... and later that may be the way to go.

Anyway, as it stands now the pyqtgraph version is only in a separate window, and the matplotlib version is only in the notebook. Both of these could be made to work the other way too but this seems like a good stopping point for the first cut. I think unless there are objections I'll merge this, and come back sometime later to add more features (as well as more engines, like plotly).

[alexcjohnson]

silence is golden, I guess...

[guenp]

@alexcjohnson Sweet! I am busy atm taking data, will look at this asap :) I didn't have problems installing pyqtgraph on my mac, but somehow I couldn't get it to work on python 3.5.. couldn't install a compatible version of PyQt4. It did work with 3.3 though.

[guenp]

@alexcjohnson alright, I checked it out. Installing pyqt for python 3.5 worked with brew and anaconda. It looks great! I made a repo with some additions to pyqtgraph, like buttons and widgets such that you can see a snapshot of a graph in-line in the ipython notebook, which could be useful for Qcodes. It's here: https://github.com/guenp/colorgraphs

[alexcjohnson]

@guenp cool! I've got to spend a little time on drivers, to get folks here up and running, but can take a look after that. Of course, any time you want to make a PR into Qcodes... 😄

[guenp]

@alexcjohnson sweet! I'm excited to see this up and running soon & can def do a PR :)