Incorporate Plotly JSON to metamodel (Closes #1745) #1747

umesh-timalsina · 2020-07-08T17:37:59Z

Checklist:

Update pipeline and project seeds
Change backend_deepforge to plotly_backend for plots
Change ExecutionIndex and PlotlyGraph panels/ widget to incorporate multiple plotly Graphs instead of instead of combining them (Could be changed)
Update plotly_backend to incorporate 3DScatter and 3DLine Plots
Update Tests to incorporate changed pipeline seeds.
Add Change metamodel for plotly related Updates.

1. Increase Timeout ExecutePipeline.spec.js 2. Update devProject seed to adhere to new MetaModel

1. Remove dependencies in SubplotsOperation in devProject.webgmex 2. Remove src/seeds/project/releases.json 3. Revert timeout in ExecutePipeline.spec.js

umesh-timalsina · 2020-07-13T13:30:45Z

src/plugins/GenerateJob/templates/plotly_backend.py

+    renderer = DeepforgePlotlyRenderer()
+    exporter = mplexporter.Exporter(renderer)
+    exporter.run(fig)
+    renderer.crawl_3d_labels(fig)


renderer.resize() method removes width, height and other properties for a plotly figure, which is fine. But, this is already done in PlotyGraphWidget. Can be done here as well.

By default, PlotlyRenderer tries its hardest to precisely mimic an mpl figure. However, plotly is pretty good with aesthetics. By running PlotlyRenderer.resize(), layout parameters are deleted. This lets plotly choose them instead of mpl.

brollb

Nice work! A couple minor things then it should be good!

brollb · 2020-07-15T15:55:37Z

src/visualizers/widgets/PlotlyGraph/PlotlyGraphWidget.js

@@ -23,8 +25,9 @@ define([
        this.$el.css('overflow', 'auto');
        this.$el.addClass(WIDGET_CLASS);
        this.nodes = {};
-        this.plotlyJSON = null;
+        this.plotlyJSONS = null;


As a pretty minor comment, this should probably be this.plotlyJSONs.

Actually, it's not clear to me that it needs to be saved on this. If possible, it would be preferable to avoid this extra statefulness and simply pass the the plotly JSON(s) to the functions that require them.

Also, this.nodes, this.layout, this.created, and this._container appear to be unused (or unnecessary).

brollb · 2020-07-15T16:13:23Z

src/plugins/ExecuteJob/metadata/Figure.js

-                }
-                this.addAxesScatterPoints(axesNode, this.node, axes);
-            });
+            this.core.setAttribute(this.node, 'data', state);


I believe the data attribute is a string, so it would be good to JSON.stringify(state). This also makes it so we can view the contents in the default webgme UI as it cannot show arbitrary JSON.

1. Merge Branch Master 2. Apply updates to the new Metamodel

brollb · 2020-07-21T15:31:21Z

src/common/updates/Updates.js

@@ -21,7 +35,7 @@ define([
                    });
            },
            apply: function(core, rootNode, META) {
-                // Create 'MyUtilities' node
+                // Create 'MyUtilities' nodeupdate


why was this changed? It seems like it was probably unintentional.

brollb · 2020-07-21T15:32:13Z

src/common/updates/Updates.js

+                    const versionString = core.getAttribute(pipelineRoot, 'version');
+                    const version = new Version(versionString);
+                    return version.lessThan(new Version('0.22.0')) &&
+                           version.greaterThan(new Version('0.19.1'));


What if it is below 0.19.1?

The changes to the metamodel i.e. adding Subgraphs, Support for 3D plots and other metanodes like [Line, ScatterPoints, etc...] was introduced in pipeline seed version 0.19.1.

brollb · 2020-07-21T15:34:56Z

src/common/updates/Updates.js

    Version,
    Q
 ) {
+    const GRAPH = 'Graph';
+    const getGraphNode = async function(core, rootNode, graphNodes={}) {


rootNode only appears to be the root node on the first call. This should probably be node instead.

Also, why are you setting the graph nodes in a dictionary rather than simply returning a list of graph nodes from the given subtree?

It would probably be good to rename the function to getGraphNodes or something so it is obvious that it returns multiple nodes.

brollb · 2020-07-21T15:40:07Z

src/common/updates/Updates.js

+            apply: async function(core, rootNode, META) {
+                let graphNodes = {};
+                await getGraphNode(core, rootNode, graphNodes);
+                const graphNodeKeys = Object.keys(graphNodes);


It doesn't look like you use graphNodeKeys for anything but fetching nodes from the graphNodes dictionary. A list would probably be more fitting.

brollb · 2020-07-21T15:51:22Z

src/seeds/pipeline/releases.jsonl

@@ -2,3 +2,5 @@
 {"version":"0.21.0","changelog":"Add provenance to metadata (via WithProvenance mixin)"}
 {"version":"0.21.1","changelog":"Update Inheritance of Subgraph, Line, Images, ScatterPoints etc.. nodes"}

+
+{"version":"0.22.0","changelog":"Incorporate PlotlyJSON into Graph metaNode"}


minor thing but this should probably be "meta node" rather than camel case.

brollb · 2020-07-21T15:58:58Z

src/visualizers/widgets/PlotlyGraph/PlotlyGraphWidget.js

            this.deleteChart();
        } else {
-            if (!this.created && !_.isEmpty(this.plotlyJSON)) {
-                Plotly.newPlot(this.$el[0], this.plotlyJSON);
+            if (!this.created && !_.isEmpty(plotlyJSONs)) {


I don't think this.created is needed, is it?

We can probably also remove _.isEmpty check. Is there a reason why this should ever be the case? If not, this is just preventing it from failing fast.

umesh-timalsina · 2020-07-21T16:40:15Z

Currently, matplotlib 3.3.0 is out. This removed method is_frame_like function(used by plotly renderer) for matplotlib.spines.Spine , which is causing any plot operation to fail.

ERROR conda.cli.main_run:execute(32): Subprocess for 'conda run ['python', 'main.py']' command failed.  (See above for error)
Traceback (most recent call last):
  File "main.py", line 31, in <module>
    result = operation.execute()
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/operations/plots_pipeline.py", line 28, in execute
    plt.show()
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/matplotlib/pyplot.py", line 336, in show
    return _backend_mod.show(*args, **kwargs)
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/plotly_backend.py", line 638, in show
    manager.canvas.send_deepforge_update()
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/plotly_backend.py", line 694, in send_deepforge_update
    state = self.figure_to_state()
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/plotly_backend.py", line 702, in figure_to_state
    figure
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/plotly_backend.py", line 206, in mpl_to_plotly
    exporter.run(fig)
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/mplexporter/exporter.py", line 51, in run
    self.crawl_fig(fig)
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/mplexporter/exporter.py", line 118, in crawl_fig
    self.crawl_ax(ax)
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/mplexporter/exporter.py", line 123, in crawl_ax
    props=utils.get_axes_properties(ax)):
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/mplexporter/renderers/base.py", line 57, in draw_axes
    self.open_axes(ax=ax, props=props)
  File "/tmp/deepforge-local-exec-755318d95a65cc2ca6deb6fab5343cf9bb763c36/plotly_backend.py", line 422, in open_axes
    super().open_axes(ax, props)
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/renderer.py", line 169, in open_axes
    bottom_spine = mpltools.get_spine_visible(ax, "bottom")
  File "/home/umesh/anaconda3/envs/deepforge_124/lib/python3.7/site-packages/plotly/matplotlylib/mpltools.py", line 368, in get_spine_visible
    spine_frame_like = spine.is_frame_like()
AttributeError: 'Spine' object has no attribute 'is_frame_like'

brollb · 2020-07-21T16:47:46Z

I would just leave updating to matplotlib 3.3.0 for another PR rather than letting this PR grow to include separate concerns.

umesh-timalsina · 2020-07-21T17:07:15Z

Are you suggesting we pin matplotlib to version 3.2.2 for this PR?

1. Pin matplotlib to v3.2.2 in environment.worker.yml and devProject 2. Remove this.created in PlotlyGraphWidget 3. Fix releases.jsonl 4. Rename getGraphNode to getGraphNodes in updates.js, use array instead of dictionary 5. Remove extra line in ExecuteJob.spec.js

-- In version 0.21.1 of the pipeline library, Inheritance of the old SubGraph and below was changed from FCO to Metadata (because of provenance). So, the current Extractor to extract plotlyJSON expects this inheritance. For older seeds, a function to add META['pipeline.Metadata'] as a Mixin is added to the updating capabilities. -- remove wrongfully pased comment

umesh-timalsina · 2020-07-23T21:35:55Z

The following gist contains different plots and corresponding deepforge operations that generated them. Cuurrently, Piecharts are not supported. This should be maintained as a separate issue once #1747 is merged.

https://gist.github.com/umesh-timalsina/7a5ba1fa6b7971d632f879b2733f8c95

brollb · 2020-07-24T13:31:00Z

I have been testing this and ran into the following exception when switching to the ExecutionIndex view:

It might be nice to make that code a little more pessimistic. (I am not sure of the underlying cause of the issue but it is trying to parse undefined.)

brollb · 2020-07-24T14:38:19Z

Another minor issue. It's a little annoying that when I have only a single plot shown that it doesn't take the whole space like before:
Before:

After:

That said, I like how it looks when showing multiple plots:

brollb · 2020-07-24T14:43:39Z

Other than that, it looks good!

umesh-timalsina · 2020-07-24T19:23:43Z

I have been testing this and ran into the following exception when switching to the ExecutionIndex view:

It might be nice to make that code a little more pessimistic. (I am not sure of the underlying cause of the issue but it is trying to parse undefined.)

I am not able to replicate this error. I have changed the code such that it checks if node.getAttribute('data') is truthy.
Also, I have changed the area for a single plot to take the full area of the container.

…bute

brollb · 2020-07-25T00:51:17Z

Nice work, @umesh-timalsina!

umesh-timalsina added 8 commits July 7, 2020 16:13

WIP- Inital Version of plotly json in metamodel

d4bd938

WIP- update devProject; increate timeout for devProject

6d9a60d

1. Increase Timeout ExecutePipeline.spec.js 2. Update devProject seed to adhere to new MetaModel

WIP- Fix node id from test seeds

3a9d140

Fix releases.jsonl

578e533

WIP- Remove Line node creation from ExecuteJob.spec.js

f40d843

Fix timeout in ExecutePipeline.spec.js and fix releases.jsonl

3e444f9

1. Remove dependencies in SubplotsOperation in devProject.webgmex 2. Remove src/seeds/project/releases.json 3. Revert timeout in ExecutePipeline.spec.js

Merge remote-tracking branch 'origin/master' into 1745-ploty-metamodel

f070af0

WIP-Add 3D support for DeepforgePlotlyRenderer

ce2807b

umesh-timalsina marked this pull request as ready for review July 10, 2020 21:55

umesh-timalsina requested a review from brollb July 10, 2020 21:55

umesh-timalsina added 3 commits July 10, 2020 17:28

WIP- Fix np masked array imports in plotly_backend.py

fbfc9ab

Merge remote-tracking branch 'origin/master' into 1745-ploty-metamodel

183dfb8

WIP-Refactor fix HelloWorld Job execution ID

4469b02

umesh-timalsina commented Jul 13, 2020

View reviewed changes

WIP- Increase timeout for sciserver to 10 minutes

b06bd55

This was referenced Jul 13, 2020

Refactor FigureExtractor.js to run independent of client Instance #1760

Closed

Default installed packages in workers #1761

Open

brollb mentioned this pull request Jul 15, 2020

Add initial version of interactive dataset explorer. Closes #1692 #1694

Merged

3 tasks

brollb requested changes Jul 15, 2020

View reviewed changes

This was referenced Jul 16, 2020

An alternative visualizer for pipeline execution view #1766

Open

Refactor FigureExtractor to have client/core features. Closes #1760 #1771

Merged

Forking when applying updates #1759

Open

Merge remote-tracking branch 'origin/master', Fix Updates

6d6a287

1. Merge Branch Master 2. Apply updates to the new Metamodel

brollb requested changes Jul 21, 2020

View reviewed changes

brollb mentioned this pull request Jul 21, 2020

Suboptimal workaround for forking when applying updates. #1785

Merged

Merge remote-tracking branch 'origin/master' into 1745-ploty-metamodel

74e8443

brollb mentioned this pull request Jul 21, 2020

Enable migrations to run before library updates #1788

Closed

This was referenced Jul 22, 2020

Parse Legends For a Matplotlib Figure #1551

Closed

Switching backends in matplotlib 3.3.0 fails #1792

Closed

umesh-timalsina added 2 commits July 22, 2020 11:55

merge branch master

a1579a7

WIP- Add beforeLibraryUpdates flag to UpdateGraphContainment

ac12c61

This was referenced Jul 22, 2020

Reorder matplotlib backend to run figure.draw() before self.send_deepforge_update() #1799

Closed

AttributeError: module 'matplotlib' has no attribute 'backends' matplotlib/matplotlib#18022

Closed

umesh-timalsina added 2 commits July 23, 2020 10:13

Merge remote-tracking branch 'origin/master' into 1745-ploty-metamodel

2340637

brollb mentioned this pull request Jul 23, 2020

Added VisPred descriptions and fixed incorrect image #1801

Merged

umesh-timalsina added 4 commits July 23, 2020 16:10

Filter provenance nodes from mixin application; Add fix for #1799

4402371

WIP- Fix version check in UpdateGraphContainment

ff7d087

WIP- Cover cases for layout.title as object

4d1e0fa

WIP- displayTitle fix for undefined cases

896b7a2

WIP- adjust height of a single plot; check falsy values in data attri…

9df083a

…bute

brollb approved these changes Jul 25, 2020

View reviewed changes

brollb merged commit 12608bc into master Jul 25, 2020

brollb deleted the 1745-ploty-metamodel branch July 25, 2020 00:51

This was referenced Jul 25, 2020

Document usage of matplotlib, plotly, supported versions of plots in DeepForge #1393

Open

PlotlyGraph Widget's Container's height defaults to zero #1805

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporate Plotly JSON to metamodel (Closes #1745) #1747

Incorporate Plotly JSON to metamodel (Closes #1745) #1747

umesh-timalsina commented Jul 8, 2020 •

edited

Loading

umesh-timalsina Jul 13, 2020 •

edited

Loading

brollb left a comment

brollb Jul 15, 2020

brollb Jul 15, 2020

brollb Jul 15, 2020

brollb Jul 15, 2020

brollb Jul 21, 2020

brollb Jul 21, 2020

umesh-timalsina Jul 23, 2020 •

edited

Loading

brollb Jul 21, 2020

brollb Jul 21, 2020

brollb Jul 21, 2020

brollb Jul 21, 2020

brollb Jul 21, 2020

brollb Jul 21, 2020

umesh-timalsina commented Jul 21, 2020 •

edited

Loading

brollb commented Jul 21, 2020

umesh-timalsina commented Jul 21, 2020

umesh-timalsina commented Jul 23, 2020 •

edited

Loading

brollb commented Jul 24, 2020 •

edited

Loading

brollb commented Jul 24, 2020

brollb commented Jul 24, 2020

umesh-timalsina commented Jul 24, 2020 •

edited

Loading

brollb commented Jul 25, 2020

Incorporate Plotly JSON to metamodel (Closes #1745) #1747

Incorporate Plotly JSON to metamodel (Closes #1745) #1747

Conversation

umesh-timalsina commented Jul 8, 2020 • edited Loading

umesh-timalsina Jul 13, 2020 • edited Loading

Choose a reason for hiding this comment

brollb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umesh-timalsina Jul 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umesh-timalsina commented Jul 21, 2020 • edited Loading

brollb commented Jul 21, 2020

umesh-timalsina commented Jul 21, 2020

umesh-timalsina commented Jul 23, 2020 • edited Loading

brollb commented Jul 24, 2020 • edited Loading

brollb commented Jul 24, 2020

brollb commented Jul 24, 2020

umesh-timalsina commented Jul 24, 2020 • edited Loading

brollb commented Jul 25, 2020

umesh-timalsina commented Jul 8, 2020 •

edited

Loading

umesh-timalsina Jul 13, 2020 •

edited

Loading

umesh-timalsina Jul 23, 2020 •

edited

Loading

umesh-timalsina commented Jul 21, 2020 •

edited

Loading

umesh-timalsina commented Jul 23, 2020 •

edited

Loading

brollb commented Jul 24, 2020 •

edited

Loading

umesh-timalsina commented Jul 24, 2020 •

edited

Loading