[ENH] Move Louvain clustering from prototypes to core #3111

pavlin-policar · 2018-07-03T09:36:48Z

Issue

Fixes #3110

Description of changes

Move Louvain clustering widget from prototypes

Includes

Code changes
Tests
Documentation

Fixes biolab#3110

lanzagar · 2018-07-06T09:52:00Z

Orange/widgets/unsupervised/owlouvainclustering.py

+_DEFAULT_K_NEIGHBORS = 30
+
+
+METRICS = [('Euclidean', 'l2'), ('Manhattan', 'l1')]


Are these the only two metrics that are supported?

For the time being, only these two metrics are supported because of the way sklearn computes nearest neighbours. The only other metrics I had considered adding was the cosine, pearsonr and spearmanr distances, but unfortunately the fast tree methods for finding nearest neighbours in sklearn don't support these and fall back to the O(n^2) brute force approach, which is prohibitively expensive for larger data sets, so we decided it is better to leave them out.

If you feel any of the other supported metrics should be added, they are listed in the source: KDTree and BallTree. Most of these don't appear anywhere else in Orange, so including them doesn't really make sense.

lanzagar · 2018-07-06T09:52:05Z

Orange/widgets/unsupervised/owlouvainclustering.py

+        self._invalidate_graph()
+        self.commit()
+
+    def _update_k_neighbors(self):


As long as you do not expect these methods will be expanded in the future with code specific to changing one of the settings I don't see a reason to have 3 methods with the same body but different names.
I think having just one _update_graph and using it as the callback for all 3 controls would be better.

lanzagar · 2018-07-06T09:57:00Z

Orange/clustering/louvain.py

+
+import Orange
+from Orange.data import Table
+from community import best_partition


I guess this import belongs to the above group of external libraries now

lanzagar · 2018-07-06T10:11:47Z

The widget works well as far as I was able to test it.

What I am missing is documentation - at least a stub with something completely basic that can be expanded later.
Also, I was wondering what numbers to set for Resolution and was thinking that a tooltip explaining what I am doing there wouldn't be bad either. But maybe when the widget has some documentation, that will be enough.

Another thought was, why not a slider for Resolution like for PCA components. If anything I would sooner expect a spinbox for the number of components (everybody understands the number, I know what I am doing when setting e.g. 5 or 20). For Resolution, I have no idea what 0.5 means and how much bigger 1.5 is or 2 (or why 5 is the max). So a resolution slider from smaller scale communities to larger scale could make more sense.

lanzagar · 2018-07-06T10:13:34Z

Oh, and the only issue I have with the functionality is that the pca slider now commits on any change so dragging it starts a lot of computations and isn't smooth. Commit should happen when I stop dragging.

…te slider on mouse release

…date slider

codecov-io · 2018-07-17T07:44:07Z

Codecov Report

Merging #3111 into master will increase coverage by 0.05%.
The diff coverage is 92.55%.

@@            Coverage Diff             @@
##           master    #3111      +/-   ##
==========================================
+ Coverage   82.54%   82.59%   +0.05%     
==========================================
  Files         337      340       +3     
  Lines       58431    58767     +336     
==========================================
+ Hits        48229    48540     +311     
- Misses      10202    10227      +25

pavlin-policar · 2018-07-17T07:53:33Z

Using sliders makes more sense, I agree. I've changed it now. As far as the documentation goes, I can write some documentation, but I can't for the life of me figure out how to incorporate that into Orange.

ajdapretnar · 2018-07-17T08:02:23Z

Ajda to the rescue! 🏇
Here is where you put the .rst file of your widget: https://github.com/biolab/orange3/tree/master/doc/visual-programming/source/widgets
Best thing is to copy an existing file and just change the content, so you keep the form. Then just include it in your PR. You will have to run make html to see how the output looks like.
Also important:

stamper for numbers on images can be found here: https://github.com/biolab/orange-tools
in that same repo you will find trim.py that removes a background of the image (say for complex workflows)
you will have to index images for tests to pass (I recommend Gimp)

If you have any questions, just drop by. I would be happy if you just provide the key content, I can later take care of the rest.

pavlin-policar · 2018-07-17T08:37:26Z

Great, thanks! I had no idea this existed.

I've added a fairly minimal stub explaining what the controls do and how the widget works. I've tried to make the stamping consistent with the other widgets. Let me know if I should add anything or anything is poorly worded.

ajdapretnar

Added some small comments to the documentation. You can opt not to include the example as the widget is easy to use, but perhaps it could be nice if an expert did it. :) However, it is best if you run make html in the visual-programming folder to inspect where the documentation needs to be fixed. The log will give you hints.

Another a small note: it would be nice to include a report option in the widget, too. Just implement send_report function and report on parameters.

ajdapretnar · 2018-07-17T09:36:27Z

doc/visual-programming/source/widgets/unsupervised/louvainclustering.rst

@@ -0,0 +1,38 @@
+Louvain Clustering
+=======


You need to extend the signs to encompass at minimum the entire title.

ajdapretnar · 2018-07-17T09:37:31Z

doc/visual-programming/source/widgets/unsupervised/louvainclustering.rst

+
+.. figure:: images/Louvain-stamped.png
+
+1. PCA processing is typically be applied to the original data to remove noise.


typically be applied

ajdapretnar · 2018-07-17T09:38:41Z

doc/visual-programming/source/widgets/unsupervised/louvainclustering.rst

+   values recover clusters containing more data points.
+5. When *Apply Automatically* is ticked, the widget will automatically
+   communicate all changes. Alternatively, click *Apply*.
+


At the bottom it is great to provide a minimum workflow for the widget. Or a cool way how to use it, preferably on a simple data set that exists in Orange.

pavlin-policar · 2018-07-19T12:27:53Z

@ajdapretnar I've added the report and a simple example showing off integration with the Network addon. I also don't know what indexing images is, or if it's a problem in this case - if it is I can look into it. Is this all right?

ajdapretnar · 2018-07-19T12:58:56Z

🤔 I think tests won't pass if images aren't indexed. This is what I had in mind. It is as simple as opening Gimp and using Image - Mode - Indexed (Convert). Then File - Export As (and click all the Export buttons that appear). This is what you add then to the repo.

pavlin-policar · 2018-07-23T13:36:35Z

@lanzagar I've fixed the images, all the tests are passing. The only thing remaining are some pointless pylint errors. Should I try to fix them and add ignore statements where necessary or is this fine?

lanzagar · 2018-07-23T14:24:32Z

Most of the pylint errors seem to be import related and easy to fix and improve the code.
There is one bare-except that probably should be except ImportError:.
The cell-var-from-loop warning might not be relevant in this case, but is one of those whatthehellishappening python things and even coupled with threading here - so I suggest to make it as clear and bug proof as possible :)
The only "pointless" warning that should probably be ignored is the broad-except (we actually want it) - a pylint ignore comment wouldn't be bad, but a single pylint warning is acceptable too.

astaric · 2018-07-23T16:18:16Z

Orange/widgets/unsupervised/owlouvainclustering.py

+                current_progress = idx / num_tasks
+                # How much progress can each task contribute to the total
+                # work to be done
+                task_percentage = len(self.__tasks) ** -1


What is wrong with 1 / len(self.__tasks)? :)

I think I wrote this code at a point where I was doing a considerable amount of calculus and I guess I found this clearer at that point.

pavlin-policar · 2018-07-26T09:36:43Z

@lanzagar The only remaining issue is the coverage. I've written all the tests that I think are reasonable for this widget, and I think most of the code is hit at least once, so I'm not sure where this poor coverage comes from. Should I look into this or can this be merged?

astaric · 2018-07-26T09:38:44Z

https://codecov.io/gh/biolab/orange3/compare/56fdfc936d0f8294a335e1ef9ff1f8889ffc0bbc...1e705f9a629ee488143b46ed6ffef7ce824c75a3/diff

lanzagar · 2018-07-26T13:24:58Z

I would like to merge this tomorrow if possible. If you can, take a look at the diff from @astaric and add a test or two to cover the bigger red parts. While I know that saying tests can also be added later, after this is merged, is not good practice, but I also wouldn't like to leave this PR open too long. It would be nice if it gets some use before the release and really should be in the next one.

pavlin-policar added 4 commits July 3, 2018 11:32

Move Louvain clustering from prototypes to core

29468a7

Fixes biolab#3110

OWLouvain: Move louvain library to own file, pylint fixes

d191e9e

OWLouvain: Remove failing doctests from library

e4c854b

OWLouvain: Add library to requirements and remove code

b10861a

pavlin-policar requested a review from lanzagar July 3, 2018 11:20

BlazZupan self-requested a review July 6, 2018 07:10

lanzagar reviewed Jul 6, 2018

View reviewed changes

pavlin-policar added 3 commits July 17, 2018 09:10

OWLouvain: Fixup code, replace resolution spinbox with slider, activa…

796a794

…te slider on mouse release

OWLouvain: Use tracking instead of mouse release event to properly up…

6c5231c

…date slider

OWLouvain: Fix crash when changing paramters too quickly

e5e3796

OWLouvain: Add documentation stub

15802fb

pavlin-policar force-pushed the louvain-clustering branch from cf0cdb3 to 15802fb Compare July 17, 2018 08:34

ajdapretnar reviewed Jul 17, 2018

View reviewed changes

pavlin-policar added 2 commits July 17, 2018 21:48

OWLouvain: Add option to send basic report

0af7f61

OWLouvain: Add example to documentation

3687c0d

OWLouvain: Index documentation images

311c1cd

astaric reviewed Jul 23, 2018

View reviewed changes

OWLouvain: Make pylint happy

1e705f9

pavlin-policar force-pushed the louvain-clustering branch from 1beb225 to 1e705f9 Compare July 25, 2018 07:51

pavlin-policar force-pushed the louvain-clustering branch from 3d27c52 to c8e9c64 Compare July 31, 2018 07:58

OWLouvain: Add test for adding and removing data

86ce71b

pavlin-policar force-pushed the louvain-clustering branch from c8e9c64 to 86ce71b Compare July 31, 2018 07:59

OWLouvain: Fix tests not waiting for commit to finish

6a0c36f

pavlin-policar force-pushed the louvain-clustering branch from fdf0556 to 6a0c36f Compare August 1, 2018 17:09

lanzagar merged commit 68d10cb into biolab:master Aug 2, 2018

pavlin-policar deleted the louvain-clustering branch November 7, 2018 11:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Move Louvain clustering from prototypes to core #3111

[ENH] Move Louvain clustering from prototypes to core #3111

pavlin-policar commented Jul 3, 2018

lanzagar Jul 6, 2018

pavlin-policar Jul 17, 2018

lanzagar Jul 6, 2018

lanzagar Jul 6, 2018

lanzagar commented Jul 6, 2018

lanzagar commented Jul 6, 2018

codecov-io commented Jul 17, 2018 •

edited

Loading

pavlin-policar commented Jul 17, 2018

ajdapretnar commented Jul 17, 2018 •

edited

Loading

pavlin-policar commented Jul 17, 2018

ajdapretnar left a comment

ajdapretnar Jul 17, 2018

ajdapretnar Jul 17, 2018

ajdapretnar Jul 17, 2018

pavlin-policar commented Jul 19, 2018

ajdapretnar commented Jul 19, 2018

pavlin-policar commented Jul 23, 2018

lanzagar commented Jul 23, 2018

astaric Jul 23, 2018

pavlin-policar Jul 25, 2018

pavlin-policar commented Jul 26, 2018

astaric commented Jul 26, 2018

lanzagar commented Jul 26, 2018

		_DEFAULT_K_NEIGHBORS = 30


		METRICS = [('Euclidean', 'l2'), ('Manhattan', 'l1')]


		.. figure:: images/Louvain-stamped.png

		1. PCA processing is typically be applied to the original data to remove noise.

[ENH] Move Louvain clustering from prototypes to core #3111

[ENH] Move Louvain clustering from prototypes to core #3111

Conversation

pavlin-policar commented Jul 3, 2018

Issue

Description of changes

Includes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lanzagar commented Jul 6, 2018

lanzagar commented Jul 6, 2018

codecov-io commented Jul 17, 2018 • edited Loading

Codecov Report

pavlin-policar commented Jul 17, 2018

ajdapretnar commented Jul 17, 2018 • edited Loading

pavlin-policar commented Jul 17, 2018

ajdapretnar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavlin-policar commented Jul 19, 2018

ajdapretnar commented Jul 19, 2018

pavlin-policar commented Jul 23, 2018

lanzagar commented Jul 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavlin-policar commented Jul 26, 2018

astaric commented Jul 26, 2018

lanzagar commented Jul 26, 2018

codecov-io commented Jul 17, 2018 •

edited

Loading

ajdapretnar commented Jul 17, 2018 •

edited

Loading