[MRG] Wikicorpus custom filters #2089

mattilyra · 2018-06-13T10:38:09Z

I wanted to add a simple method for filtering out wiki articles based on arbitrary rules. The idea is to be able to easily produce datasets like the WikiText-103.

I am currently working on making a German equivalent of that corpus (call it DeWiki103 for now), but that requires filtering out articles based on regular expressions applied to the text of each XML element (short of parsing the Wikimedia format).

This PR adds a simple filter_articles method to WikiCorpus that gets called in extract_pages. Returning None will filter out that specific article.

Let me know if you think this might be a useful addition.

…ssed in as kwargs

piskvorky · 2018-06-17T10:14:33Z

gensim/corpora/wikicorpus.py

@@ -380,6 +380,13 @@ def extract_pages(f, filter_namespaces=False):
                if ns not in filter_namespaces:
                    text = None

+            if callable(filter_articles) and text:


What if filter_articles isn't a callable? If I understand correctly, it should be a callable always, no need to check.

That's just catching the default case where filter_articles=None - I didn't see the point of introducing a dummy callable that always returns true. I guess you could make the default filter_articles=False but that confuses the type semantics of filter_articles is it bool or a callable

is not None preferred (since that's what we really want to test). But a dummy filter_articles=gensim.utils.identity is also OK.

piskvorky · 2018-06-17T10:16:36Z

Definitely useful, thanks @mattilyra ! And I like the solution (callable injection): simple and flexible 👍

piskvorky · 2018-06-17T10:18:03Z

gensim/corpora/wikicorpus.py

+        filter_articles: callable, optional
+            If set each XML article element will be passed to this callable before being processed. Only articles
+            where the callable returns an XML element are processed, returning None allows filtering out
+            some articles based on customised rules.


Are there any restrictions on the callable? Can it be an anonymous lambda function? (troublesome with parallelized multiprocessing -- lambdas cannot be pickled)

True, for the parallel self.process to work filter_articles needs to be serialisable, what's the practice for checking this in gensim? Try to pickle the callable and raise an informative exception if that fails? Is pickle the standard serialiser in gensim?

On a related "fun" note, see https://stackoverflow.com/questions/50328386/python-typing-pickle-and-serialisation

I was thinking a simple note in the docs. What does the error look like now, if a user passes something not pickleable? If it looks reasonable, I'd just keep that.

@menshikh-iv what is our policy here in general?

Well there's no specific catch for this so it'll just be python's PicklingError - that plus a note in the docs I think should be enough, but I don't know how this is handled elsewhere in gensim

I think so too. Someone advanced enough to be setting this parameter better be advanced enough to RTFM :)

…ed default value check to check against None not against callable().

piskvorky · 2018-06-18T14:20:54Z

If the tests pass, LGTM to me! Thanks Matti.

Let's wait for @menshikh-iv 's final verdict & merge.

menshikh-iv

Good work @mattilyra, please add a test for this case (check that filter function works as expected), you can use https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/test/test_data/enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2 as wiki dump with this code

from gensim.test.utils import datapath
from gensim.corpora import WikiCorpus

path = datapath("enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2")
corpus = WikiCorpus(path)

menshikh-iv · 2018-06-19T05:22:28Z

gensim/corpora/wikicorpus.py

@@ -544,10 +555,15 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
            Maximal token length.
        lower : bool, optional
             If True - convert all text to lower case.
+        filter_articles: callable or None, optional


callable, optional

menshikh-iv · 2018-06-19T05:25:57Z

gensim/corpora/wikicorpus.py

+        filter_articles: callable or None, optional
+            If set, each XML article element will be passed to this callable before being processed. Only articles
+            where the callable returns an XML element are processed, returning None allows filtering out
+            some articles based on customised rules.


Can you create an example of this function and link it in docstring like

:func:`gensim.corpora.wikicorpus.filter_example`

this can be useful for better understanding, how function works

input parameters

return values (probably True/False better that XML element/None)

you mean add an example filter_example to the wikicorpus module?

yes, only as example (no need to use it by default)

got it, very busy at work but will try to get it done this week

don't forget to resolve merge conflict too

…us_filters

mattilyra · 2018-07-09T14:01:21Z

@menshikh-iv ERROR: InvocationError for command '/home/travis/build/RaRe-Technologies/gensim/.tox/py36-linux/bin/pytest gensim/test' (exited with code 1) ????

piskvorky

LGTM after the minor code style fix (ident).

piskvorky · 2018-07-11T11:38:58Z

gensim/corpora/wikicorpus.py

@@ -365,6 +426,13 @@ def extract_pages(f, filter_namespaces=False):
                if ns not in filter_namespaces:
                    text = None

+            if filter_articles is not None:
+                if not filter_articles(elem, namespace=namespace, title=title,
+                                       text=text, page_tag=page_tag,


We use hanging indent in Gensim (not vertical alignment): https://www.python.org/dev/peps/pep-0008/#id3

Why? Makes code easier to maintain (people forget to re-indent after modifying the opening line). Plus some people (me!) also find it more consistent and easier to read :)

mattilyra · 2018-07-11T13:29:21Z

Have to respect properly enforced coding style 😄 👍

is there a gensim code style guide somewhere, I didn't see anything in the contribution guidelines and the linter does not complain about that.

piskvorky · 2018-07-11T18:50:50Z

Awesome, thanks.

Both vertical and hanging indents are OK by Python standards (PEP8), so linters wouldn't necessarly complain. Consistent hanging indent is just our project choice.

We keep developer instructions here on Github: https://github.com/RaRe-Technologies/gensim/wiki/Developer-page#code-style

piskvorky · 2018-07-13T19:39:24Z

Awesome job, thanks @mattilyra !

mattilyra added 3 commits June 13, 2018 10:30

Modified call to user provided filter_article so that all args are pa…

cb02f55

…ssed in as kwargs

Merge branch 'develop' into feature/wikicorpus_filters

9b8491e

Small readability improvements.

ef7453e

piskvorky reviewed Jun 17, 2018

View reviewed changes

mattilyra added 3 commits June 18, 2018 10:29

Added a catch for pickling errors for self.filter_function, and chang…

ff9002f

…ed default value check to check against None not against callable().

Fixed python 2 backwards compatibility issue with exception chaining.

e0012ca

Added import for PicklingError

0c97982

piskvorky changed the title ~~Wikicorpus custom filters (WIP)~~ [MRG] Wikicorpus custom filters Jun 18, 2018

Fixed linting errors.

d19a2e0

menshikh-iv suggested changes Jun 19, 2018

View reviewed changes

mattilyra added 6 commits July 5, 2018 09:01

Added an example filter function.

882e699

Resolved merge conflict with upstream develop.

b81851a

Added tests for wikicorpus filter_articles.

295f4ce

Passing tests for wikicorpus filter_articles.

5610c36

Merge remote-tracking branch 'upstream/develop' into feature/wikicorp…

224fb28

…us_filters

Fixed linting error.

bfac0c2

Fixed linting errors.

cd03fd0

piskvorky requested changes Jul 11, 2018

View reviewed changes

PEP8 hanging indent.

353de1f

piskvorky approved these changes Jul 13, 2018

View reviewed changes

piskvorky merged commit 46ccefb into piskvorky:develop Jul 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG] Wikicorpus custom filters #2089

[MRG] Wikicorpus custom filters #2089

mattilyra commented Jun 13, 2018 •

edited

Loading

piskvorky Jun 17, 2018

mattilyra Jun 17, 2018

piskvorky Jun 17, 2018

piskvorky commented Jun 17, 2018 •

edited

Loading

piskvorky Jun 17, 2018 •

edited

Loading

mattilyra Jun 17, 2018 •

edited

Loading

piskvorky Jun 17, 2018

mattilyra Jun 17, 2018

piskvorky Jun 17, 2018

piskvorky commented Jun 18, 2018 •

edited

Loading

menshikh-iv left a comment •

edited

Loading

menshikh-iv Jun 19, 2018

menshikh-iv Jun 19, 2018

mattilyra Jun 19, 2018

menshikh-iv Jun 19, 2018

mattilyra Jun 21, 2018

menshikh-iv Jun 22, 2018 •

edited

Loading

mattilyra commented Jul 9, 2018

piskvorky left a comment

piskvorky Jul 11, 2018 •

edited

Loading

mattilyra commented Jul 11, 2018

piskvorky commented Jul 11, 2018 •

edited

Loading

piskvorky commented Jul 13, 2018

[MRG] Wikicorpus custom filters #2089

[MRG] Wikicorpus custom filters #2089

Conversation

mattilyra commented Jun 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piskvorky commented Jun 17, 2018 • edited Loading

piskvorky Jun 17, 2018 • edited Loading

Choose a reason for hiding this comment

mattilyra Jun 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piskvorky commented Jun 18, 2018 • edited Loading

menshikh-iv left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menshikh-iv Jun 22, 2018 • edited Loading

Choose a reason for hiding this comment

mattilyra commented Jul 9, 2018

piskvorky left a comment

Choose a reason for hiding this comment

piskvorky Jul 11, 2018 • edited Loading

Choose a reason for hiding this comment

mattilyra commented Jul 11, 2018

piskvorky commented Jul 11, 2018 • edited Loading

piskvorky commented Jul 13, 2018

mattilyra commented Jun 13, 2018 •

edited

Loading

piskvorky commented Jun 17, 2018 •

edited

Loading

piskvorky Jun 17, 2018 •

edited

Loading

mattilyra Jun 17, 2018 •

edited

Loading

piskvorky commented Jun 18, 2018 •

edited

Loading

menshikh-iv left a comment •

edited

Loading

menshikh-iv Jun 22, 2018 •

edited

Loading

piskvorky Jul 11, 2018 •

edited

Loading

piskvorky commented Jul 11, 2018 •

edited

Loading