Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create local random generator for sample_text & add lenght #1422

Merged
merged 4 commits into from
Jun 22, 2017

Conversation

vlejd
Copy link
Contributor

@vlejd vlejd commented Jun 16, 2017

Changes regarding @gojomo comments at #308 . Add option to use local random and option to tell the function how long is the corpus (without running len and going through the whoe corpus).

class TestTextCorpus(TextCorpus):
def __init__(self):
self.data = [["document1"], ["document2"]]
class DumyTextCorpus(TextCorpus):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spelling: 'Dummy'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

if n != 0:
# This means that length was set to be smaller than nuber of items in stream.
raise ValueError("length smaller than number of documents in stream")

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this actually the opposite: that the generator has more elements than was declared?

Even so, I doubt this case should be a thrown-exception that must be handled. The user got their n docs, within the length declared – which might be exactly what they wanted/needed. So perhaps just a logged warning – "documents in excess of declared 'length' not considered for sampling".

Copy link
Contributor Author

@vlejd vlejd Jun 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be: length was set to be greater than number of items in stream indeed.

It throws an exception because user received less than n docs. Note that if length is greater than actual length of the corpus, it will not trigger the exception on its own. The exception triggers when we still need to sample some documents, but the stream run out. I add it to the comment.

The case when length was smaller than the actual corpus length does not throw anything right now (the problem was only in the documentation). Printing a warning is awkward, because we would have to actually go through all of the length elements which is unnecessary (we may sample enough elements earlier).

Copy link
Owner

@piskvorky piskvorky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vlejd !

The language needs a little cleanup (lots of missing articles).

"""
Yield n random texts from the corpus without replacement.
Yields n random documents from the corpus without replacement.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Yields" => "Yield" (imperative mode).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


Given the the number of remaingin elements in stream is remaining and we need
to choose n elements, the probability for current element to be chosen is n/remaining.
Given the number of remaining documents in corpus, we need to choose n elements.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"a corpus".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Given the the number of remaingin elements in stream is remaining and we need
to choose n elements, the probability for current element to be chosen is n/remaining.
Given the number of remaining documents in corpus, we need to choose n elements.
The probability for current element to be chosen is n/remaining.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the current element"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

If we choose it, we just decreese the n and move to the next element.
Computing corpus length may be a costly operation so you can use optional paramter
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the corpus length"
"the optional parameter `length` instead"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Args:
n (int): number of documents we want to sample.
seed (int|None): if specified, use it as a seed for local random generator.
length (int|None): if specified, use it as guess of corpus length.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"a guess"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

length (int|None): if specified, use it as guess of corpus length.

Yeilds:
list[str]: document represented as list of tokens. See get_texts method.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"a list"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

list[str]: document represented as list of tokens. See get_texts method.

Raises:
ValueError: then n is invalid or length was set incorrectly.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"then" => "when"?

Also, what does "incorrectly" mean? Above we say length is "a guess", so how can it be incorrect?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add some clarification to length parameter description.

length (int|None): if specified, use it as a guess of corpus length.
It must be positive and not greater than actual corpus length.

Yeilds:
Copy link
Contributor

@rasto2211 rasto2211 Jun 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*Yields. Spell checker can be useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.
I was wandering that I should finally install it (F6 in sublime).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Wondering". Definitely :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no words.

@menshikh-iv
Copy link
Contributor

Thank you @vlejd

@menshikh-iv menshikh-iv merged commit 0d47a6f into piskvorky:develop Jun 22, 2017
if n != 0:
# This means that length was set to be greater than number of items in corpus
# and we were not able to sample enough documents before the stream ended.
raise ValueError("length greater than number of documents in corpus")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make the message more concrete? Include the actual lengths.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


if not n <= length:
raise ValueError("n is larger than length of corpus.")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make the message more specific? Include the actual lengths.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@vlejd vlejd mentioned this pull request Jun 22, 2017
saparina pushed a commit to saparina/gensim that referenced this pull request Jul 9, 2017
* Create local random generator for sample_text & add lenght

* Fix typos
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants