Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dumps methods #157

Merged
merged 9 commits into from
Oct 14, 2020
Merged

Add dumps methods #157

merged 9 commits into from
Oct 14, 2020

Conversation

eskombro
Copy link
Member

@eskombro eskombro commented Oct 13, 2020

As stated in meilisearch/integration-guides#44, SDKs should implement the dump feature, by adding 2 new routes, testing and adding code samples

@eskombro eskombro marked this pull request as ready for review October 13, 2020 15:59
Copy link
Member

@curquiza curquiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome

Comment on lines 40 to 43
while dump_status['status'] == 'processing':
time.sleep(0.1)
dump_status = self.client.get_dump_status(dump['uid'])
assert dump_status['status'] == 'done'
Copy link
Member

@curquiza curquiza Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my POV, you are testing MeiliSearch here. The SDK should not test the dump is created, in particular in test_dump_status_route that only returns the status of a dump.
Cf the end of the conversation 🐌

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed!

But at the same time it checks that with a dump status change on MS, client recieves the new information. Doesn't hurt to have the check for free :)

If you think this is absolutely out of scope, I'm not against removing it tho!

Copy link
Member

@curquiza curquiza Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is not "for free". It adds a small complexity in test readability. A small one, but if you add more and more like this one, it's not small at all 🙂
Now MeiliSearch is greatly doing its own tests, we should try now to stay in our scope to keep our SDKs as clean as possible. A lot of our old tests should not be in our SDKs anymore because they are also testing MeiliSearch, but before we find time to clean them, I suggest we don't add any other out-of-scope tests.
Also, @bidoubiwa, what do you think? Maybe both of you think this is in the scope.

Cf the end of the conversation 🐌

Copy link
Member Author

@eskombro eskombro Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand your POV and mostly agree with it. But also, I would still prefer to wait until the dump is created (while loop) because if we add a new test below (as I just did in a new commit) we can introduce a conflict (because you are not allowed to create another dump if one is being created). So it would be better to wait until the dump is done to quit the test. In this case, I would remove just the last line of this block, which is why I said it is 'free' to check the 'done' status, WDYT?

Copy link
Member Author

@eskombro eskombro Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did abstract this waiting loop on a wait_for_dump_creation method just used in tests, to facilitate reusability and readability.

It now looks like:

    ...
    assert dump_status['status'] == 'processing'
    wait_for_dump_creation(dump_status['uid'])
    assert dump_status['status'] == 'done'

If you want me to remove the last line, react with 👎 and I'll do ;)

Copy link
Member Author

@eskombro eskombro Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is needed because if we exit the test right away, and a new test tries to create a new dump, MeiliSearch will throw an error, since you can't create two dumps at the same time

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other hand, I opted for removing the last check for the 'done' status as you suggested

Copy link
Member

@curquiza curquiza Oct 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is needed because if we exit the test right away, and a new test tries to create a new dump, MeiliSearch will throw an error, since you can't create two dumps at the same time

I was slow to understand! sorryyyyy! 🐌 I've just understood that's only for the next tests! 😅

Copy link
Member

@curquiza curquiza Oct 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if it's for the next tests, why the test test_dump_creation does not have any wait_for_pending_dump and does not seem to need it?

Copy link
Member Author

@eskombro eskombro Oct 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because at the time the create_dump test is executed, there are no documents, MeiliSearch index is completely empty, and therefore the dump creation is quite fast, almost instant. But you are right, in theory, it should also wait for its creation. On the other hand, before the get_dump_status test, MeiliSearch creates an index and populate it before creating a dump, and so it takes some time.

meilisearch/tests/client/test_client_dumps.py Outdated Show resolved Hide resolved
@eskombro
Copy link
Member Author

eskombro commented Oct 13, 2020

Removed the redundant asserts ;)

I also added a test for it to fail with a MeiliSearchApiError in case the uid doesn't exist @curquiza

Copy link
Member

@curquiza curquiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a teardown_class to delete all the indexes, or at least the index your create?
The setup_class is here to make the test context clean
The teardown_class is to clean after the test is done to let a clean MeiliSearch after running the tests. It's for maintainers convenience only: if you run only this test or if it's the last test to be processed.

Copy link
Contributor

@bidoubiwa bidoubiwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot for the PR! looks really good to me :)

meilisearch/tests/client/test_client_dumps.py Show resolved Hide resolved
bidoubiwa
bidoubiwa previously approved these changes Oct 14, 2020
Copy link
Member

@curquiza curquiza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LPTM (Looks PERFECT to me)

@eskombro eskombro requested a review from bidoubiwa October 14, 2020 12:22
@eskombro eskombro merged commit 8a7de8d into master Oct 14, 2020
@eskombro eskombro deleted the implement_dumps branch October 14, 2020 13:08
@curquiza curquiza changed the title Add dumps implementation, tests and code samples Add dumps methods Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants