Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[db-api] Performance improvements #144

Merged

Conversation

john-bodley
Copy link
Contributor

This PR improves the performance of fetchall and fetchmany by iterating on the result set rather than iterating over the cursor's iterator. The following,

import timeit

from pydruid.db.api import Cursor


def execute(self, operation, parameters=None):
    self._results = iter(range(10000))


Cursor.execute = execute


def test():
    cursor = Cursor(url=None)
    cursor.execute(operation=None)
    cursor.fetchall()


print(timeit.timeit("test()", number=1000, setup="from __main__ import test"))

resulted in an execution time of 6.698s and 0.331s for the current and proposed solutions respectively, i.e., the proposed solution is about 20x faster.

to: @betodealmeida @mistercrunch

@mistercrunch
Copy link
Member

tenor

@mistercrunch mistercrunch merged commit fa273f4 into druid-io:master Nov 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants