Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more granular lazy execution flags in query compiler #7340

Closed
sfc-gh-joshi opened this issue Jul 11, 2024 · 0 comments · Fixed by #7348
Closed

Add more granular lazy execution flags in query compiler #7340

sfc-gh-joshi opened this issue Jul 11, 2024 · 0 comments · Fixed by #7348
Labels
Interfaces and abstractions Issues with Modin's QueryCompiler, Algebra, or BaseIO objects new feature/request 💬 Requests and pull requests for new features P3 Very minor bugs, or features we can hopefully add some day.

Comments

@sfc-gh-joshi
Copy link

Modin allows backends to opt-in to certain frontend error validation checks based on the query compiler's lazy_execution flag. Some methods (like astype) use this to avoid materializing column labels, while others (like drop) avoid materializing row labels or counts.

Certain backends may defer execution for row/column axes differently: for example, Snowpark pandas eagerly evaluates columns but leaves row labels/counts lazy. We should split up the lazy_execution flag to allow backends to make this distinction in a more granular manner (proposed by @devin-petersohn):

  • lazy_row_labels for backends that defer computations of index
  • lazy_row_count to defer computation of len(index)
  • lazy_column_types to defer computation of dtypes
  • lazy_column_labels to defer computation of columns
  • lazy_column_count to defer computation of len(columns)
@sfc-gh-joshi sfc-gh-joshi added new feature/request 💬 Requests and pull requests for new features Triage 🩹 Issues that need triage labels Jul 11, 2024
@noloerino noloerino added P3 Very minor bugs, or features we can hopefully add some day. Interfaces and abstractions Issues with Modin's QueryCompiler, Algebra, or BaseIO objects and removed Triage 🩹 Issues that need triage labels Jul 11, 2024
noloerino added a commit to noloerino/modin that referenced this issue Jul 19, 2024
Signed-off-by: Jonathan Shi <jhshi07@gmail.com>
noloerino added a commit to noloerino/modin that referenced this issue Jul 22, 2024
Signed-off-by: Jonathan Shi <jhshi07@gmail.com>
devin-petersohn pushed a commit that referenced this issue Jul 29, 2024
Signed-off-by: Jonathan Shi <jhshi07@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Interfaces and abstractions Issues with Modin's QueryCompiler, Algebra, or BaseIO objects new feature/request 💬 Requests and pull requests for new features P3 Very minor bugs, or features we can hopefully add some day.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants