Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pagination, phase 1: Add unit tests for :core module with coverage. #230

Conversation

Yury-Fridlyand
Copy link

Add unit tests for :core module with coverage. Uncovered: toCursor, because it is will be changed soon. Merge target: #226

Description

Completely pass :core:checkstyleTest. Partial pass for :core:test :core:jacocoTestReport :core:jacocoTestCoverageVerification:

* What went wrong:
Execution failed for task ':core:jacocoTestCoverageVerification'.
> Rule violated for class org.opensearch.sql.planner.physical.PhysicalPlan: lines covered ratio is 0.7, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.PhysicalPlan: branches covered ratio is 0.5, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.ProjectOperator: lines covered ratio is 0.9, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.ProjectOperator: branches covered ratio is 0.8, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.PaginateOperator: branches covered ratio is 0.9, but expected minimum is 1.0

Issues Resolved

N/A

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Feb 22, 2023

Codecov Report

Merging #230 (16bdd77) into dev-pagination-v2_revC (2c1b2f9) will increase coverage by 5.16%.
The diff coverage is n/a.

@@                     Coverage Diff                      @@
##             dev-pagination-v2_revC     #230      +/-   ##
============================================================
+ Coverage                     94.66%   99.82%   +5.16%     
+ Complexity                     3270      185    -3085     
============================================================
  Files                           337       29     -308     
  Lines                          8339      582    -7757     
  Branches                        529       50     -479     
============================================================
- Hits                           7894      581    -7313     
+ Misses                          436        1     -435     
+ Partials                          9        0       -9     
Flag Coverage Δ
sql-engine 99.82% <ø> (+5.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ain/java/org/opensearch/sql/analysis/Analyzer.java
.../org/opensearch/sql/data/model/ExprValueUtils.java
...rg/opensearch/sql/executor/CanPaginateVisitor.java
...rg/opensearch/sql/executor/PaginatedPlanCache.java
...main/java/org/opensearch/sql/executor/QueryId.java
...java/org/opensearch/sql/executor/QueryService.java
.../sql/executor/execution/ContinuePaginatedPlan.java
...ensearch/sql/executor/execution/PaginatedPlan.java
.../sql/executor/execution/PaginatedQueryService.java
...earch/sql/executor/execution/QueryPlanFactory.java
... and 297 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Yury-Fridlyand
Copy link
Author

CI run for this PR: https://github.com/Bit-Quill/opensearch-project-sql/actions/runs/4247173319
No unit tests or checkstyle validations failed for :core:test

@@ -44,13 +44,13 @@ public Boolean visitRelation(Relation node, Object context) {
return Boolean.TRUE;
}

/*

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is canPaginate not used anywhere?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used in commented code below only.

public void explain(ResponseListener<ExecutionEngine.ExplainResponse> listener) {
listener.onFailure(new UnsupportedOperationException(
listener.onFailure(new NotImplementedException(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change from UnsupportedOperationException to NotImplementedException here but from NotImeplementedException to UnsuportedException ContinuePaginatedPlane.java?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to comply with nature of these exceptions. Explain for PaginatedPlan will be implemented soon (so throw NotImplementedException), but explain in ContinuePaginatedPlan won't be implemented ever (as I understand) - so UnsupportedOperationException is thrown.

} catch (Exception e) {
listener.onFailure(e);
}
executionEngine.execute(plan, ExecutionContext.emptyExecutionContext(), listener);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was the try-catch block removed? Was something changed so that we don't expect this to throw an exception anymore?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

executePlan called from execute function only which already has try-catch block.

@Yury-Fridlyand Yury-Fridlyand force-pushed the dev-pagination-v2_revC branch from 11a18c1 to a3ef2bf Compare March 3, 2023 22:35
…`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
@Yury-Fridlyand Yury-Fridlyand force-pushed the dev-pagination-v2_revC-ut-coverate-for-core branch from a35f6ce to 16bdd77 Compare March 3, 2023 22:40
@Yury-Fridlyand Yury-Fridlyand changed the title Add unit tests for :core module with coverage. Pagination, phase 1: Add unit tests for :core module with coverage. Mar 7, 2023
@MaxKsyunz
Copy link

@Yury-Fridlyand any reason this can't be merged?

@Yury-Fridlyand
Copy link
Author

No reasons, just waited for more reviews. Going to merge, it is not the last step and last opportunity to review.

@Yury-Fridlyand Yury-Fridlyand merged commit 2d29549 into dev-pagination-v2_revC Mar 8, 2023
@Yury-Fridlyand Yury-Fridlyand deleted the dev-pagination-v2_revC-ut-coverate-for-core branch March 8, 2023 01:10
Yury-Fridlyand added a commit that referenced this pull request Mar 28, 2023
* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
Yury-Fridlyand added a commit that referenced this pull request Mar 28, 2023
* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)


Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
Yury-Fridlyand added a commit that referenced this pull request Mar 28, 2023
* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
Yury-Fridlyand added a commit that referenced this pull request Mar 29, 2023
* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
acarbonetto pushed a commit that referenced this pull request Apr 28, 2023
* Support pagination in V2 engine, phase 1 (#226)

* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>

* Make scroll timeout configurable.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix IT to set cursor keep alive parameter.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove `QueryId.None`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename according to PR feedback.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove default implementations of `PushDownRequestBuilder`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Merge paginated plan optimizer into the regular optimizer. (opensearch-project#1516)

Merge paginated plan optimizer into the regular optimizer.
---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Complete rework on serialization and deserialization. (opensearch-project#1498)

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Resolve merge conflicts and fix tests.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor cleanup.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor cleanup - missing changes for the previous commit.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove paginate operator  (opensearch-project#1528)

* Remove PaginateOperator class since it is no longer used.


---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedPlan` - move logic to `QueryPlan`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove default implementations from `SerializablePlan`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add a doc.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update design graphs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* More fixes for merge from upstream/main.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
acarbonetto pushed a commit that referenced this pull request Apr 28, 2023
* Support pagination in V2 engine, phase 1 (#226)

* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix explain.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update ITs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Some file moves.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>

* Make scroll timeout configurable.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Fix IT to set cursor keep alive parameter.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove `QueryId.None`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Rename according to PR feedback.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove default implementations of `PushDownRequestBuilder`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Merge paginated plan optimizer into the regular optimizer. (opensearch-project#1516)

Merge paginated plan optimizer into the regular optimizer.
---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Complete rework on serialization and deserialization. (opensearch-project#1498)

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Resolve merge conflicts and fix tests.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor cleanup.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Minor cleanup - missing changes for the previous commit.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove paginate operator  (opensearch-project#1528)

* Remove PaginateOperator class since it is no longer used.

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

* Remove `PaginatedPlan` - move logic to `QueryPlan`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Remove default implementations from `SerializablePlan`.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Add a doc.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Update design graphs.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* More fixes for merge from upstream/main.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

---------

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants