Support pagination in V2 engine, phase 1 (#226)

[Yury-Fridlyand]

Description

https://github.com/Bit-Quill/opensearch-project-sql/blob/61767f2b200b2a7f8c8b2df32de209b3c30caa61/docs/dev/Pagination-v2.md

PHASE 1
Pagination supports queries like

select * from <index>;

For example:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"query": "SELECT * from calcs", "fetch_size": 5}'

Cursor request:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"cursor": "...." }'

All credits to @MaxKsyunz.

You can use attached script for testing as well. Command line: ./cursor_test.sh <table> <page size>. Requires jq. Rename it before use.
cursor_test.sh.txt

Scroll API usage doc:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html

Please, see team review and discussion in Bit-Quill#226

Pagination demo: https://user-images.githubusercontent.com/88679692/224208630-8d38d833-abf8-4035-8d15-d5fb4382deca.mp4

Further improvements are coming

1. Support for nested expression serialization in #249
2. Rework on serialization and deserialization in #252
3. Support queries with LIMIT, WHERE, HAVING and ORDER BY and queries without FROM into pagination. in #253 - Phase 2
4. Rework ComparisonIT to use pagination. in #254 - Phase 2

Issues Resolved

#656

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Yury-Fridlyand]

I copied unresolved questions/issues from #226

[Yury-Fridlyand]

Bit-Quill#226 (comment) by @MaxKsyunz:

Could you implement this similar to PUSH_DOWN_* rules:

• Add a pushDownPageSize method to TableScanBuilder
• Add a PUSH_DOWN_PAGE_SIZE similar to others that matches LogicalPaginate of scanBuilder and calls pushDownPageSize.

Doing so would remove the need for PushPageSize and LogicalRelation.pageSize property, be less code, and follow a pattern already used in the code base, like PUSH_DOWN_LIMIT.

[Yury-Fridlyand]

I tried to do this and faced #1470. I keep the draft, and I'll try to do thorough investigation and fix.

[Yury-Fridlyand]

Bit-Quill#226 (comment) by @penghuo:

why get TableScanOperator in StorageEngine?

@Yury-Fridlyand:

It is used in cursor deserialization/parsing, this happens in :core module. All implementation of StorageEngine are defined outside of :core and unavailable and that point.

@MaxKsyunz:

@penghuo does this design make sense? Would love to know what you think.

[GumpacG]

Setting fetch_size to be greater than the total size of the query results gives a cursor when legacy does not. Then when making a request with that cursor, it returns an empty result. Should it omit the cursor when on the last page?

For example, the query:
{ "fetch_size": 30, "query": "SELECT * FROM calcs;" }

The result would be:

Result

{
    "schema": [
        {
            "name": "bool3",
            "type": "boolean"
        },
        {
            "name": "int0",
            "type": "integer"
        },
        {
            "name": "time1",
            "type": "timestamp"
        },
        {
            "name": "bool2",
            "type": "boolean"
        },
        {
            "name": "int2",
            "type": "integer"
        },
        {
            "name": "int1",
            "type": "integer"
        },
        {
            "name": "str3",
            "type": "keyword"
        },
        {
            "name": "int3",
            "type": "integer"
        },
        {
            "name": "str1",
            "type": "keyword"
        },
        {
            "name": "str2",
            "type": "keyword"
        },
        {
            "name": "time0",
            "type": "timestamp"
        },
        {
            "name": "datetime0",
            "type": "timestamp"
        },
        {
            "name": "num1",
            "type": "double"
        },
        {
            "name": "num0",
            "type": "double"
        },
        {
            "name": "datetime1",
            "type": "timestamp"
        },
        {
            "name": "num4",
            "type": "double"
        },
        {
            "name": "bool1",
            "type": "boolean"
        },
        {
            "name": "key",
            "type": "keyword"
        },
        {
            "name": "num3",
            "type": "double"
        },
        {
            "name": "bool0",
            "type": "boolean"
        },
        {
            "name": "num2",
            "type": "double"
        },
        {
            "name": "str0",
            "type": "keyword"
        },
        {
            "name": "date3",
            "type": "timestamp"
        },
        {
            "name": "date2",
            "type": "timestamp"
        },
        {
            "name": "date1",
            "type": "timestamp"
        },
        {
            "name": "date0",
            "type": "timestamp"
        },
        {
            "name": "zzz",
            "type": "keyword"
        }
    ],
    "datarows": [
        [
            true,
            1,
            "1970-01-01 19:36:22",
            false,
            5,
            -3,
            "e",
            8,
            "CLAMP ON LAMPS",
            "one",
            "1899-12-30 21:07:32",
            "2004-07-09 10:17:35",
            8.42,
            12.3,
            null,
            null,
            true,
            "key00",
            -11.52,
            true,
            17.86,
            "FURNITURE",
            "1986-03-20 00:00:00",
            "1977-04-20 00:00:00",
            "2004-04-01 00:00:00",
            "2004-04-15 00:00:00",
            "a"
        ],
        [
            null,
            null,
            "1970-01-01 02:05:25",
            false,
            -4,
            -6,
            "e",
            13,
            "CLOCKS",
            "two",
            "1900-01-01 13:48:48",
            "2004-07-26 12:30:34",
            6.71,
            -12.3,
            null,
            10.85,
            true,
            "key01",
            -9.31,
            false,
            16.73,
            "FURNITURE",
            null,
            "1995-09-03 00:00:00",
            "2004-04-02 00:00:00",
            "1972-07-04 00:00:00",
            "b"
        ],
        [
            null,
            null,
            "1970-01-01 09:33:31",
            false,
            5,
            null,
            "e",
            2,
            "AIR PURIFIERS",
            "three",
            "1900-01-01 18:21:08",
            "2004-08-02 07:59:23",
            9.78,
            15.7,
            null,
            -13.47,
            true,
            "key02",
            -12.17,
            null,
            null,
            "OFFICE SUPPLIES",
            "1997-02-02 00:00:00",
            "1997-09-19 00:00:00",
            "2004-04-03 00:00:00",
            "1975-11-12 00:00:00",
            "c"
        ],
        [
            null,
            null,
            "1970-01-01 22:50:16",
            false,
            -5,
            -4,
            "e",
            5,
            "BINDER ACCESSORIES",
            null,
            "1900-01-01 18:51:48",
            "2004-07-05 13:14:20",
            7.43,
            -15.7,
            null,
            -6.05,
            false,
            "key03",
            -7.25,
            true,
            8.51,
            "OFFICE SUPPLIES",
            null,
            "1980-07-26 00:00:00",
            "2004-04-04 00:00:00",
            "2004-06-04 00:00:00",
            "d"
        ],
        [
            true,
            7,
            null,
            true,
            3,
            null,
            null,
            9,
            "BINDER CLIPS",
            "five",
            "1900-01-01 15:01:19",
            "2004-07-28 23:30:22",
            9.05,
            3.5,
            null,
            8.32,
            false,
            "key04",
            12.93,
            false,
            6.46,
            "OFFICE SUPPLIES",
            "1996-03-07 00:00:00",
            "1997-05-30 00:00:00",
            "2004-04-05 00:00:00",
            "2004-06-19 00:00:00",
            "e"
        ],
        [
            false,
            3,
            "1970-01-01 19:57:33",
            true,
            2,
            null,
            null,
            7,
            "BINDING MACHINES",
            "six",
            "1900-01-01 08:59:39",
            "2004-07-22 00:30:23",
            9.38,
            -3.5,
            null,
            10.71,
            false,
            "key05",
            -19.96,
            null,
            8.98,
            "OFFICE SUPPLIES",
            "1979-04-01 00:00:00",
            "1980-11-07 00:00:00",
            "2004-04-06 00:00:00",
            null,
            "f"
        ],
        [
            null,
            8,
            null,
            false,
            9,
            null,
            "e",
            18,
            "BINDING SUPPLIES",
            null,
            "1900-01-01 07:37:48",
            "2004-07-28 06:54:50",
            16.42,
            0.0,
            null,
            null,
            null,
            "key06",
            10.93,
            true,
            11.69,
            "OFFICE SUPPLIES",
            null,
            "1977-02-08 00:00:00",
            "2004-04-07 00:00:00",
            null,
            "g"
        ],
        [
            false,
            null,
            "1970-01-01 19:48:23",
            true,
            0,
            2,
            "e",
            3,
            "BUSINESS ENVELOPES",
            "eight",
            "1900-01-01 19:45:54",
            "2004-07-12 17:30:16",
            11.38,
            null,
            null,
            -10.24,
            null,
            "key07",
            3.64,
            false,
            17.25,
            "OFFICE SUPPLIES",
            null,
            "1974-05-03 00:00:00",
            "2004-04-08 00:00:00",
            null,
            "h"
        ],
        [
            false,
            null,
            "1970-01-01 22:20:14",
            false,
            -6,
            3,
            null,
            17,
            "ANSWERING MACHINES",
            "nine",
            "1900-01-01 09:00:59",
            "2004-07-04 22:49:28",
            9.47,
            10.0,
            null,
            4.77,
            null,
            "key08",
            -13.38,
            null,
            null,
            "TECHNOLOGY",
            "1983-05-22 00:00:00",
            "1976-09-09 00:00:00",
            "2004-04-09 00:00:00",
            null,
            "i"
        ],
        [
            null,
            8,
            null,
            false,
            -9,
            3,
            "e",
            2,
            "BUSINESS COPIERS",
            "ten",
            "1900-01-01 20:36:00",
            "2004-07-23 21:13:37",
            12.4,
            null,
            null,
            null,
            true,
            "key09",
            -10.56,
            null,
            11.5,
            "TECHNOLOGY",
            null,
            "1998-08-12 00:00:00",
            "2004-04-10 00:00:00",
            null,
            "j"
        ],
        [
            null,
            4,
            "1970-01-01 00:05:57",
            false,
            -3,
            null,
            "e",
            11,
            "CD-R MEDIA",
            "eleven",
            "1900-01-01 01:31:32",
            "2004-07-14 08:16:44",
            10.32,
            null,
            null,
            19.39,
            true,
            "key10",
            -4.79,
            true,
            6.8,
            "TECHNOLOGY",
            "1999-08-20 00:00:00",
            "1974-03-17 00:00:00",
            "2004-04-11 00:00:00",
            null,
            "k"
        ],
        [
            null,
            10,
            "1970-01-01 04:40:49",
            true,
            -4,
            -8,
            null,
            2,
            "CONFERENCE PHONES",
            "twelve",
            "1899-12-30 22:15:40",
            "2004-07-25 15:22:26",
            2.47,
            null,
            null,
            3.82,
            true,
            "key11",
            -10.81,
            false,
            3.79,
            "TECHNOLOGY",
            null,
            "1994-04-20 00:00:00",
            "2004-04-12 00:00:00",
            null,
            "l"
        ],
        [
            true,
            null,
            "1970-01-01 04:48:07",
            true,
            0,
            null,
            null,
            11,
            "CORDED KEYBOARDS",
            null,
            "1900-01-01 13:53:46",
            "2004-07-17 14:01:56",
            12.05,
            null,
            null,
            3.38,
            false,
            "key12",
            -6.62,
            null,
            null,
            "TECHNOLOGY",
            null,
            "2001-02-04 00:00:00",
            "2004-04-13 00:00:00",
            null,
            "m"
        ],
        [
            true,
            4,
            null,
            true,
            4,
            null,
            null,
            18,
            "CORDLESS KEYBOARDS",
            "fourteen",
            "1900-01-01 04:57:51",
            "2004-07-19 22:21:31",
            10.37,
            null,
            null,
            null,
            false,
            "key13",
            -18.43,
            null,
            13.04,
            "TECHNOLOGY",
            "1996-05-13 00:00:00",
            "1988-01-05 00:00:00",
            "2004-04-14 00:00:00",
            null,
            "n"
        ],
        [
            null,
            11,
            "1970-01-01 18:58:41",
            true,
            -8,
            null,
            "e",
            18,
            "DOT MATRIX PRINTERS",
            "fifteen",
            "1899-12-30 22:42:43",
            "2004-07-31 11:57:52",
            7.1,
            null,
            null,
            -14.21,
            false,
            "key14",
            6.84,
            true,
            null,
            "TECHNOLOGY",
            "1986-11-08 00:00:00",
            "1972-07-12 00:00:00",
            "2004-04-15 00:00:00",
            null,
            "o"
        ],
        [
            true,
            4,
            null,
            false,
            -9,
            null,
            "e",
            11,
            "DVD",
            "sixteen",
            "1899-12-30 22:24:08",
            "2004-07-14 07:43:00",
            16.81,
            null,
            null,
            6.75,
            null,
            "key15",
            -10.98,
            false,
            10.98,
            "TECHNOLOGY",
            null,
            "1995-06-04 00:00:00",
            "2004-04-16 00:00:00",
            null,
            "p"
        ],
        [
            null,
            8,
            "1970-01-01 12:33:57",
            false,
            6,
            -9,
            null,
            0,
            "ERICSSON",
            null,
            "1900-01-01 11:58:29",
            "2004-07-28 12:34:28",
            7.12,
            null,
            null,
            null,
            null,
            "key16",
            -2.6,
            null,
            7.87,
            "TECHNOLOGY",
            "1992-01-18 00:00:00",
            "2002-04-27 00:00:00",
            "2004-04-17 00:00:00",
            null,
            "q"
        ]
    ],
    "total": 17,
    "size": 17,
    "status": 200,
    "cursor": "n:1f8b08000000000000ffed9b5b73aa3a1886ff4e3b75667328abdb8b751194832c454148803b08545a02a5222afcfa1da07bb774cddaf73271c6190f21dfe9f1f3358977bbf0f05284a764c6cf446e76b73bbebd26f834bb2bc23c8977e1b14a946b794caaeae5ada866f7b3bb7218b17ea94eb3e39603b2673660c19f718eeb4894495498c4f78cd77021b5185deac04b2f38872d16c8397a91b63ee249a0c047ac1b048b6619094f8f69fa0bfdf52ef896b50280bb0019a95588d4360660ef68651ae7ca39d2d43a68e64eac1b6594e327070023d06015a0388d3598d1b1ef8e366fe89833d6608d0598e2dc7c5b8be6355acc49a2cb4de0996d88e6f59afb6affb19b4b8972f514780a783e00998ebd60710536a89b4f1acf5774f349e3f98841821c3601925e03771c1bd872c72b3a6f32f0119ba67234862bd01f0170e9734fa6fea75dacd6bfb1c69e5d529f37a16772dbee1add96a80f1c58c49b48348ab530bf505b6d80d4c6170e672caad55ab0e93cf45a9d5c82fd5cedfc83dd63e744fdb1684e8d73245cda2e2e2defedd4831d49c6859126de26b33ca3f13dd2db1542ebef7325ace4d36a01801ad17b4a7d7410777df7bd205d0bd794d6e1ec08c67b80a89f627a89bd1550f8d326cc55c1df4b958fa4624d4c0ee7a40edafac4513a7ed8fc937da1b9d0d30be86e43ce57cfc50afcfa73be33dfb3d375d1e5c1aded8ea9cca4cc4137f164023e6f7b00e4477c00db910f19ac63c47d8e9207fb341eddcae63b4781b2530c7e30a86f096a8dd4f1526e69c1c38f82f3b146aa75a636385725dcd803dceeb5c4a21394cbe6a1f50f1a77a07ec30a00a4d2629bd1b77aecfcbc24be683d517b3c9dfb13fabc87be7645bb092958db1fe21ed6cfa5ee9d74700020d107bcb0253f00903ed06bcb3eb67c145bddc796f7b1d5436c521f9b25d0dac01ef2efb0360048231f6c24f191f5394ce9edcb94f2d87088adda1cdc0f7e1c7e32a86f09ea6e4ee45e59a71e756a032ac4b4a1e95a2e6f31b06f11ec4182340c6c2641a603752f410c26419804991ad42a839a413d21a8752a035e4d06f508eac081aa613859dcd763017e32a86f0aeabe5333a859a79e12d47da766f28375eac941cd7e2832a8a704f5b0567d614b7a80ad554f09ec05eade875c48f3bfd169cde47200b4e85ebbd41f50b78632746f47236d6ce6d96efd64be80684f3bac8ca82bbe677d81773e80c3cd5fc21cbec6cbce166cbe74f57aa89bf45fdd8eb181a4872c4e38bce821ec4953876ea94b9425f2679668ee28ac5c073385b5cbf392e683f471fed68d5760998f7c50a3029e7efb1074b981b6e1b8d0818a6aba3a5bb3be39b8658d7e1d3b4c5f7f93228ac3c385a34026456e176a761684413d21a84732c4633284c99009c13d746c9b756cd6b12704f570c8899dde1baf88b0434eb70c35104e2461f263243f6462ba6e46b6b6c880be39a007e9c1b6cd99f49812d483f4609b314c7a4c08eabe53b36d73d6a9a704757f1684ad557f839a9d05b969a8fbb56a97fdc37c2c3fd859908980cdfeb7c8c09e24d86cad9a813d49b0d94a08037b5a6003b1fc91146c2306fccf46ccfdec6e5b26c53e098f38dd8587245e157172dde3b098e190e06aa66aa4f6852b4fc97bf685794dc97b8cd5391f23922d3d950f3c43b29094d939e46865643583fb2d32f5884b3d97a73fb35b5b7739691fb424dcf0caa7e7761024797cb5a16d87aafd1e40c347851162eded1a66f3e2fefefe1f2c488778714d0000"
}

And when that cursor is requested it results in:

Result

{
    "schema": [
        {
            "name": "bool3",
            "type": "boolean"
        },
        {
            "name": "int0",
            "type": "integer"
        },
        {
            "name": "time1",
            "type": "timestamp"
        },
        {
            "name": "bool2",
            "type": "boolean"
        },
        {
            "name": "int2",
            "type": "integer"
        },
        {
            "name": "int1",
            "type": "integer"
        },
        {
            "name": "str3",
            "type": "keyword"
        },
        {
            "name": "int3",
            "type": "integer"
        },
        {
            "name": "str1",
            "type": "keyword"
        },
        {
            "name": "str2",
            "type": "keyword"
        },
        {
            "name": "time0",
            "type": "timestamp"
        },
        {
            "name": "datetime0",
            "type": "timestamp"
        },
        {
            "name": "num1",
            "type": "double"
        },
        {
            "name": "num0",
            "type": "double"
        },
        {
            "name": "datetime1",
            "type": "timestamp"
        },
        {
            "name": "num4",
            "type": "double"
        },
        {
            "name": "bool1",
            "type": "boolean"
        },
        {
            "name": "key",
            "type": "keyword"
        },
        {
            "name": "num3",
            "type": "double"
        },
        {
            "name": "bool0",
            "type": "boolean"
        },
        {
            "name": "num2",
            "type": "double"
        },
        {
            "name": "str0",
            "type": "keyword"
        },
        {
            "name": "date3",
            "type": "timestamp"
        },
        {
            "name": "date2",
            "type": "timestamp"
        },
        {
            "name": "date1",
            "type": "timestamp"
        },
        {
            "name": "date0",
            "type": "timestamp"
        },
        {
            "name": "zzz",
            "type": "keyword"
        }
    ],
    "datarows": [],
    "total": 0,
    "size": 0,
    "status": 200
}

[acarbonetto]

Setting fetch_size to be greater than the total size of the query results gives a cursor when legacy does not. Then when making a request with that cursor, it returns an empty result. Should it omit the cursor when on the last page?

We need to return the cursor so long as it is still 'active' - ie, the scroll API scroll id is still stored in memory and available.

Alternatively, we could call the scroll API to close the scroll id once all the results are hit, but that's out of scope.

[Yury-Fridlyand]

Superseded by #1497.