Bump calcite SQL parser identifier max length from default 128 (SqlParser.DEFAULT_IDENTIFIER_MAX_LENGTH) to 1024. #14363
Conversation
…rser.DEFAULT_IDENTIFIER_MAX_LENGTH) to 1024.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #14363 +/- ##
============================================
+ Coverage 61.75% 63.79% +2.03%
- Complexity 207 1556 +1349
============================================
Files 2436 2660 +224
Lines 133233 145902 +12669
Branches 20636 22333 +1697
============================================
+ Hits 82274 93073 +10799
- Misses 44911 45960 +1049
- Partials 6048 6869 +821
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
Can you elaborate more on why text search might use long identifier? Identifier is column name, instead of literal. What is the overhead of increasing it? |
I have updated the PR description which explains the problem better. I don't think there is any overhead of increasing it as it is an artificial limit within the sql parser. For cases where the identifier does not exceed 128 characters, the overhead should be the same. For identifiers that exceeds 128 characters, we haven't noticed any observable overhead since query parsing constitute minimal run-time overhead compared to the query itself. It is extremely rare to encounter identifiers over 128 characters, but if we do encounter it, with this PR, queries will still be able to complete. |
For Pinot queries that involve extensive flattened JSON log matching, the default SQL parser identifier limit of 128 characters is insufficient. For example, a JSON log like
{"aaa": {"bbb": {"ccc": 1}}gets flattened automatically and converted into a structured table with a single column with the column nameaaa.bbb.ccc. In this case, the column name is 11 characters long, but in real-world applications, it could easily exceed 128 characters.This PR raises the identifier length limit from the default 128 (as defined by SqlParser.DEFAULT_IDENTIFIER_MAX_LENGTH) to 1024. Currently, this value cannot be adjusted via configuration, but if necessary in the future, the public static variable that holds this limit can be made configurable with minimal effort.