feat: basic implement of `matches` fn #4222

waynexia · 2024-06-27T07:51:49Z

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

The matches function for full-text search.

Notice this is only the first naive implementation. Some features are missing so I haven't written many docs in this PR or they will be outdated very soon. Please use the test case evaluate_matches_without_parenthesis as doc

Missing features:

handle parenthesis
validate query
operator's priority
must/negative/optional's conversion.

(I remember how things went with the last language project that was rushed in a week 😶‍🌫️ )

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR requires documentation updates.

Summary by CodeRabbit

New Features
- Introduced a full-text search functionality to enhance data querying capabilities.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

coderabbitai · 2024-06-27T07:51:56Z

Note

Reviews paused

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Walkthrough

The recent updates introduce a MatchesFunction for full-text search functionality in the matches.rs file. This includes parsing search patterns, constructing an abstract syntax tree (AST) for pattern matching, and evaluating these patterns against input data using DataFusion. Additionally, minor edits were made to the typos.toml configuration file.

Changes

File	Summary of Changes
`src/common/function/src/scalars/matches.rs`	Introduced `MatchesFunction` for full-text search, including pattern parsing, AST construction, and evaluation. Added relevant structs, enums, and methods alongside tests.
`typos.toml`	Added new configuration entry `unqualifed = "unqualifed"`

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant MatchesFunction
    participant DataFusion
    participant InputData

    User->>MatchesFunction: Provide search pattern
    MatchesFunction->>MatchesFunction: Parse search pattern
    MatchesFunction->>MatchesFunction: Build AST
    MatchesFunction->>DataFusion: Query processing with AST
    DataFusion->>InputData: Evaluate patterns
    InputData-->>DataFusion: Return results
    DataFusion-->>MatchesFunction: Return evaluation results
    MatchesFunction-->>User: Provide matching results

Poem

Beneath the code, in lines we'll find,
A function born of thoughtful mind.
It parses text, both swift and keen,
To find the match where none had been.
With DataFusion's helping aid,
A search is done, the quest well paid.
For rabbits know, in fields of green,
The best of code, is sharply seen. 🎩🐇✨

Tip

Early access features: enabled

We are currently testing the following features in early access:

OpenAI gpt-4o model for code reviews and chat: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available.

Note:

You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.
Please join our Discord Community to provide feedback and report issues.
OSS projects are currently opted into early access features by default.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 5

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8702066 and f31da99.

Files selected for processing (2)

src/common/function/src/scalars.rs (1 hunks)
src/common/function/src/scalars/matches.rs (1 hunks)

Additional context used

GitHub Check: Check typos and docs

src/common/function/src/scalars/matches.rs

[warning] 79-79:
"shoudl" should be "should".

[warning] 128-128:
"unqualifed" should be "unqualified".

Additional comments not posted (4)

src/common/function/src/scalars.rs (1)

18-18: Module addition approved.

The addition of the matches module to the scalars.rs file is correctly implemented and aligns with the PR's objectives to introduce full-text search functionality.

src/common/function/src/scalars/matches.rs (3)

36-43: Documentation suggestion for MatchesFunction.

While the derive and fmt implementations are straightforward, consider adding a brief doc comment above the MatchesFunction struct to explain its role, especially since it is central to the new functionality.

+ /// Represents the function for performing full-text search operations.
  #[derive(Clone, Debug, Default)]
  struct MatchesFunction;

222-514: Complex logic in ParserContext needs careful review.

The ParserContext struct contains complex parsing logic that converts infix expressions to RPN and builds the AST. This logic is intricate and could benefit from more detailed inline comments explaining each step, especially in the to_rpn and parse_one_impl methods.

+    /// Converts an infix expression to Reverse Polish Notation (RPN).
     fn to_rpn(mut raw_tokens: Vec<Token>) -> Result<Vec<Token>> {
         let mut operator_stack = vec![];
         let mut result = vec![];
         raw_tokens.reverse();

         while let Some(token) = raw_tokens.pop() {
             match token {
                 Token::Phase(_) => result.push(token),
                 Token::Must | Token::Negative => {
                     // unary operator with paren is not handled yet
                     let phase = raw_tokens.pop().context(InvalidFuncArgsSnafu {
                         err_msg: "Unexpected end of pattern, expected a phase after unary operator",
                     })?;
                     result.push(phase);
                     result.push(token);
                 }
                 Token::OpenParen => operator_stack.push(token),
                 Token::And | Token::Or => {
                     while let Some(op) = operator_stack.pop() {
                         if op == Token::OpenParen {
                             operator_stack.push(op);
                             break;
                         }
                         result.push(op);
                     }
                     operator_stack.push(token);
                 }
                 Token::CloseParen => {
                     while let Some(op) = operator_stack.pop() {
                         if op == Token::OpenParen {
                             break;
                         }
                         result.push(op);
                     }
                 }
             }
         }

         for operand in operator_stack.into_iter() {
            if operand == Token::OpenParen {
                return InvalidFuncArgsSnafu {
                    err_msg: "Unmatched parentheses",
                }
                .fail();
            }
            result.push(operand);
         }

         Ok(result)
     }

66-90: Ensure robust error handling and configuration handling in eval method.

The method eval correctly checks the number of columns and their types but mentions a TODO about reading case-sensitive configurations which is not addressed in the code. This could lead to unexpected behavior if not implemented.

+    // TODO: Implement reading from a configuration to handle case sensitivity.
     fn eval(&self, _func_ctx: FunctionContext, columns: &[VectorRef]) -> Result<VectorRef> {
         ensure!(
             columns.len() == 2,
             InvalidFuncArgsSnafu {
                 err_msg: format!(
                     "The length of the args is not correct, expect exactly 2, have: {}",
                     columns.len()
                 ),
             }
         );
         ensure!(
             columns[1].len() == 1,
             InvalidFuncArgsSnafu {
                 err_msg: "The second argument should be a string literal",
             }
         );
         let pattern_vector = &columns[1]
             .cast(&ConcreteDataType::string_datatype())
             .context(InvalidInputTypeSnafu {
                 err_msg: "cannot cast `pattern` to string",
             })?;
         // Safety: both length and type are checked before
         let pattern = pattern_vector.get(0).as_string().unwrap();
         self.eval(columns[0].clone(), pattern)
     }

Likely invalid or redundant comment.

Tools

GitHub Check: Check typos and docs

[warning] 79-79:
"shoudl" should be "should".

src/common/function/src/scalars/matches.rs

codecov · 2024-06-27T08:12:57Z

Codecov Report

Attention: Patch coverage is 87.91594% with 69 lines in your changes missing coverage. Please review.

Project coverage is 84.61%. Comparing base (4a42371) to head (b9572ba).
Report is 24 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4222      +/-   ##
==========================================
- Coverage   85.06%   84.61%   -0.46%     
==========================================
  Files        1031     1047      +16     
  Lines      181303   185168    +3865     
==========================================
+ Hits       154233   156678    +2445     
- Misses      27070    28490    +1420

src/common/function/src/scalars/matches.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

coderabbitai

Actionable comments posted: 4

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between f31da99 and ab87f22.

Files selected for processing (1)

src/common/function/src/scalars/matches.rs (1 hunks)

Additional comments not posted (2)

src/common/function/src/scalars/matches.rs (2)

36-43: Implement Display trait for MatchesFunction

The implementation of the Display trait for MatchesFunction is straightforward and correct. It correctly returns the name of the function.

51-63: Review of return_type and signature methods

The return_type method correctly specifies that the function returns a boolean value, which is typical for match-like functions. The signature method properly defines the expected input types as two strings. Both methods are correctly implemented.

src/common/function/src/scalars/matches.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

coderabbitai

Actionable comments posted: 3

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ab87f22 and 55715ba.

Files selected for processing (1)

src/common/function/src/scalars/matches.rs (1 hunks)

Additional context used

GitHub Check: Check typos and docs

src/common/function/src/scalars/matches.rs

[warning] 128-128:
"unqualifed" should be "unqualified".

src/common/function/src/scalars/matches.rs

zhongzc · 2024-06-28T02:23:21Z

I think the following BNF can be used to guide writing Parser:

Stmt ::= OptionList MustList MustNotList

OptionList ::= ( BoolExpr )*
MustList ::= ( '+'Primary )*
MustNotList ::= ( '-'Primary )*

BoolExpr ::= AndExpr BoolExprTail
BoolExprTail ::= 'OR' AndExpr BoolExprTail | ε

AndExpr ::= Primary AndExprTail
AndExprTail ::= 'AND' Primary AndExprTail | ε

Primary ::= Term | '(' Stmt ')'

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

coderabbitai

Actionable comments posted: 7

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 55715ba and e932f31.

Files selected for processing (2)

src/common/function/src/scalars/matches.rs (1 hunks)
typos.toml (1 hunks)

Files skipped from review due to trivial changes (1)

typos.toml

Additional comments not posted (3)

src/common/function/src/scalars/matches.rs (3)
545-605: Unit tests for tokenization appear comprehensive.

The tests cover a variety of cases including basic operations, quoted strings, and complex expressions. Ensure that edge cases, especially with nested operations and unusual whitespace or character patterns, are also covered.

128-133: Typographical error in function name should be corrected.

The function from_unqualifed_fields is likely intended to be from_unqualified_fields. This is a critical typo that affects the function's discoverability and usage.
- DFSchema::from_unqualifed_fields(
+ DFSchema::from_unqualified_fields(
Likely invalid or redundant comment.

51-62: Review the return type and signature of MatchesFunction.

The function signature specifies that it expects two string inputs and returns a boolean. Ensure that this aligns with the intended use cases for matches. If there are scenarios where matches could take different types or a different number of arguments, consider implementing overloads or more flexible parsing.

Verification successful

The implementation of MatchesFunction aligns with the expected signature and return type.

The return_type method returns a boolean.

The signature method expects two string inputs.

There are no instances of incorrect or varied signatures found in the codebase.
Scripts executed

The following scripts were executed for the analysis:

Script:
#!/bin/bash
# Description: Verify the usage of `MatchesFunction` signature across the codebase.

# Test: Search for the function usage with incorrect or varied signatures.
rg --type rust $'matches'
Length of output: 93650

Script:
#!/bin/bash
# Description: Verify the usage of `MatchesFunction` signature across the codebase.

# Test: Search for the `MatchesFunction` specifically and its usage.
rg --type rust 'MatchesFunction'
Length of output: 403

Script:
#!/bin/bash
# Description: Extract the implementation details of `MatchesFunction` in `matches.rs`.

# Extract the `MatchesFunction` implementation.
ast-grep --lang rust --pattern $'impl Function for MatchesFunction {
  $$$
}' 
Length of output: 3829

Script:
#!/bin/bash
# Description: Verify the usage of `MatchesFunction` signature across the codebase.

# Test: Search for the usage of `MatchesFunction`.
rg --type rust 'MatchesFunction'
Length of output: 403

Script:
#!/bin/bash
# Description: Verify the usage of `matches` function across the codebase.

# Test: Search for the usage of the `matches` function.
rg --type rust 'matches('
Length of output: 94

Script:
#!/bin/bash
# Description: Verify the usage of `matches` function across the codebase.

# Test: Search for the usage of the `matches` function.
rg --type rust 'matches\('
Length of output: 1860

src/common/function/src/scalars/matches.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

coderabbitai

Actionable comments posted: 5

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between e932f31 and eee620c.

Files selected for processing (1)

src/common/function/src/scalars/matches.rs (1 hunks)

Additional comments not posted (4)

src/common/function/src/scalars/matches.rs (4)

36-38: Consider enhancing the struct documentation for MatchesFunction.

The struct is crucial for the text search functionality, but the documentation is quite sparse. Providing more detailed comments about its responsibilities and behavior could improve maintainability and clarity for future developers.

40-43: Simplify the implementation of fmt::Display for MatchesFunction.

The method is straightforward but consider if additional information might be useful for debugging or logging purposes. If not, this implementation is fine.

51-53: Good use of explicit type for return in return_type method.

Clearly specifies that the function returns a boolean type, which is expected for a match/no-match scenario in text search functions.

136-147: Review the structure and use of PatternAst.

The enums and their usage in PatternAst seem well-structured for representing different types of search patterns. Ensure that all cases are handled appropriately in the parsing logic.

src/common/function/src/scalars/matches.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

coderabbitai

Actionable comments posted: 6

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between eee620c and fa3f5de.

Files selected for processing (1)

src/common/function/src/scalars/matches.rs (1 hunks)

Additional comments not posted (1)

src/common/function/src/scalars/matches.rs (1)

40-43: Consider a more descriptive implementation for fmt::Display.

The current implementation of fmt::Display for MatchesFunction only returns the string "MATCHES". While this may be sufficient for debugging, consider whether additional information could be useful for logging or error messages.

src/common/function/src/scalars/matches.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

waynexia · 2024-07-01T06:55:26Z

@coderabbitai pause
@coderabbitai resolve
@coderabbitai configuration

coderabbitai

Actionable comments posted: 7

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between fa3f5de and b9572ba.

Files selected for processing (1)

src/common/function/src/scalars/matches.rs (1 hunks)

src/common/function/src/scalars/matches.rs

coderabbitai · 2024-07-01T07:06:39Z

@waynexia: I will pause the automatic reviews, resolve all current review comments, and show the current configuration settings.

Actions performed

Comments resolved.

Reviews paused.

waynexia · 2024-07-01T07:24:09Z

@coderabbitai configuration

coderabbitai · 2024-07-01T07:24:15Z

CodeRabbit configuration

language: en-US
tone_instructions: ''
early_access: false
enable_free_tier: true
reviews:
  profile: chill
  request_changes_workflow: false
  high_level_summary: true
  high_level_summary_placeholder: '@coderabbitai summary'
  auto_title_placeholder: '@coderabbitai'
  poem: true
  review_status: true
  collapse_walkthrough: false
  path_filters: []
  path_instructions: []
  abort_on_close: true
  auto_review:
    enabled: true
    auto_incremental_review: true
    ignore_title_keywords: []
    labels: []
    drafts: false
    base_branches: []
  tools:
    shellcheck:
      enabled: true
    ruff:
      enabled: true
    markdownlint:
      enabled: true
    github-checks:
      enabled: true
      timeout_ms: 90000
    languagetool:
      enabled: true
      disabled_rules:
        - EN_UNPAIRED_BRACKETS
        - EN_UNPAIRED_QUOTES
      disabled_categories:
        - TYPOS
        - TYPOGRAPHY
        - CASING
      enabled_only: false
      level: default
    biome:
      enabled: true
    hadolint:
      enabled: true
    swiftlint:
      enabled: true
    phpstan:
      enabled: true
      level: default
    golangci-lint:
      enabled: true
    yamllint:
      enabled: true
    gitleaks:
      enabled: true
    checkov:
      enabled: true
chat:
  auto_reply: true
knowledge_base:
  learnings:
    scope: auto
  issues:
    scope: auto
  jira:
    project_keys: []
  linear:
    team_keys: []

coderabbitai · 2024-07-01T07:29:59Z

@waynexia: I will pause the review, resolve all comments, and display the current configuration settings.

Actions performed

Comments resolved.

Reviews paused.

coderabbitai · 2024-07-01T07:30:01Z

Unable to process this comment: maximum number of iterations reached.

Analysis chain

@waynexia: I will pause the automatic reviews, resolve all current review comments, and show the current configuration settings.