Better concrete syntax semantics #671

danieltrt · 2024-06-03T16:43:59Z

Overview

This pull request introduces two significant changes to the semantics of the concrete syntax matching algorithm. These changes enhance the flexibility and expressiveness of the pattern matching with concrete syntax.

Change 1: Semantics of Template Variables `:[var]`

Previous Semantics

Previously, template variables :[var] could only match entire subtrees. This means it was not possible to match sequences of sibling nodes. For example, if matching [var] = 0; against int x = 0;, the match would fail because int and x are sibling nodes and not part of the same subtree.

New Semantics

Now template variables can match multiple sequential siblings at the same level. So matching
[var] = 0; with int x = 0; is possible.

Change 2: Semantics of Matching Concrete Template

Previous Semantics

Previously, the algorithm required that the concrete template match an entire node, and only one node. This restriction limited the pattern matching to single statements rather than sequences of statements. For example:

int :[var] = 0;
:[var]++;

This template could never match, because it corresponds to two sibling nodes, rather than a single subtree. For example, the snippet

{
  int x = 0;
  x++;
  print(x);
}

has this tree representation.

Notice that int x = 0; and x++; are sibling nodes. The template would never match just ONE node.

New Semantics

Now, the template can match entire sequential sibling nodes. This allows the template to match sequences of statements.

For example, the following code:

Code:

// just a regular loop in C++
int some = 0; 
while(some < 100) { 
    float length = 3.14;
    float area = length * length;
    some++; 
}
print(some);

matches the following template.

Template:

int :[var] = 0; 
while(:[var] < 100) { 
    :[body]
    
    :[var]++; 
}

and the alignment is as follow (loosely)

danieltrt · 2024-06-03T20:25:34Z

src/tests/test_piranha_java.rs

-    replace_node = "xs",
-    replace = "@xs, 2"
+    replace_node = "*",
+    replace = "println2(@xs, 2)"


Since :[x] can match multiple sequential siblings, this would be an infinite loop if we only rewrote :[x]

danieltrt · 2024-06-03T20:29:23Z

src/tests/test_piranha_java.rs

@@ -463,7 +463,10 @@ fn test_multiple_code_bases() {
  // Note that we expect 2 matches because we have 2 code bases, and each code base has 1 match.
  // We also have another codebase `folder_3` but the `paths_to_codebase` does not include it.
  assert_eq!(output_summaries.len(), 2);
-  assert_frequency_for_matches(&output_summaries, &HashMap::from([("match_import", 2)]));
+  assert_frequency_for_matches(&output_summaries, &HashMap::from([("match_import", 4)]));


The snippet

package org.piranha.examples; import java.util.List;

[0, 0] - [4, 0] [package_declaration] [0, 0] - [0, 29] [scoped_identifier] [0, 8] - [0, 28] scope: [scoped_identifier] [0, 8] - [0, 19] scope: [identifier] [0, 8] - [0, 11] name: [identifier] [0, 12] - [0, 19] name: [identifier] [0, 20] - [0, 28] [import_declaration] [2, 0] - [2, 22] [scoped_identifier] [2, 7] - [2, 21] scope: [scoped_identifier] [2, 7] - [2, 16] scope: [identifier] [2, 7] - [2, 11] name: [identifier] [2, 12] - [2, 16] name: [identifier] [2, 17] - [2, 21]

Now has two ways to match. Entire the entire sequential children of the import_declaration node, or the last node of the program node

danieltrt · 2024-06-03T20:43:42Z

src/models/concrete_syntax.rs

-          && recursive_matches[var_name].text.trim() != current_node_code.trim()
-        {
-          return (HashMap::new(), false);
+      let mut should_match = find_next_sibling(&mut tmp_cursor);


determine whether the next recursive call to get_matches_for_node should match any node. If there are no "siblings" left, then we should not match anything

danieltrt · 2024-06-04T00:20:13Z

src/models/concrete_syntax.rs

+        let mut last_node = the_node.child(the_node.child_count() - 1);
+        if let Some(last_node_index) = indx {
+          last_node = the_node.child(last_node_index);
+          matched = matched && (last_node_index != child_incr || the_node.child_count() == 1);


A bit convoluted and should be refactored.

Here we say, we match if:

we were able to align the sequence with the templates

the sequence is of length > 1, UNLESS the node only has one child

src/models/concrete_syntax.rs

ketkarameya · 2024-06-08T19:49:18Z

src/models/concrete_syntax.rs

+///
+/// 1. Initialize cursor to the first child and iterate through siblings.
+/// 2. Use `get_matches_for_node` to attempt matching the template against the subtree starting at each sibling.
+/// 3. If a match is found, determine the range of matched nodes and return the match mapping, status, and range.


ketkarameya · 2024-06-08T19:49:50Z

src/models/concrete_syntax.rs

+/// 4. If no match is found, return an empty mapping, false status, and None for range.
+pub(crate) fn match_sequential_siblings(
+  cursor: &mut TreeCursor, source_code: &[u8], meta: &ConcreteSyntax,
+) -> (HashMap<String, CapturedNode>, bool, Option<Range>) {


you could make a struct for this

ketkarameya · 2024-06-08T19:50:14Z

src/models/concrete_syntax.rs

+/// 3. If a match is found, determine the range of matched nodes and return the match mapping, status, and range.
+/// 4. If no match is found, return an empty mapping, false status, and None for range.
+pub(crate) fn match_sequential_siblings(
+  cursor: &mut TreeCursor, source_code: &[u8], meta: &ConcreteSyntax,


rename to cs

ketkarameya · 2024-06-08T19:51:24Z

src/models/concrete_syntax.rs

+        let range = Range::from_siblings(cursor.node().range(), last_node.unwrap().range());
+        return (
+          mapping,
+          last_node_index != child_incr || parent_node.child_count() == 1,


can u add a comment around this ?
Just extract this condition into a variable, document it and pass the variable.

src/models/concrete_syntax.rs

ketkarameya · 2024-06-08T20:02:38Z

src/models/concrete_syntax.rs


-      if let (mut recursive_matches, true) =
-        get_matches_for_node(&mut tmp_cursor, source_code, &meta_advanced)
+    let mut tmp_cursor = cursor.clone();


codument - why this clone

ketkarameya · 2024-06-08T20:02:55Z

src/models/concrete_syntax.rs

+    let mut should_match = find_next_sibling(&mut tmp_cursor); // Advance the cursor to match the rest of the template
+    let mut is_final_sibling = false;
+    loop {
+      let mut walkable_cursor = tmp_cursor.clone();


ketkarameya · 2024-06-08T23:50:54Z

src/models/concrete_syntax.rs

+  parent_node
+    .children(&mut parent_node.walk())
+    .enumerate()
+    .find_map(|(i, child)| {


use filter instead of if inside find_map(...)

ketkarameya · 2024-06-08T23:58:13Z

src/models/matches.rs

@@ -321,6 +321,15 @@ impl Range {
      end_point: position_for_offset(source_code.as_bytes(), mtch.end()),
    }
  }
+
+  pub(crate) fn from_siblings(left: tree_sitter::Range, right: tree_sitter::Range) -> Self {


is this function like merging two ranges?

Co-authored-by: Ameya Ketkar <94497232+ketkarameya@users.noreply.github.com>

… struct

ketkarameya · 2024-06-13T23:04:46Z

src/models/concrete_syntax.rs

+  cursor: &mut TreeCursor, source_code: &[u8], cs: &ConcreteSyntax,
+) -> Option<MatchResult> {
+  let parent_node = cursor.node();
+  let mut child_incr = 0;


could be renamed to child_seq_match_start

danieltrt added 5 commits May 28, 2024 21:39

Better cs

9432c58

Test case for sequential sibling matching

9976491

Improvements to concrete syntax

6064515

Improvements to concrete syntax

2f3b1c7

cargo beign annoying

592ca31

danieltrt requested review from ketkarameya and dvmarcilio June 3, 2024 16:44

danieltrt and others added 3 commits June 3, 2024 11:14

Merge branch 'master' into better_concrete

a6c6896

Update test_piranha_java.rs

5b21060

Fix test case due to the semantic changes

4d11b09

danieltrt commented Jun 3, 2024

View reviewed changes

danieltrt added 3 commits June 3, 2024 14:22

Bug fix caught using python test, as well as add a new test

f9baaee

Remove infinite loop due to new CS syntax

e871518

regression

f3b708d

danieltrt commented Jun 4, 2024

View reviewed changes

danieltrt added 4 commits June 3, 2024 20:44

Refactoring code and updating docs

2c2e101

Further simplify functions

229ed32

Further simplify functions

ef2fe9d

Further simplify functions

34d7368

danieltrt changed the title ~~[WIP] Better concrete syntax semantics~~ Better concrete syntax semantics Jun 4, 2024

Merge branch 'master' into better_concrete

e49afbe

ketkarameya reviewed Jun 8, 2024

View reviewed changes

danieltrt and others added 4 commits June 8, 2024 20:17

Update src/models/concrete_syntax.rs

ccb1ebd

Co-authored-by: Ameya Ketkar <94497232+ketkarameya@users.noreply.github.com>

Renaming meta to cs, as well as improving variable names

ec33478

Better documentation and refactoring functional call results to use a…

894d5f6

… struct

Better documentation and refactoring functional call results to use a…

a873865

… struct

danieltrt mentioned this pull request Jun 11, 2024

Seeking feedback on feature flag cleanup approach with the Piranha framework #668

Open

ketkarameya reviewed Jun 13, 2024

View reviewed changes

Update concrete_syntax.rs

86c7ac4

ketkarameya approved these changes Jun 14, 2024

View reviewed changes

danieltrt requested review from dvmarcilio, stefanheule and yuxincs and removed request for dvmarcilio June 14, 2024 00:43

danieltrt added 3 commits June 27, 2024 07:15

Change syntax for :[args+] to handle multiple node matching

f4177a7

Add \n back to plugin

1abadcf

Function rename for clarity.

cf1c1e0

stefanheule approved these changes Jul 1, 2024

View reviewed changes

Merge branch 'master' into better_concrete

140f084

danieltrt merged commit 3f79dcb into master Jul 1, 2024
10 checks passed

danieltrt deleted the better_concrete branch July 2, 2024 13:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better concrete syntax semantics #671

Better concrete syntax semantics #671

danieltrt commented Jun 3, 2024 •

edited

Loading

danieltrt Jun 3, 2024 •

edited

Loading

danieltrt Jun 3, 2024

danieltrt Jun 3, 2024

danieltrt Jun 4, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 8, 2024

ketkarameya Jun 13, 2024

Better concrete syntax semantics #671

Better concrete syntax semantics #671

Conversation

danieltrt commented Jun 3, 2024 • edited Loading

Overview

Change 1: Semantics of Template Variables :[var]

Previous Semantics

New Semantics

Change 2: Semantics of Matching Concrete Template

Previous Semantics

New Semantics

danieltrt Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danieltrt commented Jun 3, 2024 •

edited

Loading

Change 1: Semantics of Template Variables `:[var]`

danieltrt Jun 3, 2024 •

edited

Loading