Change Lexing/Parsing of embedded docs to not eagerly validate #507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev-ion-doc #507 +/- ##
===============================================
+ Coverage 80.84% 80.86% +0.01%
===============================================
Files 80 80
Lines 19398 19401 +3
Branches 19398 19401 +3
===============================================
+ Hits 15683 15689 +6
- Misses 3291 3293 +2
+ Partials 424 419 -5 ☔ View full report in Codecov by Sentry. |
Conformance comparison report
Number passing in both: 5732 Number failing in both: 611 Number passing in Base (3f9d17f) but now fail: 0 Number failing in Base (3f9d17f) but now pass: 0 |
jpschorr
force-pushed
the
feat-embedded-doc
branch
from
e72d63b
to
6f4edd2
Compare
jpschorr
force-pushed
the
feat-embedded-doc
branch
from
6f4edd2
to
6ae75a7
Compare
jpschorr
commented
Oct 17, 2024
partiql-ast/src/pretty.rs
Outdated
| @@ -394,7 +394,7 @@ impl PrettyDoc for Lit { | |||
| Lit::FloatLit(inner) => arena.text(inner.to_string()), | |||
| Lit::DoubleLit(inner) => arena.text(inner.to_string()), | |||
| Lit::BoolLit(inner) => arena.text(inner.to_string()), | |||
| Lit::IonStringLit(inner) => inner.pretty_doc(arena), | |||
| Lit::EmbeddedDocLit(inner) => inner.pretty_doc(arena), // TODO better pretty for embedded doc | |||
There was a problem hiding this comment.
Work is needed to do type-specific pretty printing for embedded documents.
| @@ -1978,6 +1978,7 @@ fn lit_to_value(lit: &Lit) -> Result<Value, AstTransformError> { | |||
| Ok(val) | |||
| } | |||
|
|
|||
| // TODO | |||
There was a problem hiding this comment.
This will be moved/removed in a future PR as part of this dev-ion-doc feature branch.
am357
approved these changes
Oct 18, 2024
| @@ -46,7 +46,7 @@ time = { version = "0.3", features = ["macros"] } | |||
| criterion = "0.5" | |||
| rand = "0.8" | |||
|
|
|||
| assert_matches = "1.5" | |||
| assert_matches = "1" | |||
There was a problem hiding this comment.
Just to double-check: Do we want to move from 1.5 to 1?
There was a problem hiding this comment.
It's what all other crates use:
13:50❯ rg assert_matches -g "*.toml"
partiql-eval/Cargo.toml
35:assert_matches = "1"
partiql/Cargo.toml
49:assert_matches = "1"
partiql-logical-planner/Cargo.toml
42:assert_matches = "1"
partiql-parser/Cargo.toml
50:assert_matches = "1"
Co-authored-by: Arash Maymandi <27716912+am357@users.noreply.github.com>
alancai98
approved these changes
Oct 18, 2024
| LineAndColumn::from(offset_tracker.at(query, BytePosition::from(1)).unwrap()), | ||
| LineAndColumn::new(1, 2).unwrap() | ||
| ); | ||
| // ion is not eagerly parsed, so unterminated ion does not cause a lex/parse error |
There was a problem hiding this comment.
When will an error be thrown for the unterminated Ion string? Probably when the contents of the document is later used?
jpschorr
added a commit
that referenced
this pull request
Oct 22, 2024
This changes the lexer and parser to pass through strings enclosed in backticks un-parsed. (At current, these documents are parsed during lowering). Since embedded documents may themselves contain backticks, beginning and ending delimiters consist of an arbitrary odd numbers of backticks (e.g., `` ` ``, `` ``` ``, `` ````` `` etc.) that must be paired (e.g., `` `$ion_data_here::[]` ``, `` ```$ion_data_here::[ $string_with_embedded_backtick:"`" ]``` ``, etc.). As opening and closing delimiters are required to be odd in count of backticks, a contiguous string of backticks that is even is interpreted as an empty document.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes the lexer and parser to pass through strings enclosed in backticks un-parsed. (At current, these documents are parsed during lowering).
Since embedded documents may themselves contain backticks, beginning and ending delimiters consist of an arbitrary odd numbers of backticks (e.g.,
`,```,`````etc.) that must be paired (e.g.,`$ion_data_here::[]`,```$ion_data_here::[ $string_with_embedded_backtick:"`" ]```, etc.).As opening and closing delimiters are required to be odd in count of backticks, a contiguous string of backticks that is even is interpreted as an empty document.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.