Adds a 1.1 ValueWriter setting for writing symbol text inline
#804
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zslayton
commented
Aug 10, 2024
Comment on lines
465
to
470
| let header = type_descriptor.to_header().ok_or_else(|| { | ||
| IonError::decoding_error(format!( | ||
| "found a non-value in value position; buffer=<{:X?}>", | ||
| input.bytes_range(0, 16.min(input.bytes().len())) | ||
| )) | ||
| })?; |
There was a problem hiding this comment.
🗺️ The old error message was maddening. Now you can at least see the unexpected opcode.
Comment on lines
+77
to
+79
| if input.len() < flex_sym_end { | ||
| return IonResult::incomplete("reading a FlexSym", offset); | ||
| } |
There was a problem hiding this comment.
🗺️ If the buffer ended in the middle of a FlexSym, it would lead to a panic instead of reporting Incomplete.
src/lazy/encoder/writer.rs
Outdated
Comment on lines
210
to
231
| if self.encoding.symbol_creation_policy == SymbolCreationPolicy::WriteProvidedToken { | ||
| if self.has_inline_symbol_text() { |
There was a problem hiding this comment.
🗺️ has_inline_symbol_text is more specific than the initial SymbolCreationPolicy. It may be that I can remove that concept, but if so I'll do that in another PR.
src/lazy/encoder/writer.rs
Outdated
| // Store the tokens as they are. Text will be written as text, symbol IDs will be written | ||
| // as symbol IDs. TODO: Lookup SIDs to see if they have text? | ||
| // as symbol IDs. | ||
| // TODO: Replace SIDs w/text if they're defined? (Config setting?) |
There was a problem hiding this comment.
🗺️ If the caller passes a SID, we write it as-is. At a minimum, we should validate that it's in range.
Comment on lines
-22
to
-36
| let item = reader.next_item()?; | ||
| match item { | ||
| SystemStreamItem::VersionMarker(marker) => { | ||
| // Note that this uses the Ion version with which the IVM was encoded rather than | ||
| // the Ion version the stream is switching to. We can do this because the format | ||
| // (i.e text or binary) stays the same when the version changes. | ||
| // TODO: Use new encoding, once we have APIs to get new/old encodings for the marker. | ||
| let is_human_readable = marker.encoding().is_text(); | ||
| let value = reader.expect_next_value()?; | ||
| let value_deserializer = ValueDeserializer::new(&value, is_human_readable); | ||
| T::deserialize(value_deserializer) | ||
| } | ||
| SystemStreamItem::Value(value) => { | ||
| let value_deserializer = ValueDeserializer::new(&value, true); | ||
| T::deserialize(value_deserializer) |
There was a problem hiding this comment.
🗺️ This old implementation would return an error if an LST preceded the first value.
popematt
reviewed
Aug 11, 2024
popematt
approved these changes
Aug 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on the changes in #802.
To start, the setting currently governs symbol values and annotations. There is not yet a way to:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.