[BEAM-12883] Add coder for ReadableFileCoder that supports MetadataCoderV2 #15510
Conversation
… add-coder-readable-file-v2 # Conflicts: # sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ReadableFileCoderV2.java
|
Maybe it makes better sense to ensure the cc: @reuvenlax |
@pabloem why initially this separation was needed between the coders? |
|
I think the only way to preserver backwards compatibility here is to add a new coder and also add an opt-in mechanism for users to select this new coder. It may be worth considering making this coder more extensible so this doesn't happen in the future (e.g. storing the values as a key-value map and ignoring unknown fields, which is less efficient but probably doesn't matter for this coder vs. the chances that we'll want to add a new field in the future.) |
|
Creating more extendable coder is definitely better, I would suggest the bellow:
We can make this also in 2 phases-PRs, (start with number 2 just to solve the current issue and then enhance MetadataCoder for next future fields) |
|
alright taking a look... |
| import org.apache.beam.sdk.coders.VarIntCoder; | ||
| import java.util.Collections; | ||
| import java.util.List; | ||
| import org.apache.beam.sdk.coders.*; |
There was a problem hiding this comment.
this asterisk-based import is causing the PreCommit to fail (we try to avoid this sort of import) - can you import only the classes you need?
|
LGTM. @brachi-wernick are you able to follow up with change #1? |
I will work on it and open a diff PR for it and link it to that PR thanks! |
In case you missed it, I submitted a PR #15699 for change 1. |
…bleFileCoder that supports MetadataCoderV2 * add coder readable files * add coder readable files * :sdks:java:core:spotlessApply * [BEAM-12883] add ability tp set custom MetadataCoder for ReadableFileCoder * [BEAM-12883] add ability tp set custom MetadataCoder for ReadableFileCoder * [BEAM-12883] add ability t0 set custom MetadataCoder using StructuredCoder * [BEAM-12883] remove asterisk-based import Co-authored-by: brachipa <brachipa@moonactive.com>
Currently, there is a separated coder for Metadata which cares about
Metadata#lastModifiedMillis().The issue is that the
ReadableFileCoderalways uses the defaultMetadatacoder.We need to create a similar coder for
ReadableFileCoder:ReadableFileCoderV2which just use the correspondingMetadataCoderV2.Without doing so
lastModifiedMilliswill remain zero.There is also no way to create custom coder in our side since
FileIO.ReadableFilehas no public method to initiate new instance of it.Also check this thread in StackOverflow