Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flink avro timestamp_mapping.legacy config is not supported w/ filesystem connector #968

Closed
henneberger opened this issue Nov 5, 2024 · 1 comment
Labels
bug Something isn't working
Milestone

Comments

@henneberger
Copy link
Contributor

henneberger commented Nov 5, 2024

The filesystem connector does not acknowledge the timestamp_mapping.legacy flag for the avro format so avro records will fail to deserialize certain data types such as timestamp-millis. There currently is no workaround other than to disable the timestamp_mapping.legacy option.

CREATE TABLE ...
) WITH (
  'format' = 'avro',
  'path' = '...',
  'connector' = 'filesystem',
  'avro.timestamp_mapping.legacy' = 'false',
  'avro.encoding' = 'json'
);
@github-project-automation github-project-automation bot moved this to Backlog in DataSQRL Nov 5, 2024
@henneberger henneberger added the bug Something isn't working label Nov 5, 2024
@henneberger henneberger changed the title Flink Legacy timestamp timestamp fix is not supported when reading data with the filesystem connector Flink avro timestamp_mapping.legacy config is not supported w/ filesystem connector Nov 6, 2024
@henneberger henneberger added this to the v0.5.8 milestone Nov 8, 2024
@henneberger
Copy link
Contributor Author

henneberger commented Nov 8, 2024

Fixed in #969. We will use the new type mapping by default for all .avro schemas

@github-project-automation github-project-automation bot moved this from Backlog to Done in DataSQRL Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

1 participant