Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Fix: accept empty schema #87

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

A-Costa
Copy link

@A-Costa A-Costa commented Jun 29, 2020

Closes #88

  • Fixes bugs in the code that caused KeyError exceptions when empty Schema is received
  • Implement handling of empty schema fields as Variant datatype in Snowflake

@A-Costa A-Costa changed the title Fix: accept empty schema [WIP] Fix: accept empty schema Jun 29, 2020
@A-Costa A-Costa changed the title [WIP] Fix: accept empty schema Fix: accept empty schema Jun 29, 2020
@koszti
Copy link
Contributor

koszti commented Jun 30, 2020

thanks for sending this PR. This is a small change but changing a core functionality hence it requires unit and integration tests as well.

Can you please add these tests? Please find some hints below:

  1. Unit tests: Add something into test_db_sync.py:test_flatten_schema where no type key in properties but it's an empty dict {}.

  2. Integration tests: You can get some inspiration from test_target_snowflake.py:test_nested_schema_flattening . We need to have a new test function with a new test json with empty schema types. The test should load the data from the json with empty schema type {}, and selecting the target table should return VARIANT columns.

Also I'm not sure how we should deal with anyOf? How we can decide which datatype is the good one for snowflake when creating the target table?

The expected behaviour for anyOf should also be covered by test cases.

@A-Costa A-Costa closed this Jul 6, 2020
@A-Costa A-Costa deleted the dev branch July 6, 2020 10:15
@A-Costa A-Costa restored the dev branch July 8, 2020 15:48
@A-Costa A-Costa reopened this Jul 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Handling empty schema
2 participants