S3 / GCS staging: use correct staging filename #11768

tuliren · 2022-04-06T18:16:27Z

What

When serialized buffer file is uploaded to blob storage, its UUID filename is simplified to <partId>.<extension>. However, the original UUID filename was used in the following steps, resulting in empty data.
This PR fixes this bug by returning the new filename.

🚨 User Impact 🚨

Currently, only Snowflake S3 staging is affected by this bug. But the buggy version is not published.
So no user impact is expected.

tuliren · 2022-04-06T18:17:01Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2104281062
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2104281062
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       168     31    82%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1403    520    63%

tuliren · 2022-04-06T18:23:37Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2104317977
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2104317977
No Python unittests run

tuliren · 2022-04-06T18:23:44Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2104318541
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2104318541
No Python unittests run

...destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java

...main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StagingSqlOperations.java

tuliren · 2022-04-06T22:26:15Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2105434854
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2105434854
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       168     31    82%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1403    520    63%

tuliren · 2022-04-06T22:26:20Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2105435244
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2105435244
No Python unittests run

tuliren · 2022-04-06T22:26:25Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2105435380
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2105435380
No Python unittests run

Return uploaded filename in staging

ff87231

github-actions bot added the area/connectors Connector related issues label Apr 6, 2022

Update variable names

17c3ab5

tuliren requested a review from ChristopheDuong April 6, 2022 18:28

ChristopheDuong reviewed Apr 6, 2022

View reviewed changes

...destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java Outdated Show resolved Hide resolved

ChristopheDuong reviewed Apr 6, 2022

View reviewed changes

...main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StagingSqlOperations.java Show resolved Hide resolved

tuliren added 2 commits April 6, 2022 15:23

Parse new filename from full object key

aeaac1e

Add back sentry tracing

5465f79

tuliren merged commit d778ac7 into master Apr 6, 2022

tuliren deleted the liren/fix-s3-filename branch April 6, 2022 23:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 / GCS staging: use correct staging filename #11768

S3 / GCS staging: use correct staging filename #11768

tuliren commented Apr 6, 2022 •

edited

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

S3 / GCS staging: use correct staging filename #11768

S3 / GCS staging: use correct staging filename #11768

Conversation

tuliren commented Apr 6, 2022 • edited Loading

What

🚨 User Impact 🚨

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 • edited by github-actions bot Loading

tuliren commented Apr 6, 2022 •

edited

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 6, 2022 •

edited by github-actions bot

Loading