BigQuery Destination : Fix GCS processing of Facebook data #9415

DoNotPanicUA · 2022-01-11T16:48:07Z

What

GCS BigQuery processing uses Avro files.
If field name contains char -, the loader fails due to invalid namesapce value.
Issue OnCall #86

How

Replace - to _ in the field name and use updated names in the namespaces.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

DoNotPanicUA · 2022-01-11T16:49:25Z

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1683599893
✅ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1683599893
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

DoNotPanicUA · 2022-01-11T16:49:38Z

/test connector=destination-s3

🕑 destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1683600796
✅ destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1683600796
No Python unittests run

DoNotPanicUA · 2022-01-11T16:49:47Z

/test connector=destination-gcs

🕑 destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1683601551
✅ destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1683601551
No Python unittests run

DoNotPanicUA · 2022-01-11T18:14:44Z

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1683958140
❌ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1683958140
🐛

The fail is not related to the change. Locally it works fine. The test instance can't reach airbyte/normalization:dev image.

The same test fail on the master: https://github.com/airbytehq/airbyte/actions/runs/1683960600

DoNotPanicUA · 2022-01-12T11:58:47Z

In addition, integration service has obvious issues. Which might lead to test failure.

DoNotPanicUA · 2022-01-12T17:19:37Z

/publish connector=connectors/destination-s3

DoNotPanicUA · 2022-01-12T17:19:51Z

/publish connector=connectors/destination-bigquery

DoNotPanicUA · 2022-01-12T17:19:59Z

/publish connector=connectors/destination-bigquery-denormalized

# Conflicts: # docs/integrations/destinations/s3.md

DoNotPanicUA · 2022-01-12T19:57:58Z

/publish connector=connectors/destination-s3

DoNotPanicUA · 2022-01-12T19:58:14Z

/publish connector=connectors/destination-bigquery

DoNotPanicUA · 2022-01-12T19:58:23Z

/publish connector=connectors/destination-bigquery-denormalized

DoNotPanicUA · 2022-01-13T13:13:07Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1692853484
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1692853484

DoNotPanicUA · 2022-01-13T13:13:16Z

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1692854408
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1692854408

DoNotPanicUA · 2022-01-13T13:13:23Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1692854866
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1692854866

DoNotPanicUA · 2022-01-13T14:59:49Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693321202
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693321202

DoNotPanicUA · 2022-01-13T16:00:30Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693612171
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693612171

DoNotPanicUA added 2 commits January 11, 2022 18:43

Fix GCS Avro file processing with invalid "-" character

6b343c6

Extend test data to cover the case

21641ba

DoNotPanicUA self-assigned this Jan 11, 2022

DoNotPanicUA requested review from tuliren and alexandr-shegeda January 11, 2022 16:48

github-actions bot added the area/connectors Connector related issues label Jan 11, 2022

This comment has been minimized.

Sign in to view

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 16:51 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 16:52 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 17:20 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 18:17 Inactive

tuliren approved these changes Jan 12, 2022

View reviewed changes

alexandr-shegeda approved these changes Jan 12, 2022

View reviewed changes

DoNotPanicUA temporarily deployed to more-secrets January 12, 2022 16:04 Inactive

incr ver

d3ea38e

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 12, 2022

DoNotPanicUA temporarily deployed to more-secrets January 12, 2022 17:10 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 17:21 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 17:22 Inactive

DoNotPanicUA added 2 commits January 12, 2022 21:55

Merge remote-tracking branch 'origin/master' into aleonets/oncall-86

18655ef

# Conflicts: # docs/integrations/destinations/s3.md

s3 ver upd

1245daf

DoNotPanicUA temporarily deployed to more-secrets January 12, 2022 19:58 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 20:00 Inactive

Merge remote-tracking branch 'origin/master' into aleonets/oncall-86

c6e6b88

DoNotPanicUA temporarily deployed to more-secrets January 13, 2022 13:14 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 13, 2022 13:15 Inactive

add dependency

ee5c236

DoNotPanicUA temporarily deployed to more-secrets January 13, 2022 15:00 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 13, 2022 15:02 Inactive

add dependency

121e6a7

DoNotPanicUA temporarily deployed to more-secrets January 13, 2022 15:57 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 13, 2022 16:02 Inactive

DoNotPanicUA temporarily deployed to more-secrets January 13, 2022 16:49 Inactive

DoNotPanicUA merged commit c904b44 into master Jan 13, 2022

DoNotPanicUA deleted the aleonets/oncall-86 branch January 13, 2022 17:16

octavia-squidington-iii mentioned this pull request Jan 20, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery Destination : Fix GCS processing of Facebook data #9415

BigQuery Destination : Fix GCS processing of Facebook data #9415

DoNotPanicUA commented Jan 11, 2022

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

This comment has been minimized.

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 11, 2022 •

edited

Loading

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

BigQuery Destination : Fix GCS processing of Facebook data #9415

BigQuery Destination : Fix GCS processing of Facebook data #9415

Conversation

DoNotPanicUA commented Jan 11, 2022

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

DoNotPanicUA commented Jan 11, 2022 • edited by github-actions bot Loading

This comment has been minimized.

DoNotPanicUA commented Jan 11, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 11, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 11, 2022 • edited Loading

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 12, 2022

DoNotPanicUA commented Jan 13, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 13, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 13, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 13, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 13, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 11, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 11, 2022 •

edited

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jan 13, 2022 •

edited by github-actions bot

Loading