🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering #7719

sherifnada · 2021-11-08T01:32:52Z

What

JDBC Destinations, when running in INSERT mode, typically write records as follows:

Accumulate records in a buffer that can grow up to 10_000 records in size
Flush buffer to memory when it's full
Flush buffer once the sync is complete

This approach is problematic because it does not take into account how large those records are, which means wide rows can be problematic (10k 5mb rows has very different memory implications than 10k 4kb rows). This is especially exacerbated because when flushing records to the destination, the JDBC drivers needs to hold a SQL string containing all the rows in memory (i.e: INSERT xyz INTO table where xyz is the content of all the rows), so if the queue is about to write N bytes, we actually need to have a multiple of N bytes capacity in memory (somewhere between 2 and 3 as far as I can tell looking under the JDBC hood).

Recently, a user was syncing very wide rows from a Postgres DB which caused this issue to appear. So this PR changes the buffering strategy to count bytes instead of number of records.

I've confirmed this fix works for the user in question.

How

There are two changes being made here:

keep track of the buffer size in bytes and flush when the size has reached the maximum alloted. This requires an extra serialization of the input records. This is ineffecient but didn't seem to be a huge cost when I was profiling. We definitely need more rigorous measurement of the CPU overhead of this (and we can probably even remove the need for it) but I think we should do that separately, as this fix unblocks using Snowflake destination for this category of data sources. We can ship a more CPU efficient solution as an incremental improvement. I've set the default buffer size to 256Mib. This way even if we hit a 3x multiple, we are using less than 1Gib of memory.
When writing records, cap all SQL INSERT statements to at most 10k records because some destinations like Snowflake have limitations on how many records can be written at a time (snowflake limits to 16k).

Pre-merge Checklist

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

…snowflake-oom

sherifnada · 2021-11-08T01:33:57Z

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1432867826
❌ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1432867826
🐛 https://gradle.com/s/tpzamgexz3aho

sherifnada · 2021-11-08T01:34:05Z

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1432868024
❌ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1432868024
🐛 https://gradle.com/s/nqciyaqdr4o7m

sherifnada · 2021-11-08T01:34:12Z

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1432868262
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1432868262
🐛 https://gradle.com/s/xkyrnd3fder56

sherifnada · 2021-11-08T01:34:17Z

/test connector=destination-mysql

🕑 destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1432868483
❌ destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1432868483
🐛 https://gradle.com/s/pb3hu46aedkcy

...ava/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.java

airbyte-integrations/connectors/destination-snowflake/build.gradle

…e-based-buffering-destinations

sherifnada · 2021-11-08T22:09:56Z

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1436924952
✅ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1436924952
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%

…e-based-buffering-destinations

sherifnada · 2021-11-09T02:57:53Z

/publish connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1437738459
✅ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1437738459

sherifnada · 2021-11-09T02:57:59Z

/publish connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1437738541
✅ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1437738541

sherifnada · 2021-11-09T02:58:08Z

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1437738884
✅ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1437738884

…e wide rows by using byte-based record buffering (airbytehq#7719)

davinchia and others added 12 commits November 4, 2021 00:11

Set up for remote debugging.

99ce58c

Merge branch 'master' of github.com:airbytehq/airbyte into davinchia/…

4b95edd

…snowflake-oom

Merge branch 'master' of github.com:airbytehq/airbyte into davinchia/…

677c765

…snowflake-oom

Merge branch 'master' of github.com:airbytehq/airbyte into davinchia/…

a2405c5

…snowflake-oom

Merge branch 'master' of github.com:airbytehq/airbyte into davinchia/…

6f77044

…snowflake-oom

yourkit

f2b553b

save repro

ee48300

save

3d6214e

Merge branch 'master' of github.com:airbytehq/airbyte into davinchia/…

6de777e

…snowflake-oom

WIDE ROWS BOW DOWN TO ME

3bd9d09

remove unused test

66be3c0

uncomment yourkit

0c39dbb

sherifnada requested review from cgardens, ChristopheDuong and tuliren November 8, 2021 01:32

github-actions bot added the area/connectors Connector related issues label Nov 8, 2021

jrhizor temporarily deployed to more-secrets November 8, 2021 01:36 Inactive

tuliren approved these changes Nov 8, 2021

View reviewed changes

...ava/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.java Show resolved Hide resolved

airbyte-integrations/connectors/destination-snowflake/build.gradle Outdated Show resolved Hide resolved

return max jvm percentage flag

231c757

sherifnada temporarily deployed to more-secrets November 8, 2021 08:23 Inactive

sherifnada added 2 commits November 8, 2021 14:08

Merge branch 'master' of github.com:airbytehq/airbyte into sherif/byt…

f23b9a2

…e-based-buffering-destinations

renames

71eb71a

jrhizor temporarily deployed to more-secrets November 8, 2021 22:32 Inactive

docsdocsdocs

18bd969

sherifnada changed the title ~~Buffer records in destinations based on byte size~~ 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering Nov 8, 2021

sherifnada added 3 commits November 8, 2021 16:00

Merge branch 'master' of github.com:airbytehq/airbyte into sherif/byt…

5034b82

…e-based-buffering-destinations

Merge branch 'master' of github.com:airbytehq/airbyte into sherif/byt…

b2a4276

…e-based-buffering-destinations

use correct normalization image name

70194e8

github-actions bot added area/documentation Improvements or additions to documentation area/platform issues related to the platform area/worker Related to worker labels Nov 9, 2021

sherifnada temporarily deployed to more-secrets November 9, 2021 02:57 Inactive

jrhizor temporarily deployed to more-secrets November 9, 2021 02:59 Inactive

jrhizor temporarily deployed to more-secrets November 9, 2021 03:00 Inactive

sherifnada merged commit efb5151 into master Nov 9, 2021

sherifnada deleted the sherif/byte-based-buffering-destinations branch November 9, 2021 03:26

sherifnada mentioned this pull request Nov 9, 2021

Update connector versions #7771

Merged

sherifnada added a commit that referenced this pull request Nov 9, 2021

Update connector versions from #7719 (#7771)

363a047

jrhizor mentioned this pull request Nov 9, 2021

Bump Airbyte version from 0.30.35-alpha to 0.30.36-alpha #7772

Merged

sherifnada mentioned this pull request Nov 9, 2021

Fix BufferedStreamConsumer tests #7773

Merged

sherifnada mentioned this pull request Nov 17, 2021

[EPIC] Use byte-based buffering in all java destinations #8039

Closed

5 tasks

alexandr-shegeda mentioned this pull request Nov 22, 2021

🎉 Destination BigQuery: Use byte-based buffering batching #8174

Closed

4 tasks

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handl…

29a063f

…e wide rows by using byte-based record buffering (airbytehq#7719)

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

Update connector versions from airbytehq#7719 (airbytehq#7771)

0a00332

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering #7719

🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering #7719

sherifnada commented Nov 8, 2021 •

edited

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading

🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering #7719

🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering #7719

Conversation

sherifnada commented Nov 8, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

sherifnada commented Nov 8, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 8, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 8, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 8, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 8, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 9, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 9, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 9, 2021 • edited by github-actions bot Loading

sherifnada commented Nov 8, 2021 •

edited

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 8, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading

sherifnada commented Nov 9, 2021 •

edited by github-actions bot

Loading