Update destinations to handle new state messages #13670
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgardens
requested review from
tuliren,
subodh1810 and
jdpgrailsdev
and removed request for
tuliren,
subodh1810 and
jdpgrailsdev
cgardens
force-pushed
the
cgardens/collect-state-messages-in-dest
branch
from
128d4c0
to
1c2e2e2
Compare
cgardens
requested review from
edgao,
tuliren,
grishick,
jdpgrailsdev and
subodh1810
|
Just tagged all of the Dbs/Dws (@grishick , @tuliren , @subodh1810 , @edgao ) team because I didn't know who the right person to review this would be. Happy to add more context as needed. Would really appreciate a review by any one person on the team on Monday. Also tagged @jdpgrailsdev as he is doing the analogous implementation here on the source side. |
cgardens
commented
Jun 11, 2022
...e/integrations/destination/dest_state_lifecycle_manager/DestStreamStateLifecycleManager.java
Outdated
Show resolved
Hide resolved
cgardens
changed the title
tuliren
approved these changes
Jun 12, 2022
| // still succeed. | ||
| if (lastFlushedToTmpDstState == null) { | ||
| // if no state was emitted (i.e. full refresh) and if there were still no failures, we claim | ||
| // success. this is the full refresh case. if there was no state AND a failure, we assume failure. |
There was a problem hiding this comment.
Nitpick. This comment is just describing the first if branch. It does not add more information than the code itself. Seems redundant and can be removed.
There was a problem hiding this comment.
updated the comment. there is a piece of product context that isn't obvious from the conditional that i want to capture. i think the new comment might help.
| * | ||
| * @return list of state messages | ||
| */ | ||
| Queue<AirbyteMessage> listFlushed(); |
There was a problem hiding this comment.
Nitpick. In test, it is nice to have this listFlushed method to inspect the elements. However, in production, we only need to know whether there is any flushed state or not. What about adding a hasFlushed method specifically for production to eliminate the overhead of creating those lists over and over?
| Comparator.nullsFirst(Comparator.naturalOrder())) // namespace is allowed to be null | ||
| .thenComparing(entry -> entry.getKey().getName())) | ||
| .map(Entry::getValue) | ||
| .collect(Collectors.toCollection(LinkedList::new)); |
There was a problem hiding this comment.
Nitpick. When the size of the list is known and the list is immutable, ArrayList has a lower memory overhead than LinkedList.
There was a problem hiding this comment.
ArrayList doesn't implement Queue though.
jdpgrailsdev
approved these changes
Jun 13, 2022
LGTM with one small nit.
...e/integrations/destination/dest_state_lifecycle_manager/DestSingleStateLifecycleManager.java
Outdated
Show resolved
Hide resolved
subodh1810
approved these changes
Jun 14, 2022
There was a problem hiding this comment.
This makes sense.
Not in the scope of the PR but just a thought : Have we thought about how we would want to send acknowledgement to the source server if it comes as a use case. For instance imagine a source connector needs to clean up things on the source server but in order to do that it needs to know that the data it just synced has been successfully committed to the destination. Just a thought.
pdulapalli
added a commit
to falkonai/airbyte
that referenced
this pull request
Jun 29, 2022
* source-notion: Updated docs as per new format (#13708) * Keep doc up-to-date with changes (#13667) `airbyte-db/lib` has been renamed to `airbyte-db/db-lib` * Updated BigQuery, Google Sheets, Facebook Marketing, Salesforce docs (#13717) * initial changes * Edited google Sheets doc * More edits * edited the intro and prereqs for BigQuery * edited the data loading section * more edits * Grammatical edits * Formatting edits * Adds zombie removal tool (#13718) * Adds zombie removal tool * Corrects endpoint adds comments * Adds API links * Changes search logic, fixes escape character * Corrects help text * Fix connection read object building (#13568) Make sure the sourceCatalogId field is set with the value stored in the database. * fix modal overlay problem (#13724) * Improve Gradle build and fix storybook (#13719) * Improve Gradle build and fix storybook * Run all copy tasks after copyDocker * fix stream descriptor typo (#13726) * Save streams to reset in job config when creating reset job (#13703) * save streams to reset in job config when creating reset job * change streamDescriptors to streamsToReset * :bug: Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655) * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * add timestamp datatype test * Bump version in Dockerfile * auto-bump connector version Co-authored-by: grishick <greg@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump version in Dockerfile to match source-postgres version (#13732) * Fail the connection state instead of throwing an exception (#13728) * Fail the connection state instead of throwin exception * Fix test * rm import * Update airbyte-workers/src/main/java/io/airbyte/workers/general/DefaultCheckConnectionWorker.java Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Evan Tahler <evan@airbyte.io> * [low-code connectors] add a few unit tests (#13666) * add a couple of unit tests * refactor tests * 🎉 Destination databricks: rename to databricks lakehouse (#13722) * Update Databricks naming * Update destination_spec * Update BOOTSTRAP.md * Update Dockerfile * Update README.md * Update spec.json * Update databricks.md * Update databricks.md * Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md Co-authored-by: LiRen Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren.git@outlook.com> * Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745) * Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump source-postgres-strict-encrypt version to 0.4.23 to match source-postgres version (#13747) * S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483) * S3 destination: Updating processing data types for Avro/Parquet formats * S3 destination: handle comparing data types * S3 destination: clean code * S3 destination: clean code * S3 destination: handle case with unexpected json schema type * S3 destination: clean code * S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class * S3 destination: clean code * S3 destination: clean code * GCS destination: Update data types processing for Avro/Parquet formats * GCS destination: clean redundant code * S3 destination: handle case with numbers inside array * S3 destination: clean code * S3 destination: add unit test * S3 destination: update unit test cases with number types. * S3 destination: update unit tests. * S3 destination: bump version for s3 and gcs * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Salesforce: fix sync capped streams with more records than page size (#13658) * change logic for counting records * update doc * correct unit test * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Docs: update normalization doc (#13739) * update dbt version in custom normalization (#13742) * Kube fix and Docs for open telemetry integration (#13701) * Create interface, factory for metric client * remove unused func * change count val to use long * PR fix * otel metric client implementation * merge conflicts resolve * build fix * add a test, moved version into deps catalog * fix test * add docs for open telemetry * fix kube setting for otel, and add doc * helm related fields update for opentel * cdk: fix typo in build.gradle.hbs (#13761) * Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha (#13759) Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> * Update destinations to handle new state messages (#13670) * 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Postgres source strict encrypt trucnated precision bump version (#13769) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version * Postgres source strict encrypt bump version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Migrate StateDB to support per stream states (#13731) * Update StateDB to support per Stream states. * Add `StateType` type * Add `steam_name`, `namespace` and `type` to `state` table. * Set the default StateType to LEGACY * Update getting-started-with-airbyte-cloud.md (#13741) * Update getting-started-with-airbyte-cloud.md Edited Getting Started with Airbyte Cloud guide to match the updated Cloud UI. * Update getting-started-with-airbyte-cloud.md Updated based on Amruta's suggestions. * 🐛 Normalization correctly propagates deletions to the final tables (#12846) * Update Airbyte Protocol Docs (#13709) * Source Facebook marketing: fix `not syncing any data` issue (#13749) * #253 oncall source fb marketing: fix issue * #253 oncall: upd changelog * Update docs/integrations/sources/facebook-marketing.md * auto-bump connector version Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Ignore custom connectors during connectors update (#13581) * Ignore Custom connectors during connectors update * Skip any connector flagged as custom as well * Format * [low-code connectors] Support for list-based stream slicers (#13668) * list slicer * Add comment * test * ast eval * refactor * fix * remove print * rename * return a generator * smaller fix and reset to master * union type hint * assert * comment * format * 📝 Postgres source: add troubleshooting for postgres hot standby (#13784) * Add troubleshooting section in postgres doc * Update doc * Add one more solution * upgrade debezium version for postgres to 1.9.2 (#13368) * upgrade debezium version for postgres to 1.9.2 * fix test+build * fix build * address review comments * update link to docs * fix test * 🐛 Source Amplitude: Fix an infinite loop (#13638) * Fix infinite loop when fetching Amplitude data * Update changelog and version * Address review * fix: unit tests were failing * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * octavia-cli: allow user to set custom API HTTP headers(#12893) * 🎉 Source Github: Fix API sorting, fix `get_starting_point` caching (#13707) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 🐛 Source Zendesk Support: fixed 429 error for `TicketMetrics` stream (#13757) * Add some dev-facing normalization docs (#13780) * Fix mysql cdc acceptance test (#13795) * fix mysql cdc acceptance test * remove comments * doc: add link to forum in plopfile (#13560) * Make connector adaptable based on deployment mode (#13522) * Add deployment mode to env shared with jobs * Add adaptive runners * Migrate postgres source to use adaptive runner * Add an array of specs in docker image spec definition * Add copyright * Parse docker image spec with specs list * Update spec yaml files * Pass in DEPLOYMENT_MODE to docker compose file * Revert "Parse docker image spec with specs list" This reverts commit 8fe41dd3b7fa5306ba8b62d660c002420adb9b23. * Revert changes in docker image spec * Read cloud specific spec files based on deployment mode * Revert "Update spec yaml files" This reverts commit 059f326432dba1f8da4ad4dfd8d5ac7a14de66c3. * Publish cloud spec file if necessary * Fix upload script * Move test files * Update docker compose file * Format code * Add comment about spec filename * Add unit tests * Remove redundant jdbc acceptance test When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog: - `wakeup_at` - `last_visited_at` - `last_comment_at` These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally. The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test. Given that the JDBC strict encrypt test is basically redundant, it is removed. * Update color palette, add scss colors, borders, and spacing (#13802) * Add updated color palette to SCSS and theme * Migrate current color palette to new colors * Add border and spacing variables * Remove legacy color mapping from scss side * Add style guide to front end repo (#13418) * Add .md from google doc draft * update per review * cleanup * Update STYLEGUIDE.md Co-authored-by: Tim Roes <tim@airbyte.io> * Edited Stripe, HubSpot, Intercom docs (#13810) * Initial edits * Updated the Stripe connector doc * Updated the Stripe doc * Updated HubSpot and Intercom docs * Minor sidebar edit * Remove the catalog from the sidebar since the file has been deleted in Update Airbyte Protocol Docs (#13709) (#13813) * Fix broken link (#13815) * fixing broken links * docusaurus updates * Add METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT to .env files to fix kube deployments (#13817) * octavia-cli: per workspace state (#13070) * Fix LabeledSwitch component width (#13798) * Fix LabeledSwitch component width * Switch to new color palette * Replace color * Fix color variable * Update 4-connection-checking.md (#13812) * Update 4-connection-checking.md As discussed in https://airbytehq-team.slack.com/archives/C02UQ9MJ4GG/p1655312684160609 - this example is misleading, as it does not actually connect to any API to validate connectivity. * Update 4-connection-checking.md Added a link to OneSignal check_connection, and move the "Note" to below the example * Bump Airbyte version from 0.39.18-alpha to 0.39.19-alpha (#13820) Co-authored-by: edgao <edgao@users.noreply.github.com> * Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) * Restart workflow after activity failure instead of quarantining (#13779) * use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries * retry workflow after delay instead of quarantining * allow activity env vars to be configured in docker and kube * add env var for workflow restart delay and refactor slightly * update tests to handle new restart behavior * update test name * add empty env var values to .env files * fail attempt before job in cleanJobState to prevent state machine failure * change default value of max activity attempt retries from 10 to 5 * Bump Airbyte version from 0.39.19-alpha to 0.39.20-alpha (#13829) Co-authored-by: lmossman <lmossman@users.noreply.github.com> * 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823) * Bump postgres version * Update doc * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * update readme (#13836) * add evan, jonathan, charles as protocol reviews in codeowners (#13776) * 📚 update MSSQL docs to reflect that we support normalization (#13826) * [low-code connectors] Cartesian product stream slicer (#13740) * list slicer * Add comment * product stream slicer * comment * rename * format * Update comment * split on 2 lines for readability * Revert "rename" This reverts commit e801f2d1f859306c6a2b0d66d265753ee9e32a54. * updated stacktrace format in java trace messages (#13847) * updated stacktrace format in java trace messages * test checks specifically on stacktrace in trace message * remove unused import * source-facebook-marketing: 🎉 Add fields to the ads_set stream (#13623) * 🎉 Source Github: use GraphQL for `pull_request_stats` stream (#13763) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Tweak schema validation code for readability. (#13721) * Refactor schema validation for better reading. * Better comments. * Format. * source-facebook-marketing: bump to 0.2.53 (#13853) * Source Hubspot - fix missing data issue (#13837) * #278 oncall: source hubspot - fix missing data issue * #278 oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Amplitude: try-catch BadZipFile exception (#13846) * #272 oncall: source Amplitude - try-catch BadZipFile exception * #272-oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🐛 Server can display streams with the same name in different namespaces (#13855) * Updated connector catalog (#13862) * Fix the running job counter (#13833) * Add StateType and StateWrapper objects to the model (#13858) * Emit orphan running jobs metrics (#13865) That metrics show the number of jobs currently flagged as running that are associated to inactive or deprecated connections. * Add meaningful tests for OTEL client (#13831) * Add meaningful tests for OTEL client * PR comment fix * code clean ups * Persist state to reset connection job config (#13867) * add current connection state to JobResetConnectionConfig * pass state from reset connection config to job sync config in generate input activity * format * Source Google Analytics: Add threshold_days option, uncomment statement for incremental tests (#13871) * Harshith/test pr 12536 (#13874) * 🎉 New Source: Firebolt (#1) * fix: Boolean type cast * test: Improve format testing and doc * refactor: Move some db functionality * docs: Adding types doc link in utils * feat: Use future-proof Auth in SDK * fix: integration tests are failing * chore: update seed file Co-authored-by: Petro Tiurin <93913847+ptiurin@users.noreply.github.com> Co-authored-by: ptiurin <petro.tiurin@firebolt.io> * :bug: Source Relational DB : remove unicode nulls from cursors (#13854) * Make sure that cursor value doesn't contain unicode null. * Test that cursors doesn't contain unicode null * incr version * enableReinitialize on account form (#13622) * #13888 source instagram: fix build (#13889) * Fix md and lg spacing scss vars (#13886) * Publishing java connectors with better formatted stacktrace in Trace messages (#13852) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Fix md and lg spacing scss vars (#13886) * Delete TEMP-testing-command.yml * auto-bump connector version * Delete publish-command.yml * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * Update _variables.scss Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * octavia-cli: fix workspace not having anonymous_data_collection property (#13869) * Update connection update calls to use central utility to ensure connection update has all data (#13564) * Update connection updates with build update utility * Add buildConnectionUpdate utility * Update components that update the connection to use utility when necessary * Use conection name when saving connection from replication view to prevent override from refreshed catalog * Improve connection check on ReplicationView onSubmit function * Display connection state in connection setting page (#13394) * Display Connection State in Setting page * memoize callback * rendering and confirmaton * setState API * Input validation * remove JSON step * rename apiMethod to `updateState` * test and adjust route * skip if sync is running * prevent state update when sync is running * code editor component * errors fixed * scss style * make linter happy * Back to monaco editor * Remove ability to edit state * Adjust FE code * Fix CSS problem * Update airbyte-webapp/src/locales/en.json Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * just use PRE to render state for now Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * update api for per stream (#13835) * Update airbyte-protocol.md (#13892) * Update airbyte-protocol.md * Fix typo * Fix prose * Add protocol reviewers for protocol documentation * Remove duplicate * Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) * deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) * Do not hide unexpected errors in the check connection (#13903) * Do not hide unexpected errors in the check connection * Fix test * Common code to deserialize a state message in the new format (#13772) * Common code to deserialize a state message in the new format * PR comments and type changed to typed * Format * Add StateType and StateWrapper objects to the model * Use state wrapper instead of Either * Switch to optional * PR comments * Support array legacy state * format Co-authored-by: Jimmy Ma <jimmy@airbyte.io> * 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633) * start and end date for finacial stream should not be more than 180 days apart * improve unit tests * make changes to start date for finance stream * update tests * lint changes * update version to 0.2.22 for source-amazon-seller-partner * Normalization: Fix incorrect jinja2 macro `json_extract_array` call (#13894) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Docs: fixed the broken links (#13915) * 0.2.5 -> 0.2.6 (#13924) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 13546 Fix integration tests source-postgres Mac OS (#13872) * 13546 Fix integration tests source-postgres Mac OS * 13548 Fixed integration tests source-tidb Mac OS (#13927) * Source MsSql : incr ver to include changes #13854 (#13887) * incr version * put PR id * docker ver * connectors that published (#13932) * Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) * Removed part_size from connectors that use StreamTransferManager * fixed S3DestinationConfigTest * fixed S3JsonlFormatConfigTest * upadate changelog and bump version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * upadate changelog and bump version for Redshift and Snowflake destinations * auto-bump connector version * fix GCS staging test * fix GCS staging test * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Reverted changes in SshBastionContainer (#13934) * 🎉 New Source Dockerhub (#13931) * init * implement working source + tests * add docs * add docs * fix bad comments * Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml * Update airbyte-integrations/connectors/source-dockerhub/Dockerfile * Update airbyte-integrations/connectors/source-dockerhub/.dockerignore * Apply suggestions from code review * Update docs/integrations/sources/dockerhub.md * Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py Co-authored-by: George Claireaux <george@airbyte.io> * address @Phlair's feedback * address @Phlair's feedback * each record is now a Docker image rather than response page * format * fix unit tests * fix acceptance tests * add icon, definition and generate seed spec * add requests to requirements Co-authored-by: sw-yx <shawnthe1@gmail.com> * commented out non-relevant tests (#13940) * Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938) Co-authored-by: alafanechere <alafanechere@users.noreply.github.com> * newaction (#13942) * remove test action (#13944) * 🎉Source-mysql: aligned datatype test (#13945) * [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing * #13958 Source Stripe: fix configured catalogs (#13959) * 🐛 Source: Typeform - Update schema for Responses stream (#13935) * Upd responses schema * Upd docs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * :window: Updated email invitation flow that enables invited users to set name and create password (#12788) * First pass accepting email link invitation * Update Auth service with signInWithEmailLink calls * Add AcceptEmailInvite component * Update FirebaseActionRoute to handle sign in mode * Rename ResetPasswordAction to FirebseActionRoute * Add create password setp to AcceptEmailInvite component * Remove continueURL from invite fetch * Update accept email invite for user to enter both email and password together * Set name during email link signup * Update AcceptEmailInvite to send name * Add updateName to UserService * Update AuthService to set name during sign up * Remove steps from AcceptEmailInvite component Remove setPassword from AuthService * Add header and title to accept invite page * Move invite error messages to en file * For invite link pages, show login link instead of sign up * Disable name update on sign in via email lnk * Resend email invite when the invite link is expired * Fix status message in accept email invite page * Re-enable set user's name during sign up email invite * Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name * Update comments on GoogleAuthService signInWithEmailLink * Add newsletter and accept terms checkboxes to accept email invite component * Extract signup form from signup page * Extract fields from signup form * Update accept email invite component to use field components from signup form * Ensure that sign up button is disable until form is valid and security checkbox is checked * Make error status text color in accept email link red * Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces * Add coment around continueUrl param usage in UserService * Remove usless default case in GoogleAuthService * Source Marketo: process fail during creation of an export job (#13930) * #9322 source Marketo: process fail during creation of an export job * #9322 source marketo: upd changelog * #9322 source marketo: fix unit test * #9322 source marketo: fix SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * :window: :wrench: Add eslint rules for CSS modules (#13952) * add eslint-plugin-css-modules rules * Fixes: - turn on eslint css modules rule as error - remove unused styles * add warning message if styled components is used * Revert "add warning message if styled components is used" This reverts commit 4e92b8b2110142bb679f15aeb034e377e0dcc69c. * replace rule severity with words * Update salesforce.md Fixed broken link * :window: 🔧 Add auto-fixable linting rules to webapp (#13462) * Add new eslint rules that fit with our code style and downgrade rules to warn * allowExpressions in fragment eslint rule * Enable function-component-definition in eslint and fix styles * Cleanup lint file * Fix react/function-component-definition warnings manually * Add more auto-fixable rules and fix * Fix functions that require usless returns * Update array-type rule to array-simple * Fix eslint errors manually disable assignmentExpression for arrays in prefer-destructuring rule * Auto fix new linting issues after rebase * Enhance /publish to allow for multiple connectors and parallel execution (#13864) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Delete TEMP-testing-command.yml * auto-bump connector version * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * destinations * + snowflake * saved * auto-bump connector version * auto-bump connector version * java source bumps * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * remove twice-defined methods * label things * revert action * using the new test action * point at action * wrong tag on action * update pool label * update to use new ec2-github-runner fork * this needs to be more generic than publisher * change publish to run on pool * add comment about runner-pool usage * updated publish command docs for multi & parallel connector runs * auto-bump connector version * auto-bump connector version * auto-bump connector version * unbump failed publish versions * missed dockerfiles * remove failed docs * mssql fix * overhauled the git comment output * bumping a test connector that should work * slight order switcheroo * output connectors properly in first message * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979) Co-authored-by: Phlair <Phlair@users.noreply.github.com> * Parker/temporal cloud (#13243) * switch to temporal cloud client for now * format * use client cert/key env secret instead of path to secret * add TODO comments * format * add logging to debug timeout issue * add more logging * change workflow task timeout * PR feedback: consolidate as much as possible, add missing javadoc * fix acceptance test, needs to specify localhost * add internal-use only comments * format * refactor to clean up TemporalClient and prepare it for future dependency injection framework * remove extraneous log statements * PR feedback * fix test * return isInitialized true in test * 📄 Postgres source: fix CDC setup order in docs (#13949) * postgres source: fix CDC setup order docs * Update docs/integrations/sources/postgres.md Co-authored-by: Liren Tu <tuliren@gmail.com> * Per-stream state support for Postgres source (#13609) * WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback * Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984) Co-authored-by: pmossman <pmossman@users.noreply.github.com> * Adds test for new workflow (#13986) * Adds test for new workflow * Adds airbyte repo * remove testing line * Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472) * write out new request options provider and refactor components and parts of the YAML config * fix formatting * pr feedback to consolidate body_data_provider to simplify the code * pr feedback get rid of extraneous optional * publish oss for cloud (#13978) workflow to publish oss artifacts that cloud needs to build against use docker buildx to create arm images for local development * skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) * 🎉 Source Github: break point added for workflows_runs stream (#13926) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866) * 6339: debug info * 6339: not using 'USE' on Azure SQL servers * 6339: cleanup * 6339: cleanup2 * 6339: cleanup3 * 6339: versions/changelogs updated * 6339: merge from master (consolidation issue) * 6339: dev connector version (for testing in airbyte cloud) * 6339: code review implementation * 6339: apply formatting * in case runners fail to spin up, this needs to run on github-hosted (#13996) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * :tada: New Source: Webflow (#13617) * Added webflow code * Updated readme * Updated README * Added webflow to source_definitions.yaml * Enhanced documentation for the Webflow source connector * Improved webflow source connector instructions * Moved Site ID to before API token in Spec.yaml (for presentation in the UI) * Addressed comments in PR. * Changes to address requests in PR review * Removed version from config * Minor udpate to spec.yaml for clarity * Updated to pass the accept-version as a constant rather than parameter * Updated check_connection to hit the collections API that requires both site id and the authentication token. * Fixed the test_check_connection to use the new check_connection function * Added a streams test for generate_streams * Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK * Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog * Fixed a typo in a comment * Renamed some classes to be more intuitive * Renamed class to be more intuitive * Minor change to an internal method name * Made _get_collection_name_to_id_dict staticmethod * Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests". This was caused by Mocked settings from test_source.py leaking into test_streams.py * format: add double quotes and remove unused import * readme: remove semantic version naming of connector in build commands * Updated spec.yaml * auto-bump connector version * format files * add changelog * update dockerfile * auto-bump connector version Co-authored-by: sajarin <sajarindider@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Source-oracle: fixed tests + checkstyle (#13997) * Source-oracle: fixed tests + checkstyle * 🐛Destination-mysql: fixed integration test and build process (#13302) * [13180] destination-mysql: fixed integration test * update changelog to include debezium version upgrade (#13844) * make table headers look less like successes (#13999) * source-twilio: implement lookback windows (#13896) * Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) This reverts commit aa28d448d820df9d79c2c0d06b38978d1108fb2c. * Revert "6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)" (#14011) This reverts commit 0d870bd37bc3b5cd798b92115d73bcc45a42d8f7. * [low-code connectors] BasicHttpAuthenticator (#13733) * implement basichttpauthenticator * add optional refresh access token authenticator * remove prints * type hints * Fix and unit test * missing test * Add class to __init__ file * Add comment * migrate JsonSchemas to use basic path instead of JSONPath (#13917) * scaffold for catalog diff, needs fixing on type handling and tests (#13786) * Prepare release of JDBC connectors (#13987) * Prepare release of JDBC connectors * Update source definitions manually * use built in check for if path is definite (#13834) * 13535 Fixed bastion network for integration tests (#14007) * doc: add error troubleshooting `docker-compose up` (#13765) * fix: duplicate resource allocations in `airbyte-temporal` deployment (#13816) * helm-chart: Fix worker deployment format error (#13839) * add catalog diff connection read (#13918) * doc: fix small typo on Shopify documentation (#13992) * add streams to reset to job info (#13919) * Generate api for changes in #13370 and make code compatible (#14014) * Generate api for per-stream updates #13835 (#14021) * Revert "Prepare release of JDBC connectors (#13987)" (#14029) This reverts commit df759b30778082508e2872513800fac34d98ff7c. * Fix per stream state protocol backward compatibility (#14032) * rename state type field to fix backwards compatibility issue * replace usages of stateType with type * support semi incremental by adding extractor record filter (#13520) * support semi incremental by adding extractor record filter * refactor extractor into a record_selector that supports extraction and filtering of response records * Remove pydantic spec from amazon ads and use YAML spec (#13988) * add EdDSA support in SSH tunnel (#9494) * add EdDSA support * verify EdDSA support works correct Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com> * 🎉New source connector: source-metabase (#13752) * Add docs * Close metabase session when sync finishes * Close session in check_connection * Add source definition to seed * Add icon * improve cdc check for connectors (#14005) * improve should use cdc check * Revert "improve should use cdc check" This reverts commit 7d01727279d21d33a6c18ed3227ee94432636120. * improve should use cdc check * add unit test * Update webflow.md * Update webflow.md * Update webflow.md * Remove legacy sentry code from cdk (#14016) * rip sentry out of cdk * remove sentry dsn from gsc * Update webflow.md * Update webflow.md * Fixed broken links (#14071) * 🪟Persist unsaved changes on schema refresh (#13895) * add form values tracker context * add clarifying comment * add same functionality to create connection * Update airbyte-webapp/src/components/CreateConnectionContent/CreateConnectionContent.tsx Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * Fixes broken links so we can deploy again (#14075) also adds better error message for when this happens to others * Adds symmary.md to gitignore (#14078) * Added webflow icon (#14069) * Added webflow icon * Added icon * Build create connection form build failure (#14081) * Fix CDK obfuscation of nested secrets (#14035) * Added Buy Credits section to Managing Airbyte Cloud (#13905) * Added Buy Credits section to Managing Airbyte Cloud * Made some style changes * Made edits based on Natalie's suggestions * Deleted link * Deleted line * Edited email address * Updated reaching out to sales sentence * disable es-lit to fix build (#14087) * Release source connectors (#14077) * Release source connectors * Fix issue with database connection in test * Fix failing tests due to authentication * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump Airbyte version from 0.39.23-alpha to 0.39.24-alpha (#14094) Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> * Emit the state to remove in the airbyte empty source (#13725) What This PR updates the EmptyAirbyteSource in order to perform a partial update and handle the new state message format. How The empty will now emit different messages based on the type of state being provided: Per stream: it will emit one message per stream that have been reset Global: It will emit one global message that will contain null for the stream that have been reset including the shared state Co-authored-by: Jimmy Ma <jimmy@airbyte.io> * Add StatePersistence object (#13900) Add a StatePersistence object that supports Read/Writes of States to the DB with StreamDescriptor fields The only migrations that is supported are * moving from LEGACY to GLOBAL * moving from LEGACY to STREAM * All other state type migrations are expected to go through an explicit reset beforehand. * secret-persistence: Hashicorp Vault Secret Store (#13616) Co-authored-by: Amanda Murphy <amanda.murphy@heapanalytics.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> * 🐛 Source Hubspot: remove `AirbyteSentry` dependency (#14102) * fixed * updated changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * fix: format VaultSecretPersistenceTest.java (#14110) * Source Hubspot: extend error logging (#14054) * #291 incall - source Hubspot: extend error logging * huspot: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Update webflow.md (#14083) * Update webflow.md Removed a description that is only applicable to people that are writing connector code, not to _users_ of the connector. * Update webflow.md * Update webflow.md * Update webflow.md * Update webflow.md * 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * 12708: specs updated * 12708: specs updated * 12708: removing autogenerated files from PR * 12708: changelog updated * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source PayPal Transaction: Update Transaction Schema (#13682) * Update transaction schema. * Transform money values from strings to floats or integers. Co-authored-by: nataly <nataly@airbyte.io> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * fix(jsonSchemas): raise error when items property not provided (#14018) * fix stream name in stream transformation update (#14044) * 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery. * Remove skiptests option (#14100) * update sentry release script (#14123) * Remove "additionalProperties": false from specs for connectors with staging (#14114) * Remove "additionalProperties": false from spec for connectors with staging * Remove "additionalProperties": false from spec for Redshift destination * bump versions * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * [14003] source-oracle: added custom jdbc field (#14092) * [14003] source-oracle: added custom jdbc field * Add JobErrorReporter for sending sync job connector failures to Sentry (#13899) * skeleton for reporting connector errors to sentry * report on job failures instead of attempt failures * report sync job failures with relevant metadata using JobErrorReporter * send stack traces from python connectors to sentry * test JobCreationAndStatusUpdate and JobErrorReporter * logs * refactor into helper, initial tests * using sentry * run format * load reporting client from env * load sentry dsn from env * send java stack traces to sentry * test sentryclient, refactor to use Hub instance * ErrorReportingClient.report -> .reportJobFailureReason * inject exception helper, test stack trace parse error tagging * rm logs * more stack trace tests * remove logs * fix failing tests * rename ErrorReportingClient to JobErrorReportingClient * rename vars in docker-compose * Return an Optional instead of null when parsing stack traces * dont remove airbyte prefix when setting release name * from_trace_message static * remove failureSummary from jobfailure input, get from Job * send stacktrace string if we weren't able to parse * set deployment mode tag * update .env * just log if something goes wrong * Use StateMessageHelper in source (#14125) * Use StateMessageHelper in source * PR feedback and formatting * More PR feedback * Revert change * Revert changes * Bump Airbyte version from 0.39.24-alpha to 0.39.25-alpha (#14124) Co-authored-by: brianjlai <brianjlai@users.noreply.github.com> * Refactor acceptance tests and utils (#13950) * Refactor Basic acceptance tests and utils * Refactor Advanced acceptance tests and utils * Remove unused code * Clear destination db data during cleanup * Cleanup comments * cleanup init code * test creating new desintation db for each test * cleanup desintation db init * Allow to edit api client * pull in temporal cloud changes * Rename helper to harness; set some funcs to private; turn init into constructor * add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests * update javadoc Co-authored-by: Davin Chia <davinchia@gmail.com> * fix javadoc formatting * fix var naming Co-authored-by: Davin Chia <davinchia@gmail.com> * Bump Airbyte version from 0.39.25-alpha to 0.39.26-alpha (#14141) Co-authored-by: terencecho <terencecho@users.noreply.github.com> * 🎉 octavia-cli: Add ability to get existing resources (#13254) * 13541 Fixed integration tests source-db2 Mac OS (#14133) * 13523 Fix integration tests destination-cassandra Mac OS (#14134) * 🐛 Source Hubspot: fixed SAT test, commented out expected_records (#14140) * :bug: Source Intercom: extend `Contacts` schema with new properties (#14099) * Source Twilio: adopt best practices (#14000) * #1946 Source twilio: aopt best practices - tune tests * #1946 add expected_records to acceptance-test-config.yml * #1946 source twilio - upd schema and changelog * #1946 fix expected_records * #1946 source twilio: rm alerts from expected records as they expire in 30 days * #1946 source twilio: bump version * 🎉 Source BingAds: expose hourly/daily/weekly/monthly options from configuration (#13801) * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: config settings for hourly/daily/weekly/monthly reports added: default value for all periodic reports to True * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused class variables, if-statement * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused variables from config * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * remove VersionMismatchServer (#14076) * remove VersionMismatchServer * remove VersionMismatchServerTest * revert intended changes * Increase instance termination time limit to 3 hours to accommodate connector builds. (#14181) * Use correct bash comment symbol. (#14183) * 🎉 New Source: Orbit.love (#13390) * source-orbit: add definition and specs (#14189) * 🎉 Base Norrmalization: clean-up Redshift `tmp_schemas` after SAT (#14015) Now after `base-normalization` SAT the Destination Redshift will be automatically cleaned up from test leftovers. Other destinations are not covered yet. * Source Salesforce: fix customIntegrationTest for SAT (#14172) * Source Amazon Ads: increase timeout for SAT (#14167) * 🎉 Introduce Google Analytics Data API source (#12701) * Introduce Google Analytics Data API source https://developers.google.com/analytics/devguides/reporting/data/v1 * Add Google Analytics Data API source PR link * Add `client` class for Google Analytics Data API * Move dimensions and metrics extraction to the `client` class In the Google Analytics Data API * Change the copyright date to 2022 in Google Analytics Data API * fix: removing incremental syncs * fix: change project_id to string * fix: flake check is failing * chore: added it to source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) * Publish acceptance test utils maven artifact (#14142) * Fix StatePersistence Legacy read/write (#14129) StatePersistence will wrap/unwrap legacy state on write/read to ensure compatibility with the old behavior/data. * 🎉 Destination connectors: Improved "SecondSync" checks in Standard Destination Acceptance tests (#14184) * [11731] Improved "SecondSync" checks in Standard Destination Acceptance tests * 🐛 Source Zendesk Support: fixed "Retry-After" non integer value (#14112) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Source Tiktok Marketing: Videometrics (#13650) * added video metrics in streams.py * common metrics list updated. * updated streams.py with extended metrics required. * updated stream_test * updated configured_catalog * video metrics required list updated. * chore: formatting * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * 🎉 Source Github: secondary rate limits has to retry (#13955) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Harshith/test pr 13118 (#14192) * Firebolt destination * feat: Write method dropdown * feat: Use future-proof Auth in SDK * refactor: Move writer instantiation * fix: tests are failing * fix: tests are failing * fix: tests are failing * chore: added connector to definitions * fix: formatting and spec * fix: formatting for orbit Co-authored-by: ptiurin <petro.tiurin@firebolt.io> * 🪟 :art: Show credit usage on chart's specific day (#13503) * add tooltip to chart * Fixes: - update main chart color; - change onHover background color * change chart color pallet to grey 500 * update color reference * remove opacity from UsageCell * 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) * Improve documentation for Postgres Source (#13830) * Improve documentation for Postgres Source * add information about additional JDBC params * add anchors for doc sections * fix link to CDC on Bare Metal * add more details about parsing date/time values * add doc link to SSH fields * Handle null reset source config (#14202) * handle null reset source config * format * Wait indefinitely if connection is not active (#14200) * also wait indefinitely if connection is deleted * fix test * Bump Airbyte version from 0.39.26-alpha to 0.39.27-alpha (#14204) Co-authored-by: lmossman <lmossman@users.noreply.github.com> * Bmoric/feature flag for state deserialization (#14127) * Add Feature flag * Add default feature flag value * Update test * remove unsused * tmp * Update tests * rm unwanted change * PR comments * [low-code connectors] default types and default values (#14004) * default types and default values * cleanup * fixes so read works * remove prints and trycatch * comment * remove unused param * split file * extract method * extract methods * comment * optional * fix test * cleanup * delete interpolated request header provider * simplify next page url paginator interface * comment * format * add state type endpoint (#14111) * Bump Airbyte version from 0.39.27-alpha to 0.39.28-alpha (#14210) Co-authored-by: sherifnada <sherifnada@users.noreply.github.com> * 🐛 source-orbit: remove workspace_old.json (#14208) * Fix: Docs plural login redirecting to wrong URL (#14207) * [docs] fix numbering and incorrect filename in CDK docs (#13045) * [docs] fix numbering in CDK docs * Update 5-declare-schema.md * Update 5-declare-schema.md * Update 6-read-data.md * Update 8-test-your-connector.md * Remove the old scheduler from HelmCharts helper (#14187) * Remove the old scheduler from HelmCharts helper The old scheduler was removed as part of https://github.com/airbytehq/airbyte/pull/13400 * Remove legacy `scheduler` comment in HelmCharts * Source Gitlab: add GroupIssueBoards stream (#13252) * GitLab Source: add GroupIssueBoards stream * Address stream schema comments * Address comments * Bump version * Add as empty stream * run seed file source (#14215) * fix 'cannot reach server' error on demo instance (#10020) * Update CODEOWNERS (#14209) * 🎉 Source Github: use GraphQL for `reviews` stream (#13989) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * workflow for publishing artifacts for cloud (#14199) * fix sentry org slug change (#14218) * Source File: correct spec json to match json format (#13738) * Upgrade spotless version and remove jvmargs workaround (#13705) * Source Zendesk Chat: Process large amount of data in batches for incremental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Remove all @ts-ignore (#14221) * Bump hadoop to use version 3.3.3 (#14182) * Change the persistence activity to use the new persistence layer (#14205) * Change the persistence activity to use the new persistence layer * Use lombok * format * Use new State message helper * Fix build (#14225) * Fix build * Fix test * Use new state persistence for state reads (#14126) * Inject StatePersistence into DefaultJobCreator * Read the state from StatePersistence instead of ConfigRepository * Add a conversion helper to convert StateWrapper to State * Remove unused ConfigRepository.getConnectionState * Temporal per stream resets (#13990) * remove reset flags from workflow state + refactor * bring back cancelledForReset, since we need to distinguish between that case and a normal cancel * delete reset job streams on cancel or success * extract isResetJob to method * merge with master * set sync modes on streams in reset job correctly * format * Add test for getAllStreamsForConnection * fix tests * update more tests * add StreamResetActivityTests * fix tests for default job creator * remove outdated comment * remove debug lines * remove unused enum value * fix tests * fix constant equals ordering * make job mock not static * DRY and add comments * add comment about deleted streams * Remove io.airbyte.config.StreamDescriptor * regisster stream reset activity impl * refetch connection workflow when checking job id, since it may have been restarted * only cancel if workflow is running, to allow reset signal to always succeed even if batched with a workflow start * fix reset signal to use new doneWaiting workflow state prop * try to fix tests * fix reset cancel case * add acceptance test for resetting while sync is running * format * fix new acceptance test * lower sleep on test * raise sleep * increase sleep and timeout, and remove repeated test * use CatalogHelpers to extract stream descriptors * raise sleep and timeout to prevent transient failures * format Co-authored-by: alovew <anne@airbyte.io> Co-authored-by: Luis Gomez <781929+lgomezm@users.noreply.github.com> Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Topher Lubaway <asimplechris@gmail.com> Co-authored-by: Malik Diarra <malik@airbyte.io> Co-authored-by: Teal Larson <LARSON.TEAL@GMAIL.COM> Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: VitaliiMaltsev <39538064+VitaliiMaltsev@users.noreply.github.com> Co-authored-by: grishick <greg@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: Andy <andy@airbyte.io> Co-authored-by: LiRen Tu <tuliren.git@outlook.com> Co-authored-by: oneshcheret <33333155+sashaNeshcheret@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Xiaohan Song <xiaohan@airbyte.io> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Charles <charles@airbyte.io> Co-authored-by: Sophia Wiley <106352739+sophia-wiley@users.noreply.github.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: Denis Davydov <davydov.den18@gmail.com> Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com> Co-authored-by: Bastien Gandouet <bastien@mozaiqu.es> Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> Co-authored-by: Paweł Kociński <pawel93kocinski@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: Alexander Marquardt <alexander.marquardt@gmail.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: lmossman <lmossman@users.noreply.github.com> Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: mutt-santiago-stachuk <106173005+mutt-santiago-stachuk@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: Petro Tiurin <93913847+ptiurin@users.noreply.github.com> Co-authored-by: ptiurin <petro.tiurin@firebolt.io> Co-authored-by: Andrii Leonets <30464745+DoNotPanicUA@users.noreply.github.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Jimmy Ma <jimmy@airbyte.io> Co-authored-by: Ganpat Agarwal <gagarwal@artica.com> Co-authored-by: Rajakavitha Kodhandapani <krajakavitha@gmail.com> Co-authored-by: Yevhen Sukhomud <suhomud@gmail.com> Co-authored-by: sw-yx <shawnthe1@gmail.com> Co-authored-by: alafanechere <alafanechere@users.noreply.github.com> Co-authored-by: Eugene <etsybaev@gmail.com> Co-authored-by: Anna Lvova <37615075+annalvova05@users.noreply.github.com> Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> Co-authored-by: Phlair <Phlair@users.noreply.github.com> Co-authored-by: Parker Mossman <parker@airbyte.io> Co-authored-by: Adam <adam-bloom@users.noreply.github.com> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: pmossman <pmossman@users.noreply.github.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> Co-authored-by: Peter Hu <peter@airbyte.io> Co-authored-by: Tuhai Maksym <kimerinn@gmail.com> Co-authored-by: sajarin <sajarindider@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: steve withington <steve@digitalmine.com> Co-authored-by: Leo Sussan <leosussan@gmail.com> Co-authored-by: cenegd <cenegd@live.com> Co-authored-by: Tomas Perez Alvarez <72174660+Tomperez98@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: Stella Chung <schung507@gmail.com> Co-authored-by: Amanda Murphy <amanda.murphy@heapanalytics.com> Co-authored-by: Mohamed Magdy <mohamed.magdy@canary.is> Co-authored-by: nataly <nataly@airbyte.io> Co-authored-by: Tyler Russell <tylerrussell85@gmail.com> Co-authored-by: Alexander Tsukanov <alexander.tsukanovvv@gmail.com> Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> Co-authored-by: brianjlai <brianjlai@users.noreply.github.com> Co-authored-by: terencecho <terence@airbyte.io> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Daniel Diamond <33811744+danieldiamond@users.noreply.github.com> Co-authored-by: drrest <dr.rest@gmail.com> Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io> Co-authored-by: Zawar Khan <zawar.khan@getmercury.io> Co-authored-by: sherifnada <sherifnada@users.noreply.github.com> Co-authored-by: Sachin Jangid <sachinjangid832@gmail.com> Co-authored-by: Chris Wu <chris@faros.ai> Co-authored-by: Jared Rhizor <me@jaredrhizor.com> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com> Co-authored-by: alovew <anne@airbyte.io>
pdulapalli
added a commit
to falkonai/airbyte
that referenced
this pull request
Jun 29, 2022
* Adds zombie removal tool (#13718) * Adds zombie removal tool * Corrects endpoint adds comments * Adds API links * Changes search logic, fixes escape character * Corrects help text * Fix connection read object building (#13568) Make sure the sourceCatalogId field is set with the value stored in the database. * fix modal overlay problem (#13724) * Improve Gradle build and fix storybook (#13719) * Improve Gradle build and fix storybook * Run all copy tasks after copyDocker * fix stream descriptor typo (#13726) * Save streams to reset in job config when creating reset job (#13703) * save streams to reset in job config when creating reset job * change streamDescriptors to streamsToReset * :bug: Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655) * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * Postgres Source: fixed unsupposted date-time datatypes during incremental sync * updated CHANGELOG * add tests for incremental cursor check * removed star import * add timestamp datatype test * Bump version in Dockerfile * auto-bump connector version Co-authored-by: grishick <greg@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump version in Dockerfile to match source-postgres version (#13732) * Fail the connection state instead of throwing an exception (#13728) * Fail the connection state instead of throwin exception * Fix test * rm import * Update airbyte-workers/src/main/java/io/airbyte/workers/general/DefaultCheckConnectionWorker.java Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Evan Tahler <evan@airbyte.io> * [low-code connectors] add a few unit tests (#13666) * add a couple of unit tests * refactor tests * 🎉 Destination databricks: rename to databricks lakehouse (#13722) * Update Databricks naming * Update destination_spec * Update BOOTSTRAP.md * Update Dockerfile * Update README.md * Update spec.json * Update databricks.md * Update databricks.md * Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md Co-authored-by: LiRen Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren.git@outlook.com> * Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745) * Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump source-postgres-strict-encrypt version to 0.4.23 to match source-postgres version (#13747) * S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483) * S3 destination: Updating processing data types for Avro/Parquet formats * S3 destination: handle comparing data types * S3 destination: clean code * S3 destination: clean code * S3 destination: handle case with unexpected json schema type * S3 destination: clean code * S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class * S3 destination: clean code * S3 destination: clean code * GCS destination: Update data types processing for Avro/Parquet formats * GCS destination: clean redundant code * S3 destination: handle case with numbers inside array * S3 destination: clean code * S3 destination: add unit test * S3 destination: update unit test cases with number types. * S3 destination: update unit tests. * S3 destination: bump version for s3 and gcs * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Salesforce: fix sync capped streams with more records than page size (#13658) * change logic for counting records * update doc * correct unit test * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Docs: update normalization doc (#13739) * update dbt version in custom normalization (#13742) * Kube fix and Docs for open telemetry integration (#13701) * Create interface, factory for metric client * remove unused func * change count val to use long * PR fix * otel metric client implementation * merge conflicts resolve * build fix * add a test, moved version into deps catalog * fix test * add docs for open telemetry * fix kube setting for otel, and add doc * helm related fields update for opentel * cdk: fix typo in build.gradle.hbs (#13761) * Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha (#13759) Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> * Update destinations to handle new state messages (#13670) * 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Postgres source strict encrypt trucnated precision bump version (#13769) * Postgres Source: fixed truncated precision if the value of the millisecond or second is 0 * check CI with 1.15.3 testcontainer * check CI with 1.15.3 testcontainer * returned latest version of testcontainer * fixed checkstyle * fixed checkstyle * returned latest testcontainer version * updated CHANGELOG * bump version * auto-bump connector version * Postgres source strict encrypt bump version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Migrate StateDB to support per stream states (#13731) * Update StateDB to support per Stream states. * Add `StateType` type * Add `steam_name`, `namespace` and `type` to `state` table. * Set the default StateType to LEGACY * Update getting-started-with-airbyte-cloud.md (#13741) * Update getting-started-with-airbyte-cloud.md Edited Getting Started with Airbyte Cloud guide to match the updated Cloud UI. * Update getting-started-with-airbyte-cloud.md Updated based on Amruta's suggestions. * 🐛 Normalization correctly propagates deletions to the final tables (#12846) * Update Airbyte Protocol Docs (#13709) * Source Facebook marketing: fix `not syncing any data` issue (#13749) * #253 oncall source fb marketing: fix issue * #253 oncall: upd changelog * Update docs/integrations/sources/facebook-marketing.md * auto-bump connector version Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Ignore custom connectors during connectors update (#13581) * Ignore Custom connectors during connectors update * Skip any connector flagged as custom as well * Format * [low-code connectors] Support for list-based stream slicers (#13668) * list slicer * Add comment * test * ast eval * refactor * fix * remove print * rename * return a generator * smaller fix and reset to master * union type hint * assert * comment * format * 📝 Postgres source: add troubleshooting for postgres hot standby (#13784) * Add troubleshooting section in postgres doc * Update doc * Add one more solution * upgrade debezium version for postgres to 1.9.2 (#13368) * upgrade debezium version for postgres to 1.9.2 * fix test+build * fix build * address review comments * update link to docs * fix test * 🐛 Source Amplitude: Fix an infinite loop (#13638) * Fix infinite loop when fetching Amplitude data * Update changelog and version * Address review * fix: unit tests were failing * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * octavia-cli: allow user to set custom API HTTP headers(#12893) * 🎉 Source Github: Fix API sorting, fix `get_starting_point` caching (#13707) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 🐛 Source Zendesk Support: fixed 429 error for `TicketMetrics` stream (#13757) * Add some dev-facing normalization docs (#13780) * Fix mysql cdc acceptance test (#13795) * fix mysql cdc acceptance test * remove comments * doc: add link to forum in plopfile (#13560) * Make connector adaptable based on deployment mode (#13522) * Add deployment mode to env shared with jobs * Add adaptive runners * Migrate postgres source to use adaptive runner * Add an array of specs in docker image spec definition * Add copyright * Parse docker image spec with specs list * Update spec yaml files * Pass in DEPLOYMENT_MODE to docker compose file * Revert "Parse docker image spec with specs list" This reverts commit 8fe41dd3b7fa5306ba8b62d660c002420adb9b23. * Revert changes in docker image spec * Read cloud specific spec files based on deployment mode * Revert "Update spec yaml files" This reverts commit 059f326432dba1f8da4ad4dfd8d5ac7a14de66c3. * Publish cloud spec file if necessary * Fix upload script * Move test files * Update docker compose file * Format code * Add comment about spec filename * Add unit tests * Remove redundant jdbc acceptance test When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog: - `wakeup_at` - `last_visited_at` - `last_comment_at` These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally. The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test. Given that the JDBC strict encrypt test is basically redundant, it is removed. * Update color palette, add scss colors, borders, and spacing (#13802) * Add updated color palette to SCSS and theme * Migrate current color palette to new colors * Add border and spacing variables * Remove legacy color mapping from scss side * Add style guide to front end repo (#13418) * Add .md from google doc draft * update per review * cleanup * Update STYLEGUIDE.md Co-authored-by: Tim Roes <tim@airbyte.io> * Edited Stripe, HubSpot, Intercom docs (#13810) * Initial edits * Updated the Stripe connector doc * Updated the Stripe doc * Updated HubSpot and Intercom docs * Minor sidebar edit * Remove the catalog from the sidebar since the file has been deleted in Update Airbyte Protocol Docs (#13709) (#13813) * Fix broken link (#13815) * fixing broken links * docusaurus updates * Add METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT to .env files to fix kube deployments (#13817) * octavia-cli: per workspace state (#13070) * Fix LabeledSwitch component width (#13798) * Fix LabeledSwitch component width * Switch to new color palette * Replace color * Fix color variable * Update 4-connection-checking.md (#13812) * Update 4-connection-checking.md As discussed in https://airbytehq-team.slack.com/archives/C02UQ9MJ4GG/p1655312684160609 - this example is misleading, as it does not actually connect to any API to validate connectivity. * Update 4-connection-checking.md Added a link to OneSignal check_connection, and move the "Note" to below the example * Bump Airbyte version from 0.39.18-alpha to 0.39.19-alpha (#13820) Co-authored-by: edgao <edgao@users.noreply.github.com> * Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) * Restart workflow after activity failure instead of quarantining (#13779) * use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries * retry workflow after delay instead of quarantining * allow activity env vars to be configured in docker and kube * add env var for workflow restart delay and refactor slightly * update tests to handle new restart behavior * update test name * add empty env var values to .env files * fail attempt before job in cleanJobState to prevent state machine failure * change default value of max activity attempt retries from 10 to 5 * Bump Airbyte version from 0.39.19-alpha to 0.39.20-alpha (#13829) Co-authored-by: lmossman <lmossman@users.noreply.github.com> * 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823) * Bump postgres version * Update doc * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * update readme (#13836) * add evan, jonathan, charles as protocol reviews in codeowners (#13776) * 📚 update MSSQL docs to reflect that we support normalization (#13826) * [low-code connectors] Cartesian product stream slicer (#13740) * list slicer * Add comment * product stream slicer * comment * rename * format * Update comment * split on 2 lines for readability * Revert "rename" This reverts commit e801f2d1f859306c6a2b0d66d265753ee9e32a54. * updated stacktrace format in java trace messages (#13847) * updated stacktrace format in java trace messages * test checks specifically on stacktrace in trace message * remove unused import * source-facebook-marketing: 🎉 Add fields to the ads_set stream (#13623) * 🎉 Source Github: use GraphQL for `pull_request_stats` stream (#13763) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Tweak schema validation code for readability. (#13721) * Refactor schema validation for better reading. * Better comments. * Format. * source-facebook-marketing: bump to 0.2.53 (#13853) * Source Hubspot - fix missing data issue (#13837) * #278 oncall: source hubspot - fix missing data issue * #278 oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Amplitude: try-catch BadZipFile exception (#13846) * #272 oncall: source Amplitude - try-catch BadZipFile exception * #272-oncall: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🐛 Server can display streams with the same name in different namespaces (#13855) * Updated connector catalog (#13862) * Fix the running job counter (#13833) * Add StateType and StateWrapper objects to the model (#13858) * Emit orphan running jobs metrics (#13865) That metrics show the number of jobs currently flagged as running that are associated to inactive or deprecated connections. * Add meaningful tests for OTEL client (#13831) * Add meaningful tests for OTEL client * PR comment fix * code clean ups * Persist state to reset connection job config (#13867) * add current connection state to JobResetConnectionConfig * pass state from reset connection config to job sync config in generate input activity * format * Source Google Analytics: Add threshold_days option, uncomment statement for incremental tests (#13871) * Harshith/test pr 12536 (#13874) * 🎉 New Source: Firebolt (#1) * fix: Boolean type cast * test: Improve format testing and doc * refactor: Move some db functionality * docs: Adding types doc link in utils * feat: Use future-proof Auth in SDK * fix: integration tests are failing * chore: update seed file Co-authored-by: Petro Tiurin <93913847+ptiurin@users.noreply.github.com> Co-authored-by: ptiurin <petro.tiurin@firebolt.io> * :bug: Source Relational DB : remove unicode nulls from cursors (#13854) * Make sure that cursor value doesn't contain unicode null. * Test that cursors doesn't contain unicode null * incr version * enableReinitialize on account form (#13622) * #13888 source instagram: fix build (#13889) * Fix md and lg spacing scss vars (#13886) * Publishing java connectors with better formatted stacktrace in Trace messages (#13852) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Fix md and lg spacing scss vars (#13886) * Delete TEMP-testing-command.yml * auto-bump connector version * Delete publish-command.yml * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * Update _variables.scss Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * octavia-cli: fix workspace not having anonymous_data_collection property (#13869) * Update connection update calls to use central utility to ensure connection update has all data (#13564) * Update connection updates with build update utility * Add buildConnectionUpdate utility * Update components that update the connection to use utility when necessary * Use conection name when saving connection from replication view to prevent override from refreshed catalog * Improve connection check on ReplicationView onSubmit function * Display connection state in connection setting page (#13394) * Display Connection State in Setting page * memoize callback * rendering and confirmaton * setState API * Input validation * remove JSON step * rename apiMethod to `updateState` * test and adjust route * skip if sync is running * prevent state update when sync is running * code editor component * errors fixed * scss style * make linter happy * Back to monaco editor * Remove ability to edit state * Adjust FE code * Fix CSS problem * Update airbyte-webapp/src/locales/en.json Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * just use PRE to render state for now Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * update api for per stream (#13835) * Update airbyte-protocol.md (#13892) * Update airbyte-protocol.md * Fix typo * Fix prose * Add protocol reviewers for protocol documentation * Remove duplicate * Edited Amplitude, Mailchimp, and Zendesk Support docs (#13897) * deleting SUMMARY.md since we don't need it for docusaurus builds (#13901) * Do not hide unexpected errors in the check connection (#13903) * Do not hide unexpected errors in the check connection * Fix test * Common code to deserialize a state message in the new format (#13772) * Common code to deserialize a state message in the new format * PR comments and type changed to typed * Format * Add StateType and StateWrapper objects to the model * Use state wrapper instead of Either * Switch to optional * PR comments * Support array legacy state * format Co-authored-by: Jimmy Ma <jimmy@airbyte.io> * 🐛 Source Amazon Seller Partner: handle start date for financial stream (#13633) * start and end date for finacial stream should not be more than 180 days apart * improve unit tests * make changes to start date for finance stream * update tests * lint changes * update version to 0.2.22 for source-amazon-seller-partner * Normalization: Fix incorrect jinja2 macro `json_extract_array` call (#13894) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Docs: fixed the broken links (#13915) * 0.2.5 -> 0.2.6 (#13924) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 13546 Fix integration tests source-postgres Mac OS (#13872) * 13546 Fix integration tests source-postgres Mac OS * 13548 Fixed integration tests source-tidb Mac OS (#13927) * Source MsSql : incr ver to include changes #13854 (#13887) * incr version * put PR id * docker ver * connectors that published (#13932) * Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) * Removed part_size from connectors that use StreamTransferManager * fixed S3DestinationConfigTest * fixed S3JsonlFormatConfigTest * upadate changelog and bump version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * upadate changelog and bump version for Redshift and Snowflake destinations * auto-bump connector version * fix GCS staging test * fix GCS staging test * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Reverted changes in SshBastionContainer (#13934) * 🎉 New Source Dockerhub (#13931) * init * implement working source + tests * add docs * add docs * fix bad comments * Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml * Update airbyte-integrations/connectors/source-dockerhub/Dockerfile * Update airbyte-integrations/connectors/source-dockerhub/.dockerignore * Apply suggestions from code review * Update docs/integrations/sources/dockerhub.md * Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py Co-authored-by: George Claireaux <george@airbyte.io> * address @Phlair's feedback * address @Phlair's feedback * each record is now a Docker image rather than response page * format * fix unit tests * fix acceptance tests * add icon, definition and generate seed spec * add requests to requirements Co-authored-by: sw-yx <shawnthe1@gmail.com> * commented out non-relevant tests (#13940) * Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938) Co-authored-by: alafanechere <alafanechere@users.noreply.github.com> * newaction (#13942) * remove test action (#13944) * 🎉Source-mysql: aligned datatype test (#13945) * [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing * #13958 Source Stripe: fix configured catalogs (#13959) * 🐛 Source: Typeform - Update schema for Responses stream (#13935) * Upd responses schema * Upd docs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * :window: Updated email invitation flow that enables invited users to set name and create password (#12788) * First pass accepting email link invitation * Update Auth service with signInWithEmailLink calls * Add AcceptEmailInvite component * Update FirebaseActionRoute to handle sign in mode * Rename ResetPasswordAction to FirebseActionRoute * Add create password setp to AcceptEmailInvite component * Remove continueURL from invite fetch * Update accept email invite for user to enter both email and password together * Set name during email link signup * Update AcceptEmailInvite to send name * Add updateName to UserService * Update AuthService to set name during sign up * Remove steps from AcceptEmailInvite component Remove setPassword from AuthService * Add header and title to accept invite page * Move invite error messages to en file * For invite link pages, show login link instead of sign up * Disable name update on sign in via email lnk * Resend email invite when the invite link is expired * Fix status message in accept email invite page * Re-enable set user's name during sign up email invite * Update signUpWithEmailLink so that sign up is successful even if we fail to update the user's name * Update comments on GoogleAuthService signInWithEmailLink * Add newsletter and accept terms checkboxes to accept email invite component * Extract signup form from signup page * Extract fields from signup form * Update accept email invite component to use field components from signup form * Ensure that sign up button is disable until form is valid and security checkbox is checked * Make error status text color in accept email link red * Update workspace check in DefaultView so that user lands in workspace selector when there are no workspaces * Add coment around continueUrl param usage in UserService * Remove usless default case in GoogleAuthService * Source Marketo: process fail during creation of an export job (#13930) * #9322 source Marketo: process fail during creation of an export job * #9322 source marketo: upd changelog * #9322 source marketo: fix unit test * #9322 source marketo: fix SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * :window: :wrench: Add eslint rules for CSS modules (#13952) * add eslint-plugin-css-modules rules * Fixes: - turn on eslint css modules rule as error - remove unused styles * add warning message if styled components is used * Revert "add warning message if styled components is used" This reverts commit 4e92b8b2110142bb679f15aeb034e377e0dcc69c. * replace rule severity with words * Update salesforce.md Fixed broken link * :window: 🔧 Add auto-fixable linting rules to webapp (#13462) * Add new eslint rules that fit with our code style and downgrade rules to warn * allowExpressions in fragment eslint rule * Enable function-component-definition in eslint and fix styles * Cleanup lint file * Fix react/function-component-definition warnings manually * Add more auto-fixable rules and fix * Fix functions that require usless returns * Update array-type rule to array-simple * Fix eslint errors manually disable assignmentExpression for arrays in prefer-destructuring rule * Auto fix new linting issues after rebase * Enhance /publish to allow for multiple connectors and parallel execution (#13864) * start * revert * azblob * bq * bq denorm * megapublish baaaabyyyy * fix needs * matrix connectors * auto-bump connector version * dont failfast and max parallel 5 * multi runno * minor * testing matrix agents * name * testing multi agents * tmp fix * new multi agents * multi test * tryy * let's do this * magico * fix * label test * couple more connector bumps * temp * things * check this * lets gooo * more connectors * Delete TEMP-testing-command.yml * auto-bump connector version * added comment describing bash part * running single thread * catch sentry cli * auto-bump connector version * destinations * + snowflake * saved * auto-bump connector version * auto-bump connector version * java source bumps * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * remove twice-defined methods * label things * revert action * using the new test action * point at action * wrong tag on action * update pool label * update to use new ec2-github-runner fork * this needs to be more generic than publisher * change publish to run on pool * add comment about runner-pool usage * updated publish command docs for multi & parallel connector runs * auto-bump connector version * auto-bump connector version * auto-bump connector version * unbump failed publish versions * missed dockerfiles * remove failed docs * mssql fix * overhauled the git comment output * bumping a test connector that should work * slight order switcheroo * output connectors properly in first message * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979) Co-authored-by: Phlair <Phlair@users.noreply.github.com> * Parker/temporal cloud (#13243) * switch to temporal cloud client for now * format * use client cert/key env secret instead of path to secret * add TODO comments * format * add logging to debug timeout issue * add more logging * change workflow task timeout * PR feedback: consolidate as much as possible, add missing javadoc * fix acceptance test, needs to specify localhost * add internal-use only comments * format * refactor to clean up TemporalClient and prepare it for future dependency injection framework * remove extraneous log statements * PR feedback * fix test * return isInitialized true in test * 📄 Postgres source: fix CDC setup order in docs (#13949) * postgres source: fix CDC setup order docs * Update docs/integrations/sources/postgres.md Co-authored-by: Liren Tu <tuliren@gmail.com> * Per-stream state support for Postgres source (#13609) * WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback * Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984) Co-authored-by: pmossman <pmossman@users.noreply.github.com> * Adds test for new workflow (#13986) * Adds test for new workflow * Adds airbyte repo * remove testing line * Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472) * write out new request options provider and refactor components and parts of the YAML config * fix formatting * pr feedback to consolidate body_data_provider to simplify the code * pr feedback get rid of extraneous optional * publish oss for cloud (#13978) workflow to publish oss artifacts that cloud needs to build against use docker buildx to create arm images for local development * skip debezium engine startup in case no table is in INCREMENTAL mode (#13870) * 🎉 Source Github: break point added for workflows_runs stream (#13926) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866) * 6339: debug info * 6339: not using 'USE' on Azure SQL servers * 6339: cleanup * 6339: cleanup2 * 6339: cleanup3 * 6339: versions/changelogs updated * 6339: merge from master (consolidation issue) * 6339: dev connector version (for testing in airbyte cloud) * 6339: code review implementation * 6339: apply formatting * in case runners fail to spin up, this needs to run on github-hosted (#13996) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * :tada: New Source: Webflow (#13617) * Added webflow code * Updated readme * Updated README * Added webflow to source_definitions.yaml * Enhanced documentation for the Webflow source connector * Improved webflow source connector instructions * Moved Site ID to before API token in Spec.yaml (for presentation in the UI) * Addressed comments in PR. * Changes to address requests in PR review * Removed version from config * Minor udpate to spec.yaml for clarity * Updated to pass the accept-version as a constant rather than parameter * Updated check_connection to hit the collections API that requires both site id and the authentication token. * Fixed the test_check_connection to use the new check_connection function * Added a streams test for generate_streams * Re-named "autentication" object to "auth" to be more consistent with the way it is created by the CDK * Added in an explict line to instantiante an "auth" object from WebflowTokenAuthenticator, to make it easier to describe in the blog * Fixed a typo in a comment * Renamed some classes to be more intuitive * Renamed class to be more intuitive * Minor change to an internal method name * Made _get_collection_name_to_id_dict staticmethod * Fixed a unit-test error that only appeared when running " python -m pytest -s unit_tests". This was caused by Mocked settings from test_source.py leaking into test_streams.py * format: add double quotes and remove unused import * readme: remove semantic version naming of connector in build commands * Updated spec.yaml * auto-bump connector version * format files * add changelog * update dockerfile * auto-bump connector version Co-authored-by: sajarin <sajarindider@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Source-oracle: fixed tests + checkstyle (#13997) * Source-oracle: fixed tests + checkstyle * 🐛Destination-mysql: fixed integration test and build process (#13302) * [13180] destination-mysql: fixed integration test * update changelog to include debezium version upgrade (#13844) * make table headers look less like successes (#13999) * source-twilio: implement lookback windows (#13896) * Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) This reverts commit aa28d448d820df9d79c2c0d06b38978d1108fb2c. * Revert "6339: error when attempting to use azure sql database within an elastic pool as source for cdc based replication (#13866)" (#14011) This reverts commit 0d870bd37bc3b5cd798b92115d73bcc45a42d8f7. * [low-code connectors] BasicHttpAuthenticator (#13733) * implement basichttpauthenticator * add optional refresh access token authenticator * remove prints * type hints * Fix and unit test * missing test * Add class to __init__ file * Add comment * migrate JsonSchemas to use basic path instead of JSONPath (#13917) * scaffold for catalog diff, needs fixing on type handling and tests (#13786) * Prepare release of JDBC connectors (#13987) * Prepare release of JDBC connectors * Update source definitions manually * use built in check for if path is definite (#13834) * 13535 Fixed bastion network for integration tests (#14007) * doc: add error troubleshooting `docker-compose up` (#13765) * fix: duplicate resource allocations in `airbyte-temporal` deployment (#13816) * helm-chart: Fix worker deployment format error (#13839) * add catalog diff connection read (#13918) * doc: fix small typo on Shopify documentation (#13992) * add streams to reset to job info (#13919) * Generate api for changes in #13370 and make code compatible (#14014) * Generate api for per-stream updates #13835 (#14021) * Revert "Prepare release of JDBC connectors (#13987)" (#14029) This reverts commit df759b30778082508e2872513800fac34d98ff7c. * Fix per stream state protocol backward compatibility (#14032) * rename state type field to fix backwards compatibility issue * replace usages of stateType with type * support semi incremental by adding extractor record filter (#13520) * support semi incremental by adding extractor record filter * refactor extractor into a record_selector that supports extraction and filtering of response records * Remove pydantic spec from amazon ads and use YAML spec (#13988) * add EdDSA support in SSH tunnel (#9494) * add EdDSA support * verify EdDSA support works correct Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com> * 🎉New source connector: source-metabase (#13752) * Add docs * Close metabase session when sync finishes * Close session in check_connection * Add source definition to seed * Add icon * improve cdc check for connectors (#14005) * improve should use cdc check * Revert "improve should use cdc check" This reverts commit 7d01727279d21d33a6c18ed3227ee94432636120. * improve should use cdc check * add unit test * Update webflow.md * Update webflow.md * Update webflow.md * Remove legacy sentry code from cdk (#14016) * rip sentry out of cdk * remove sentry dsn from gsc * Update webflow.md * Update webflow.md * Fixed broken links (#14071) * 🪟Persist unsaved changes on schema refresh (#13895) * add form values tracker context * add clarifying comment * add same functionality to create connection * Update airbyte-webapp/src/components/CreateConnectionContent/CreateConnectionContent.tsx Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * Fixes broken links so we can deploy again (#14075) also adds better error message for when this happens to others * Adds symmary.md to gitignore (#14078) * Added webflow icon (#14069) * Added webflow icon * Added icon * Build create connection form build failure (#14081) * Fix CDK obfuscation of nested secrets (#14035) * Added Buy Credits section to Managing Airbyte Cloud (#13905) * Added Buy Credits section to Managing Airbyte Cloud * Made some style changes * Made edits based on Natalie's suggestions * Deleted link * Deleted line * Edited email address * Updated reaching out to sales sentence * disable es-lit to fix build (#14087) * Release source connectors (#14077) * Release source connectors * Fix issue with database connection in test * Fix failing tests due to authentication * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump Airbyte version from 0.39.23-alpha to 0.39.24-alpha (#14094) Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> * Emit the state to remove in the airbyte empty source (#13725) What This PR updates the EmptyAirbyteSource in order to perform a partial update and handle the new state message format. How The empty will now emit different messages based on the type of state being provided: Per stream: it will emit one message per stream that have been reset Global: It will emit one global message that will contain null for the stream that have been reset including the shared state Co-authored-by: Jimmy Ma <jimmy@airbyte.io> * Add StatePersistence object (#13900) Add a StatePersistence object that supports Read/Writes of States to the DB with StreamDescriptor fields The only migrations that is supported are * moving from LEGACY to GLOBAL * moving from LEGACY to STREAM * All other state type migrations are expected to go through an explicit reset beforehand. * secret-persistence: Hashicorp Vault Secret Store (#13616) Co-authored-by: Amanda Murphy <amanda.murphy@heapanalytics.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> * 🐛 Source Hubspot: remove `AirbyteSentry` dependency (#14102) * fixed * updated changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * fix: format VaultSecretPersistenceTest.java (#14110) * Source Hubspot: extend error logging (#14054) * #291 incall - source Hubspot: extend error logging * huspot: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Update webflow.md (#14083) * Update webflow.md Removed a description that is only applicable to people that are writing connector code, not to _users_ of the connector. * Update webflow.md * Update webflow.md * Update webflow.md * Update webflow.md * 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) * 12708: Add an option to use encryption with staging in Redshift Destination (#13675) * 12708: Add an option to use encryption with staging in Redshift Destination * 12708: docs/docker configs updated * 12708: merge with master * 12708: merge fix * 12708: code review implementation * 12708: fix for older configs * 12708: fix for older configs in check * 12708: merge from master (consolidation issue) * 12708: versions updated * 12708: specs updated * 12708: specs updated * 12708: removing autogenerated files from PR * 12708: changelog updated * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source PayPal Transaction: Update Transaction Schema (#13682) * Update transaction schema. * Transform money values from strings to floats or integers. Co-authored-by: nataly <nataly@airbyte.io> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * fix(jsonSchemas): raise error when items property not provided (#14018) * fix stream name in stream transformation update (#14044) * 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery. * Remove skiptests option (#14100) * update sentry release script (#14123) * Remove "additionalProperties": false from specs for connectors with staging (#14114) * Remove "additionalProperties": false from spec for connectors with staging * Remove "additionalProperties": false from spec for Redshift destination * bump versions * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * [14003] source-oracle: added custom jdbc field (#14092) * [14003] source-oracle: added custom jdbc field * Add JobErrorReporter for sending sync job connector failures to Sentry (#13899) * skeleton for reporting connector errors to sentry * report on job failures instead of attempt failures * report sync job failures with relevant metadata using JobErrorReporter * send stack traces from python connectors to sentry * test JobCreationAndStatusUpdate and JobErrorReporter * logs * refactor into helper, initial tests * using sentry * run format * load reporting client from env * load sentry dsn from env * send java stack traces to sentry * test sentryclient, refactor to use Hub instance * ErrorReportingClient.report -> .reportJobFailureReason * inject exception helper, test stack trace parse error tagging * rm logs * more stack trace tests * remove logs * fix failing tests * rename ErrorReportingClient to JobErrorReportingClient * rename vars in docker-compose * Return an Optional instead of null when parsing stack traces * dont remove airbyte prefix when setting release name * from_trace_message static * remove failureSummary from jobfailure input, get from Job * send stacktrace string if we weren't able to parse * set deployment mode tag * update .env * just log if something goes wrong * Use StateMessageHelper in source (#14125) * Use StateMessageHelper in source * PR feedback and formatting * More PR feedback * Revert change * Revert changes * Bump Airbyte version from 0.39.24-alpha to 0.39.25-alpha (#14124) Co-authored-by: brianjlai <brianjlai@users.noreply.github.com> * Refactor acceptance tests and utils (#13950) * Refactor Basic acceptance tests and utils * Refactor Advanced acceptance tests and utils * Remove unused code * Clear destination db data during cleanup * Cleanup comments * cleanup init code * test creating new desintation db for each test * cleanup desintation db init * Allow to edit api client * pull in temporal cloud changes * Rename helper to harness; set some funcs to private; turn init into constructor * add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests * update javadoc Co-authored-by: Davin Chia <davinchia@gmail.com> * fix javadoc formatting * fix var naming Co-authored-by: Davin Chia <davinchia@gmail.com> * Bump Airbyte version from 0.39.25-alpha to 0.39.26-alpha (#14141) Co-authored-by: terencecho <terencecho@users.noreply.github.com> * 🎉 octavia-cli: Add ability to get existing resources (#13254) * 13541 Fixed integration tests source-db2 Mac OS (#14133) * 13523 Fix integration tests destination-cassandra Mac OS (#14134) * 🐛 Source Hubspot: fixed SAT test, commented out expected_records (#14140) * :bug: Source Intercom: extend `Contacts` schema with new properties (#14099) * Source Twilio: adopt best practices (#14000) * #1946 Source twilio: aopt best practices - tune tests * #1946 add expected_records to acceptance-test-config.yml * #1946 source twilio - upd schema and changelog * #1946 fix expected_records * #1946 source twilio: rm alerts from expected records as they expire in 30 days * #1946 source twilio: bump version * 🎉 Source BingAds: expose hourly/daily/weekly/monthly options from configuration (#13801) * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: config settings for hourly/daily/weekly/monthly reports added: default value for all periodic reports to True * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused class variables, if-statement * #12489 - expose hourly/daily/weekly/monthly reports in discovery by default instead of in the connector's configuration settings removed: unused variables from config * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * remove VersionMismatchServer (#14076) * remove VersionMismatchServer * remove VersionMismatchServerTest * revert intended changes * Increase instance termination time limit to 3 hours to accommodate connector builds. (#14181) * Use correct bash comment symbol. (#14183) * 🎉 New Source: Orbit.love (#13390) * source-orbit: add definition and specs (#14189) * 🎉 Base Norrmalization: clean-up Redshift `tmp_schemas` after SAT (#14015) Now after `base-normalization` SAT the Destination Redshift will be automatically cleaned up from test leftovers. Other destinations are not covered yet. * Source Salesforce: fix customIntegrationTest for SAT (#14172) * Source Amazon Ads: increase timeout for SAT (#14167) * 🎉 Introduce Google Analytics Data API source (#12701) * Introduce Google Analytics Data API source https://developers.google.com/analytics/devguides/reporting/data/v1 * Add Google Analytics Data API source PR link * Add `client` class for Google Analytics Data API * Move dimensions and metrics extraction to the `client` class In the Google Analytics Data API * Change the copyright date to 2022 in Google Analytics Data API * fix: removing incremental syncs * fix: change project_id to string * fix: flake check is failing * chore: added it to source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) * Publish acceptance test utils maven artifact (#14142) * Fix StatePersistence Legacy read/write (#14129) StatePersistence will wrap/unwrap legacy state on write/read to ensure compatibility with the old behavior/data. * 🎉 Destination connectors: Improved "SecondSync" checks in Standard Destination Acceptance tests (#14184) * [11731] Improved "SecondSync" checks in Standard Destination Acceptance tests * 🐛 Source Zendesk Support: fixed "Retry-After" non integer value (#14112) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Source Tiktok Marketing: Videometrics (#13650) * added video metrics in streams.py * common metrics list updated. * updated streams.py with extended metrics required. * updated stream_test * updated configured_catalog * video metrics required list updated. * chore: formatting * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> * 🎉 Source Github: secondary rate limits has to retry (#13955) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Harshith/test pr 13118 (#14192) * Firebolt destination * feat: Write method dropdown * feat: Use future-proof Auth in SDK * refactor: Move writer instantiation * fix: tests are failing * fix: tests are failing * fix: tests are failing * chore: added connector to definitions * fix: formatting and spec * fix: formatting for orbit Co-authored-by: ptiurin <petro.tiurin@firebolt.io> * 🪟 :art: Show credit usage on chart's specific day (#13503) * add tooltip to chart * Fixes: - update main chart color; - change onHover background color * change chart color pallet to grey 500 * update color reference * remove opacity from UsageCell * 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) * Improve documentation for Postgres Source (#13830) * Improve documentation for Postgres Source * add information about additional JDBC params * add anchors for doc sections * fix link to CDC on Bare Metal * add more details about parsing date/time values * add doc link to SSH fields * Handle null reset source config (#14202) * handle null reset source config * format * Wait indefinitely if connection is not active (#14200) * also wait indefinitely if connection is deleted * fix test * Bump Airbyte version from 0.39.26-alpha to 0.39.27-alpha (#14204) Co-authored-by: lmossman <lmossman@users.noreply.github.com> * Bmoric/feature flag for state deserialization (#14127) * Add Feature flag * Add default feature flag value * Update test * remove unsused * tmp * Update tests * rm unwanted change * PR comments * [low-code connectors] default types and default values (#14004) * default types and default values * cleanup * fixes so read works * remove prints and trycatch * comment * remove unused param * split file * extract method * extract methods * comment * optional * fix test * cleanup * delete interpolated request header provider * simplify next page url paginator interface * comment * format * add state type endpoint (#14111) * Bump Airbyte version from 0.39.27-alpha to 0.39.28-alpha (#14210) Co-authored-by: sherifnada <sherifnada@users.noreply.github.com> * 🐛 source-orbit: remove workspace_old.json (#14208) * Fix: Docs plural login redirecting to wrong URL (#14207) * [docs] fix numbering and incorrect filename in CDK docs (#13045) * [docs] fix numbering in CDK docs * Update 5-declare-schema.md * Update 5-declare-schema.md * Update 6-read-data.md * Update 8-test-your-connector.md * Remove the old scheduler from HelmCharts helper (#14187) * Remove the old scheduler from HelmCharts helper The old scheduler was removed as part of https://github.com/airbytehq/airbyte/pull/13400 * Remove legacy `scheduler` comment in HelmCharts * Source Gitlab: add GroupIssueBoards stream (#13252) * GitLab Source: add GroupIssueBoards stream * Address stream schema comments * Address comments * Bump version * Add as empty stream * run seed file source (#14215) * fix 'cannot reach server' error on demo instance (#10020) * Update CODEOWNERS (#14209) * 🎉 Source Github: use GraphQL for `reviews` stream (#13989) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * workflow for publishing artifacts for cloud (#14199) * fix sentry org slug change (#14218) * Source File: correct spec json to match json format (#13738) * Upgrade spotless version and remove jvmargs workaround (#13705) * Source Zendesk Chat: Process large amount of data in batches for incremental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Remove all @ts-ignore (#14221) * Bump hadoop to use version 3.3.3 (#14182) * Change the persistence activity to use the new persistence layer (#14205) * Change the persistence activity to use the new persistence layer * Use lombok * format * Use new State message helper * Fix build (#14225) * Fix build * Fix test * Use new state persistence for state reads (#14126) * Inject StatePersistence into DefaultJobCreator * Read the state from StatePersistence instead of ConfigRepository * Add a conversion helper to convert StateWrapper to State * Remove unused ConfigRepository.getConnectionState * Temporal per stream resets (#13990) * remove reset flags from workflow state + refactor * bring back cancelledForReset, since we need to distinguish between that case and a normal cancel * delete reset job streams on cancel or success * extract isResetJob to method * merge with master * set sync modes on streams in reset job correctly * format * Add test for getAllStreamsForConnection * fix tests * update more tests * add StreamResetActivityTests * fix tests for default job creator * remove outdated comment * remove debug lines * remove unused enum value * fix tests * fix constant equals ordering * make job mock not static * DRY and add comments * add comment about deleted streams * Remove io.airbyte.config.StreamDescriptor * regisster stream reset activity impl * refetch connection workflow when checking job id, since it may have been restarted * only cancel if workflow is running, to allow reset signal to always succeed even if batched with a workflow start * fix reset signal to use new doneWaiting workflow state prop * try to fix tests * fix reset cancel case * add acceptance test for resetting while sync is running * format * fix new acceptance test * lower sleep on test * raise sleep * increase sleep and timeout, and remove repeated test * use CatalogHelpers to extract stream descriptors * raise sleep and timeout to prevent transient failures * format Co-authored-by: alovew <anne@airbyte.io> * Log Marketo response errors * Additional message if Marketo response has failed with empty errors Co-authored-by: Topher Lubaway <asimplechris@gmail.com> Co-authored-by: Malik Diarra <malik@airbyte.io> Co-authored-by: Teal Larson <LARSON.TEAL@GMAIL.COM> Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: VitaliiMaltsev <39538064+VitaliiMaltsev@users.noreply.github.com> Co-authored-by: grishick <greg@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: Andy <andy@airbyte.io> Co-authored-by: LiRen Tu <tuliren.git@outlook.com> Co-authored-by: oneshcheret <33333155+sashaNeshcheret@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Xiaohan Song <xiaohan@airbyte.io> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Charles <charles@airbyte.io> Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com> Co-authored-by: Sophia Wiley <106352739+sophia-wiley@users.noreply.github.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: Denis Davydov <davydov.den18@gmail.com> Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com> Co-authored-by: Bastien Gandouet <bastien@mozaiqu.es> Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> Co-authored-by: Paweł Kociński <pawel93kocinski@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Alexander Marquardt <alexander.marquardt@gmail.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: lmossman <lmossman@users.noreply.github.com> Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: mutt-santiago-stachuk <106173005+mutt-santiago-stachuk@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: Petro Tiurin <93913847+ptiurin@users.noreply.github.com> Co-authored-by: ptiurin <petro.tiurin@firebolt.io> Co-authored-by: Andrii Leonets <30464745+DoNotPanicUA@users.noreply.github.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Jimmy Ma <jimmy@airbyte.io> Co-authored-by: Ganpat Agarwal <gagarwal@artica.com> Co-authored-by: Rajakavitha Kodhandapani <krajakavitha@gmail.com> Co-authored-by: Yevhen Sukhomud <suhomud@gmail.com> Co-authored-by: sw-yx <shawnthe1@gmail.com> Co-authored-by: alafanechere <alafanechere@users.noreply.github.com> Co-authored-by: Eugene <etsybaev@gmail.com> Co-authored-by: Anna Lvova <37615075+annalvova05@users.noreply.github.com> Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> Co-authored-by: Phlair <Phlair@users.noreply.github.com> Co-authored-by: Parker Mossman <parker@airbyte.io> Co-authored-by: Adam <adam-bloom@users.noreply.github.com> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: pmossman <pmossman@users.noreply.github.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> Co-authored-by: Peter Hu <peter@airbyte.io> Co-authored-by: Tuhai Maksym <kimerinn@gmail.com> Co-authored-by: sajarin <sajarindider@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: steve withington <steve@digitalmine.com> Co-authored-by: Leo Sussan <leosussan@gmail.com> Co-authored-by: cenegd <cenegd@live.com> Co-authored-by: Tomas Perez Alvarez <72174660+Tomperez98@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: Stella Chung <schung507@gmail.com> Co-authored-by: Amanda Murphy <amanda.murphy@heapanalytics.com> Co-authored-by: Mohamed Magdy <mohamed.magdy@canary.is> Co-authored-by: nataly <nataly@airbyte.io> Co-authored-by: Tyler Russell <tylerrussell85@gmail.com> Co-authored-by: Alexander Tsukanov <alexander.tsukanovvv@gmail.com> Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> Co-authored-by: brianjlai <brianjlai@users.noreply.github.com> Co-authored-by: terencecho <terence@airbyte.io> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Daniel Diamond <33811744+danieldiamond@users.noreply.github.com> Co-authored-by: drrest <dr.rest@gmail.com> Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io> Co-authored-by: Zawar Khan <zawar.khan@getmercury.io> Co-authored-by: sherifnada <sherifnada@users.noreply.github.com> Co-authored-by: Sachin Jangid <sachinjangid832@gmail.com> Co-authored-by: Chris Wu <chris@faros.ai> Co-authored-by: Jared Rhizor <me@jaredrhizor.com> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com> Co-authored-by: alovew <anne@airbyte.io>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #13646
What
Per the conversation in the Per-Stream Tech Spec, destinations need to be able to handle state new types of state messages (GLOBAL and STREAM). The latter requires being able to store and emit multiple state messages (one per stream). To avoid any memory pressure, we always only store the last message. For GLOBAL and LEGACY that means one state record. For STREAM that means one state message per stream. This approach means that this change should NOT affect the memory footprint in the destinations do the new state types.
This change is backwards compatible. All old state messages will continue to function in exactly the way they did before.
How
DestStateLifecycleManagerto allow for better separation of concerns and testing. Especially helpful now as the new state types add more complexity here.DefaultDestStateLifecycleManageris responsible for routing state to the correct manager (e.g. theDestSingleStateLifecycleManagerfor GLOBAL and LEGACY and theDestStreamStateLifecycleManagerfor STREAM).Reading Order
DestStateLifecycleManager.javaDefaultDestStateLifecycleManager.java(and testsDefaultDestStateLifecycleManagerto see behavior)DestSingleStateLifecycleManager.java(and testsDestSingleStateLifecycleManagerto see behavior)DestStreamStateLifecycleManager.java(and testsDestStreamStateLifecycleManagerto see behavior)BufferedStreamConsumer.java