Dataproc config for serverless in profile and fix quoting #578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a `dataproc_batch` key for specifying the Dataproc Batch configuration. At runtime this is used to populate the google.cloud.dataproc_v1.types.Batch object before it is submitted to the Dataproc service. To avoid having to add explicit support for every option offered by the service, and having to chase after a moving target as Google's API evolves, this key accepts arbitrary yaml, which is mapped to the Batch object on a best effort basis. Signed-off-by: Torkjel Hongve <th@kinver.io>
- Make dataproc_batch key optional. - Unit tests - Move configuration of the `google.cloud.dataproc_v1.Batch` object to a separate function. Signed-off-by: Torkjel Hongve <th@kinver.io>
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide. |
…p python_submissions.py
McKnight-42
reviewed
Mar 3, 2023
dev-requirements.txt
Outdated
| @@ -1,7 +1,7 @@ | |||
| # install latest changes in dbt-core | |||
| # TODO: how to automate switching from develop to version branches? | |||
| git+https://github.com/dbt-labs/dbt-core.git#egg=dbt-core&subdirectory=core | |||
| git+https://github.com/dbt-labs/dbt-core.git#egg=dbt-tests-adapter&subdirectory=tests/adapter | |||
| git+https://github.com/dbt-labs/dbt-core.git@addPyRelationNameMacro#egg=dbt-core&subdirectory=core | |||
There was a problem hiding this comment.
nit: dbt-core pointer once approved
McKnight-42
reviewed
Mar 3, 2023
| @@ -184,6 +221,25 @@ def test_acquire_connection_oauth_validations(self, mock_open_connection): | |||
| connection.handle | |||
| mock_open_connection.assert_called_once() | |||
|
|
|||
| @patch('dbt.adapters.bigquery.connections.get_bigquery_defaults', return_value=('credentials', 'project_id')) | |||
| @patch('dbt.adapters.bigquery.BigQueryConnectionManager.open', return_value=_bq_conn()) | |||
| def test_acquire_connection_dataproc_serverless(self, mock_open_connection, mock_get_bigquery_defaults): | |||
There was a problem hiding this comment.
nit: does this go beyond flake8 and black line length?
nssalian
approved these changes
Mar 9, 2023
|
@cla-bot check |
1 similar comment
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.4.latest 1.4.latest
# Navigate to the new working tree
cd .worktrees/backport-1.4.latest
# Create a new branch
git switch --create backport-578-to-1.4.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f2f0e4272c42b2c2b59847279373c36fb08e3e37
# Push it to GitHub
git push --set-upstream origin backport-578-to-1.4.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.4.latestThen, create a pull request where the |
colin-rogers-dbt
added a commit
that referenced
this pull request
Mar 10, 2023
colin-rogers-dbt
added a commit
that referenced
this pull request
Mar 10, 2023
3 tasks
3 tasks
1 task
2 tasks
mirnawong1
added a commit
to dbt-labs/docs.getdbt.com
that referenced
this pull request
Jul 3, 2023
#3661) Resolves #3382 ## What are you changing in this pull request and why? The dataproc_batch config field for configuring Python models running on GCP Dataproc Serverless was added in [dbt-labs/dbt-bigquery#578](dbt-labs/dbt-bigquery#578), but it isn't documented yet. This PR adds the missing documentation for this field. ## Checklist - [ ] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) and [About versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) so my content adheres to these guidelines. - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #350
Description
Checklist
changie newto create a changelog entry