From 5ab0fe22d8c318b39ac1cf7713214ee6000b8d84 Mon Sep 17 00:00:00 2001 From: arithmetic1728 <58957152+arithmetic1728@users.noreply.github.com> Date: Fri, 7 Aug 2020 17:35:44 -0700 Subject: [PATCH 001/158] chore: move samples from python-docs-sample (#66) * Add XMPP Sample * Add Dataproc Sample * Add more region tags * Minor dataproc fixes * Fix Dataproc e2e for Python 3 * Update reqs * updating requirements [(#358)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/358) Change-Id: I6177a17fad021e26ed76679d9db34848c17b62a8 * Update Reqs * Wrong arg description * Auto-update dependencies. [(#456)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/456) * Auto-update dependencies. [(#459)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/459) * Fix import order lint errors Change-Id: Ieaf7237fc6f925daec46a07d2e81a452b841198a * bump Change-Id: I02e7767d13ba267ee9fc72c5b68a57013bb8b8d3 * Auto-update dependencies. [(#486)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/486) * Auto-update dependencies. [(#540)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/540) * Auto-update dependencies. [(#542)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/542) * Move to google-cloud [(#544)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/544) * Auto-update dependencies. [(#584)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/584) * Auto-update dependencies. [(#629)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/629) * Update samples to support latest Google Cloud Python [(#656)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/656) * Update README.md [(#691)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/691) * Auto-update dependencies. [(#715)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/715) * Auto-update dependencies. [(#735)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/735) * Auto-update dependencies. * Fix language OCR sample * Remove unused import * Auto-update dependencies. [(#790)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/790) * Remove usage of GoogleCredentials [(#810)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/810) * Fix a typo [(#813)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/813) * Remove cloud config fixture [(#887)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/887) * Remove cloud config fixture * Fix client secrets * Fix bigtable instance * Fix reference to our testing tools * Auto-update dependencies. [(#914)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/914) * Auto-update dependencies. * xfail the error reporting test * Fix lint * Auto-update dependencies. [(#922)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/922) * Auto-update dependencies. * Fix pubsub iam samples * Auto-update dependencies. [(#1005)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1005) * Auto-update dependencies. * Fix bigtable lint * Fix IOT iam interaction * Auto-update dependencies. [(#1011)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1011) * Properly forwarding the "region" parameter provided as an input argument. [(#1029)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1029) * Auto-update dependencies. [(#1055)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1055) * Auto-update dependencies. * Explicitly use latest bigtable client Change-Id: Id71e9e768f020730e4ca9514a0d7ebaa794e7d9e * Revert language update for now Change-Id: I8867f154e9a5aae00d0047c9caf880e5e8f50c53 * Remove pdb. smh Change-Id: I5ff905fadc026eebbcd45512d4e76e003e3b2b43 * Fix region handling and allow to use an existing cluster. [(#1053)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1053) * Auto-update dependencies. [(#1094)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1094) * Auto-update dependencies. * Relax assertions in the ocr_nl sample Change-Id: I6d37e5846a8d6dd52429cb30d501f448c52cbba1 * Drop unused logging apiary samples Change-Id: I545718283773cb729a5e0def8a76ebfa40829d51 * Auto-update dependencies. [(#1133)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1133) * Auto-update dependencies. * Fix missing http library Change-Id: I99faa600f2f3f1f50f57694fc9835d7f35bda250 * Auto-update dependencies. [(#1186)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1186) * Auto-update dependencies. [(#1199)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1199) * Auto-update dependencies. * Fix iot lint Change-Id: I6289e093bdb35e38f9e9bfc3fbc3df3660f9a67e * Fixed Failed Kokoro Test (Dataproc) [(#1203)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1203) * Fixed Failed Kokoro Test (Dataproc) * Fixed Lint Error * Update dataproc_e2e_test.py * Update dataproc_e2e_test.py * Fixing More Lint Errors * Fixed b/65407087 * Revert "Merge branch 'master' of https://github.com/michaelawyu/python-docs-samples" This reverts commit 1614c7d3ef33630a8ab095792b27fc25fd91f0ad, reversing changes made to cd1dbfd25997a154a8a85cc754cc2a85b18a63c4. * Revert "Fixed b/65407087" This reverts commit cd1dbfd25997a154a8a85cc754cc2a85b18a63c4. * Fixed Lint Error * Fixed Lint Error * Auto-update dependencies. [(#1208)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1208) * Dataproc GCS sample plus doc touchups [(#1151)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1151) * Auto-update dependencies. [(#1217)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1217) * Auto-update dependencies. [(#1239)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1239) * Added "Open in Cloud Shell" buttons to README files [(#1254)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1254) * Auto-update dependencies. [(#1282)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1282) * Auto-update dependencies. * Fix storage acl sample Change-Id: I413bea899fdde4c4859e4070a9da25845b81f7cf * Auto-update dependencies. [(#1309)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1309) * Auto-update dependencies. [(#1320)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1320) * Auto-update dependencies. [(#1355)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1355) * Auto-update dependencies. [(#1359)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1359) * Auto-update dependencies. * update Dataproc region tags to standard format [(#1826)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1826) * Update submit_job_to_cluster.py [(#1708)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1708) switch region to new 'global' region and remove unnecessary function. * Auto-update dependencies. [(#1846)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1846) ACK, merging. * Need separate install for google-cloud-storage [(#1863)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1863) * Revert "Update dataproc/submit_job_to_cluster.py" [(#1864)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1864) * Revert "Remove test configs for non-testing directories [(#1855)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1855)" This reverts commit 73a73321579337312e8ba85c34fe9c37b42b7f6e. * Revert "Auto-update dependencies. [(#1846)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1846)" This reverts commit 3adc94f4d0c14453153968c3851fae100e2c5e44. * Revert "Tweak slack sample [(#1847)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1847)" This reverts commit a48c010481c166968d9f1bd58106054c5d1c58f9. * Revert "Non-client library example of constructing a Signed URL [(#1837)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1837)" This reverts commit fc3284d995a8a35c473a207e80490fad265782af. * Revert "GCF samples: handle {empty JSON, GET} requests + remove commas [(#1832)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1832)" This reverts commit 6928491ed3d52b0bec694e6b30257f08caac5f2b. * Revert "Correct the maintenance event types [(#1830)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1830)" This reverts commit c22840fd23586349b7b665d851dea046a94ba7c7. * Revert "Fix GCF region tags [(#1827)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1827)" This reverts commit 0fbfef27d35cea23ad0e20fd2c9df3e8a4a046cb. * Revert "Updated to Flask 1.0 [(#1819)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1819)" This reverts commit d52ccf99503311bba2cec2881e8cb0f9b5a6f2bf. * Revert "Fix deprecation warning [(#1801)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1801)" This reverts commit 981737e85f60eca5cc337f172249deddca9b291b. * Revert "Update submit_job_to_cluster.py [(#1708)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1708)" This reverts commit df1f2b22547b7ca86bbdb791ad930003a815a677. * Create python-api-walkthrough.md [(#1966)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1966) * Create python-api-walkthrough.md This Google Cloud Shell walkthrough is linked to Cloud Dataproc documentation to be published at: https://cloud.google.com/dataproc/docs/tutorials/python-library-example * Update python-api-walkthrough.md * Update list_clusters.py [(#1887)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1887) * Auto-update dependencies. [(#1980)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1980) * Auto-update dependencies. * Update requirements.txt * Update requirements.txt * Update Dataproc samples. [(#2158)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2158) * Update requirements.txt * Update python-api-walkthrough.md * Update submit_job_to_cluster.py * Update list_clusters.py * Update python-api-walkthrough.md [(#2172)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2172) * Adds updates including compute [(#2436)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2436) * Adds updates including compute * Python 2 compat pytest * Fixing weird \r\n issue from GH merge * Put asset tests back in * Re-add pod operator test * Hack parameter for k8s pod operator * feat: adding samples for dataproc - create cluster [(#2536)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2536) * adding sample for cluster create * small fix * Add create cluster samples * Fixed copyright, added 'dataproc' to region tag and changed imports from 'dataproc' to 'dataproc_v1' * Fix copyright in create_cluster.py * Auto-update dependencies. [(#2005)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh * feat: dataproc quickstart sample added and create_cluster updated [(#2629)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2629) * Adding quickstart sample * Added new quickstart sample and updated create_cluster sample * Fix to create_cluster.py * deleted dataproc quickstart files not under dataproc/quickstart/ * Added quickstart test * Linting and formatting fixes * Revert "Linting and formatting fixes" This reverts commit c5afcbcdf9deccbb7a21ddd82ae0fc305e79c008. * Added bucket cleanup to quickstart test * Changes to samples and tests * Linting fixes * Removed todos in favor of clearer docstring * Fixed lint error Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * Update Python Cloud Shell walkthrough script [(#2733)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2733) Cloud Shell walkthrough scripts no longer support enabling APIs. APIs must be enabled by linking to the console. Updated product name: "Cloud Dataproc" -> "Dataproc". * fix: added cli functionality to dataproc quickstart example [(#2734)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2734) * Added CLI functionality to quickstart * Fixed Dataproc quickstart test to properly clean up GCS bucket [(#3001)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3001) * splitting up #2651 part 1/3 - dataproc + endpoints [(#3025)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3025) * splitting up #2651 * fix typos * chore(deps): update dependency google-auth to v1.11.2 [(#2724)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2724) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-cloud-storage to v1.26.0 [(#3046)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3046) * chore(deps): update dependency google-cloud-storage to v1.26.0 * chore(deps): specify dependencies by python version * chore: up other deps to try to remove errors Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole * chore(deps): update dependency google-cloud-dataproc to v0.7.0 [(#3083)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3083) * feat: added dataproc workflows samples [(#3056)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3056) * Added workflows sample * chore(deps): update dependency grpcio to v1.27.2 [(#3173)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3173) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio](https://grpc.io) | minor | `==1.25.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | minor | `==1.23.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | minor | `==1.26.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | patch | `==1.27.1` -> `==1.27.2` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Simplify noxfile setup. [(#2806)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot * fix: add mains to samples [(#3284)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3284) Added mains to two samples: create_cluster and instantiate_inline_workflow_templates. Fixed their associated tests to accommodate this. Removed subprocess from quickstart/quickstart_test.py to fix [2873](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/2873) fixes #2873 * Update dependency grpcio to v1.28.1 [(#3276)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3276) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * Update dependency google-auth to v1.14.0 [(#3148)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3148) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-auth to v1.14.1 [(#3464)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3464) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.0` -> `==1.14.1` | | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | minor | `==1.11.2` -> `==1.14.1` | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.14.1`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1141-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1140v1141-2020-04-21) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.0...v1.14.1)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-storage to v1.28.0 [(#3260)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3260) Co-authored-by: Takashi Matsuo * chore(deps): update dependency google-auth to v1.14.2 [(#3724)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3724) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.1` -> `==1.14.2` | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.14.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1142-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1141v1142-2020-05-07) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.1...v1.14.2)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore: some lint fixes [(#3743)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3743) * chore(deps): update dependency google-auth to v1.14.3 [(#3728)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3728) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.2` -> `==1.14.3` | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.14.3`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1143-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1142v1143-2020-05-11) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.2...v1.14.3)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [x] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency grpcio to v1.29.0 [(#3786)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3786) * chore(deps): update dependency google-cloud-storage to v1.28.1 [(#3785)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3785) * chore(deps): update dependency google-cloud-storage to v1.28.1 * [asset] testing: use uuid instead of time Co-authored-by: Takashi Matsuo * update google-auth to 1.15.0 part 3 [(#3816)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3816) * Update dependency google-cloud-dataproc to v0.8.0 [(#3837)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3837) * chore(deps): update dependency google-auth to v1.16.0 [(#3903)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3903) * update google-auth part 3 [(#3963)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/3963) * chore(deps): update dependency google-cloud-dataproc to v0.8.1 [(#4015)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4015) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | patch | `==0.8.0` -> `==0.8.1` | --- ### Release Notes
googleapis/python-dataproc ### [`v0.8.1`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#​081-httpswwwgithubcomgoogleapispython-dataproccomparev080v081-2020-06-05) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v0.8.0...v0.8.1)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4022) * Update dependency google-auth to v1.17.0 [(#4058)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4058) * chore(deps): update dependency google-auth to v1.17.1 [(#4073)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4073) * Update dependency google-auth to v1.17.2 [(#4083)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4083) * Update dependency google-auth to v1.18.0 [(#4125)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4125) * Update dependency google-cloud-dataproc to v1 [(#4109)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4109) Co-authored-by: Takashi Matsuo * chore(deps): update dependency google-cloud-storage to v1.29.0 [(#4040)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4040) * chore(deps): update dependency grpcio to v1.30.0 [(#4143)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4143) Co-authored-by: Takashi Matsuo * Update dependency google-auth-httplib2 to v0.0.4 [(#4255)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4255) Co-authored-by: Takashi Matsuo * chore(deps): update dependency pytest to v5.4.3 [(#4279)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole * chore(deps): update dependency google-auth to v1.19.0 [(#4293)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4293) * chore(deps): update dependency google-cloud-dataproc to v1.0.1 [(#4309)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4309) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | patch | `==1.0.0` -> `==1.0.1` | --- ### Release Notes
googleapis/python-dataproc ### [`v1.0.1`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#​101-httpswwwgithubcomgoogleapispython-dataproccomparev100v101-2020-07-16) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v1.0.0...v1.0.1)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-auth to v1.19.1 [(#4304)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4304) * chore(deps): update dependency google-auth to v1.19.2 [(#4321)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4321) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.19.1` -> `==1.19.2` | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.19.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1192-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1191v1192-2020-07-17) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.19.1...v1.19.2)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Update dependency google-auth to v1.20.0 [(#4387)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4387) * Update dependency pytest to v6 [(#4390)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4390) * Update dependency grpcio to v1.31.0 [(#4438)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4438) * chore(deps): update dependency google-auth to v1.20.1 [(#4452)](https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4452) * chore: update templates Co-authored-by: Bill Prin Co-authored-by: Bill Prin Co-authored-by: Jon Wayne Parrott Co-authored-by: Eran Kampf Co-authored-by: DPE bot Co-authored-by: aman-ebay Co-authored-by: Martial Hue Co-authored-by: Gioia Ballin Co-authored-by: michaelawyu Co-authored-by: michaelawyu Co-authored-by: Alix Hamilton Co-authored-by: James Winegar Co-authored-by: Charles Engelke Co-authored-by: Gus Class Co-authored-by: Brad Miro Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: WhiteSource Renovate Co-authored-by: Leah Cole Co-authored-by: Takashi Matsuo --- dataproc/snippets/README.md | 84 +++++ dataproc/snippets/create_cluster.py | 77 +++++ dataproc/snippets/create_cluster_test.py | 47 +++ dataproc/snippets/dataproc_e2e_donttest.py | 32 ++ .../instantiate_inline_workflow_template.py | 107 +++++++ ...stantiate_inline_workflow_template_test.py | 31 ++ dataproc/snippets/list_clusters.py | 63 ++++ dataproc/snippets/noxfile.py | 224 ++++++++++++++ dataproc/snippets/pyspark_sort.py | 28 ++ dataproc/snippets/pyspark_sort_gcs.py | 30 ++ dataproc/snippets/python-api-walkthrough.md | 170 +++++++++++ dataproc/snippets/quickstart/quickstart.py | 159 ++++++++++ .../snippets/quickstart/quickstart_test.py | 71 +++++ dataproc/snippets/requirements-test.txt | 1 + dataproc/snippets/requirements.txt | 6 + dataproc/snippets/single_job_workflow.py | 209 +++++++++++++ dataproc/snippets/submit_job_to_cluster.py | 288 ++++++++++++++++++ 17 files changed, 1627 insertions(+) create mode 100644 dataproc/snippets/README.md create mode 100644 dataproc/snippets/create_cluster.py create mode 100644 dataproc/snippets/create_cluster_test.py create mode 100644 dataproc/snippets/dataproc_e2e_donttest.py create mode 100644 dataproc/snippets/instantiate_inline_workflow_template.py create mode 100644 dataproc/snippets/instantiate_inline_workflow_template_test.py create mode 100644 dataproc/snippets/list_clusters.py create mode 100644 dataproc/snippets/noxfile.py create mode 100644 dataproc/snippets/pyspark_sort.py create mode 100644 dataproc/snippets/pyspark_sort_gcs.py create mode 100644 dataproc/snippets/python-api-walkthrough.md create mode 100644 dataproc/snippets/quickstart/quickstart.py create mode 100644 dataproc/snippets/quickstart/quickstart_test.py create mode 100644 dataproc/snippets/requirements-test.txt create mode 100644 dataproc/snippets/requirements.txt create mode 100644 dataproc/snippets/single_job_workflow.py create mode 100644 dataproc/snippets/submit_job_to_cluster.py diff --git a/dataproc/snippets/README.md b/dataproc/snippets/README.md new file mode 100644 index 000000000000..98622be7dc16 --- /dev/null +++ b/dataproc/snippets/README.md @@ -0,0 +1,84 @@ +# Cloud Dataproc API Examples + +[![Open in Cloud Shell][shell_img]][shell_link] + +[shell_img]: http://gstatic.com/cloudssh/images/open-btn.png +[shell_link]: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=dataproc/README.md + +Sample command-line programs for interacting with the Cloud Dataproc API. + +See [the tutorial on the using the Dataproc API with the Python client +library](https://cloud.google.com/dataproc/docs/tutorials/python-library-example) +for information on a walkthrough you can run to try out the Cloud Dataproc API sample code. + +Note that while this sample demonstrates interacting with Dataproc via the API, the functionality demonstrated here could also be accomplished using the Cloud Console or the gcloud CLI. + +`list_clusters.py` is a simple command-line program to demonstrate connecting to the Cloud Dataproc API and listing the clusters in a region. + +`submit_job_to_cluster.py` demonstrates how to create a cluster, submit the +`pyspark_sort.py` job, download the output from Google Cloud Storage, and output the result. + +`single_job_workflow.py` uses the Cloud Dataproc InstantiateInlineWorkflowTemplate API to create an ephemeral cluster, run a job, then delete the cluster with one API request. + +`pyspark_sort.py_gcs` is the same as `pyspark_sort.py` but demonstrates + reading from a GCS bucket. + +## Prerequisites to run locally: + +* [pip](https://pypi.python.org/pypi/pip) + +Go to the [Google Cloud Console](https://console.cloud.google.com). + +Under API Manager, search for the Google Cloud Dataproc API and enable it. + +## Set Up Your Local Dev Environment + +To install, run the following commands. If you want to use [virtualenv](https://virtualenv.readthedocs.org/en/latest/) +(recommended), run the commands within a virtualenv. + + * pip install -r requirements.txt + +## Authentication + +Please see the [Google cloud authentication guide](https://cloud.google.com/docs/authentication/). +The recommended approach to running these samples is a Service Account with a JSON key. + +## Environment Variables + +Set the following environment variables: + + GOOGLE_CLOUD_PROJECT=your-project-id + REGION=us-central1 # or your region + CLUSTER_NAME=waprin-spark7 + ZONE=us-central1-b + +## Running the samples + +To run list_clusters.py: + + python list_clusters.py $GOOGLE_CLOUD_PROJECT --region=$REGION + +`submit_job_to_cluster.py` can create the Dataproc cluster or use an existing cluster. To create a cluster before running the code, you can use the [Cloud Console](console.cloud.google.com) or run: + + gcloud dataproc clusters create your-cluster-name + +To run submit_job_to_cluster.py, first create a GCS bucket (used by Cloud Dataproc to stage files) from the Cloud Console or with gsutil: + + gsutil mb gs:// + +Next, set the following environment variables: + + BUCKET=your-staging-bucket + CLUSTER=your-cluster-name + +Then, if you want to use an existing cluster, run: + + python submit_job_to_cluster.py --project_id=$GOOGLE_CLOUD_PROJECT --zone=us-central1-b --cluster_name=$CLUSTER --gcs_bucket=$BUCKET + +Alternatively, to create a new cluster, which will be deleted at the end of the job, run: + + python submit_job_to_cluster.py --project_id=$GOOGLE_CLOUD_PROJECT --zone=us-central1-b --cluster_name=$CLUSTER --gcs_bucket=$BUCKET --create_new_cluster + +The script will setup a cluster, upload the PySpark file, submit the job, print the result, then, if it created the cluster, delete the cluster. + +Optionally, you can add the `--pyspark_file` argument to change from the default `pyspark_sort.py` included in this script to a new script. diff --git a/dataproc/snippets/create_cluster.py b/dataproc/snippets/create_cluster.py new file mode 100644 index 000000000000..b4d63d2e13f5 --- /dev/null +++ b/dataproc/snippets/create_cluster.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python + +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This sample walks a user through creating a Cloud Dataproc cluster using +# the Python client library. +# +# This script can be run on its own: +# python create_cluster.py ${PROJECT_ID} ${REGION} ${CLUSTER_NAME} + + +import sys + +# [START dataproc_create_cluster] +from google.cloud import dataproc_v1 as dataproc + + +def create_cluster(project_id, region, cluster_name): + """This sample walks a user through creating a Cloud Dataproc cluster + using the Python client library. + + Args: + project_id (string): Project to use for creating resources. + region (string): Region where the resources should live. + cluster_name (string): Name to use for creating a cluster. + """ + + # Create a client with the endpoint set to the desired cluster region. + cluster_client = dataproc.ClusterControllerClient(client_options={ + 'api_endpoint': f'{region}-dataproc.googleapis.com:443', + }) + + # Create the cluster config. + cluster = { + 'project_id': project_id, + 'cluster_name': cluster_name, + 'config': { + 'master_config': { + 'num_instances': 1, + 'machine_type_uri': 'n1-standard-1' + }, + 'worker_config': { + 'num_instances': 2, + 'machine_type_uri': 'n1-standard-1' + } + } + } + + # Create the cluster. + operation = cluster_client.create_cluster(project_id, region, cluster) + result = operation.result() + + # Output a success message. + print(f'Cluster created successfully: {result.cluster_name}') + # [END dataproc_create_cluster] + + +if __name__ == "__main__": + if len(sys.argv) < 4: + sys.exit('python create_cluster.py project_id region cluster_name') + + project_id = sys.argv[1] + region = sys.argv[2] + cluster_name = sys.argv[3] + create_cluster(project_id, region, cluster_name) diff --git a/dataproc/snippets/create_cluster_test.py b/dataproc/snippets/create_cluster_test.py new file mode 100644 index 000000000000..6b1d6806100e --- /dev/null +++ b/dataproc/snippets/create_cluster_test.py @@ -0,0 +1,47 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import uuid + +from google.cloud import dataproc_v1 as dataproc +import pytest + +import create_cluster + + +PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] +REGION = 'us-central1' +CLUSTER_NAME = 'py-cc-test-{}'.format(str(uuid.uuid4())) + + +@pytest.fixture(autouse=True) +def teardown(): + yield + + cluster_client = dataproc.ClusterControllerClient(client_options={ + 'api_endpoint': f'{REGION}-dataproc.googleapis.com:443' + }) + # Client library function + operation = cluster_client.delete_cluster(PROJECT_ID, REGION, CLUSTER_NAME) + # Wait for cluster to delete + operation.result() + + +def test_cluster_create(capsys): + # Wrapper function for client library function + create_cluster.create_cluster(PROJECT_ID, REGION, CLUSTER_NAME) + + out, _ = capsys.readouterr() + assert CLUSTER_NAME in out diff --git a/dataproc/snippets/dataproc_e2e_donttest.py b/dataproc/snippets/dataproc_e2e_donttest.py new file mode 100644 index 000000000000..44cc03bfd428 --- /dev/null +++ b/dataproc/snippets/dataproc_e2e_donttest.py @@ -0,0 +1,32 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" Integration tests for Dataproc samples. + +Creates a Dataproc cluster, uploads a pyspark file to Google Cloud Storage, +submits a job to Dataproc that runs the pyspark file, then downloads +the output logs from Cloud Storage and verifies the expected output.""" + +import os + +import submit_job_to_cluster + +PROJECT = os.environ['GOOGLE_CLOUD_PROJECT'] +BUCKET = os.environ['CLOUD_STORAGE_BUCKET'] +CLUSTER_NAME = 'testcluster3' +ZONE = 'us-central1-b' + + +def test_e2e(): + output = submit_job_to_cluster.main( + PROJECT, ZONE, CLUSTER_NAME, BUCKET) + assert b"['Hello,', 'dog', 'elephant', 'panther', 'world!']" in output diff --git a/dataproc/snippets/instantiate_inline_workflow_template.py b/dataproc/snippets/instantiate_inline_workflow_template.py new file mode 100644 index 000000000000..f9358376f9f9 --- /dev/null +++ b/dataproc/snippets/instantiate_inline_workflow_template.py @@ -0,0 +1,107 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This sample walks a user through instantiating an inline +# workflow for Cloud Dataproc using the Python client library. +# +# This script can be run on its own: +# python instantiate_inline_workflow_template.py ${PROJECT_ID} ${REGION} + + +import sys + +# [START dataproc_instantiate_inline_workflow_template] +from google.cloud import dataproc_v1 as dataproc + + +def instantiate_inline_workflow_template(project_id, region): + """This sample walks a user through submitting a workflow + for a Cloud Dataproc using the Python client library. + + Args: + project_id (string): Project to use for running the workflow. + region (string): Region where the workflow resources should live. + """ + + # Create a client with the endpoint set to the desired region. + workflow_template_client = dataproc.WorkflowTemplateServiceClient( + client_options={ + 'api_endpoint': f'{region}-dataproc.googleapis.com:443' + } + ) + + parent = workflow_template_client.region_path(project_id, region) + + template = { + 'jobs': [ + { + 'hadoop_job': { + 'main_jar_file_uri': 'file:///usr/lib/hadoop-mapreduce/' + 'hadoop-mapreduce-examples.jar', + 'args': [ + 'teragen', + '1000', + 'hdfs:///gen/' + ] + }, + 'step_id': 'teragen' + }, + { + 'hadoop_job': { + 'main_jar_file_uri': 'file:///usr/lib/hadoop-mapreduce/' + 'hadoop-mapreduce-examples.jar', + 'args': [ + 'terasort', + 'hdfs:///gen/', + 'hdfs:///sort/' + ] + }, + 'step_id': 'terasort', + 'prerequisite_step_ids': [ + 'teragen' + ] + }], + 'placement': { + 'managed_cluster': { + 'cluster_name': 'my-managed-cluster', + 'config': { + 'gce_cluster_config': { + # Leave 'zone_uri' empty for 'Auto Zone Placement' + # 'zone_uri': '' + 'zone_uri': 'us-central1-a' + } + } + } + } + } + + # Submit the request to instantiate the workflow from an inline template. + operation = workflow_template_client.instantiate_inline_workflow_template( + parent, template + ) + operation.result() + + # Output a success message. + print('Workflow ran successfully.') + # [END dataproc_instantiate_inline_workflow_template] + + +if __name__ == "__main__": + if len(sys.argv) < 3: + sys.exit('python instantiate_inline_workflow_template.py ' + + 'project_id region') + + project_id = sys.argv[1] + region = sys.argv[2] + instantiate_inline_workflow_template(project_id, region) diff --git a/dataproc/snippets/instantiate_inline_workflow_template_test.py b/dataproc/snippets/instantiate_inline_workflow_template_test.py new file mode 100644 index 000000000000..22673e4ee086 --- /dev/null +++ b/dataproc/snippets/instantiate_inline_workflow_template_test.py @@ -0,0 +1,31 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +import instantiate_inline_workflow_template + + +PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] +REGION = 'us-central1' + + +def test_workflows(capsys): + # Wrapper function for client library function + instantiate_inline_workflow_template.instantiate_inline_workflow_template( + PROJECT_ID, REGION + ) + + out, _ = capsys.readouterr() + assert "successfully" in out diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py new file mode 100644 index 000000000000..1639c4134685 --- /dev/null +++ b/dataproc/snippets/list_clusters.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Sample command-line program to list Cloud Dataproc clusters in a region. + +Example usage: +python list_clusters.py --project_id=my-project-id --region=global + +""" +import argparse + +from google.cloud import dataproc_v1 +from google.cloud.dataproc_v1.gapic.transports import ( + cluster_controller_grpc_transport) + + +# [START dataproc_list_clusters] +def list_clusters(dataproc, project, region): + """List the details of clusters in the region.""" + for cluster in dataproc.list_clusters(project, region): + print(('{} - {}'.format(cluster.cluster_name, + cluster.status.State.Name( + cluster.status.state)))) +# [END dataproc_list_clusters] + + +def main(project_id, region): + + if region == 'global': + # Use the default gRPC global endpoints. + dataproc_cluster_client = dataproc_v1.ClusterControllerClient() + else: + # Use a regional gRPC endpoint. See: + # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints + client_transport = ( + cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address='{}-dataproc.googleapis.com:443'.format(region))) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient( + client_transport) + + list_clusters(dataproc_cluster_client, project_id, region) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=( + argparse.RawDescriptionHelpFormatter)) + parser.add_argument( + '--project_id', help='Project ID to access.', required=True) + parser.add_argument( + '--region', help='Region of clusters to list.', required=True) + + args = parser.parse_args() + main(args.project_id, args.region) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py new file mode 100644 index 000000000000..ba55d7ce53ca --- /dev/null +++ b/dataproc/snippets/noxfile.py @@ -0,0 +1,224 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function + +import os +from pathlib import Path +import sys + +import nox + + +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING +# DO NOT EDIT THIS FILE EVER! +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING + +# Copy `noxfile_config.py` to your directory and modify it instead. + + +# `TEST_CONFIG` dict is a configuration hook that allows users to +# modify the test configurations. The values here should be in sync +# with `noxfile_config.py`. Users will copy `noxfile_config.py` into +# their directory and modify it. + +TEST_CONFIG = { + # You can opt out from the test for specific Python versions. + 'ignored_versions': ["2.7"], + + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + 'gcloud_project_env': 'GOOGLE_CLOUD_PROJECT', + # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', + + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + 'envs': {}, +} + + +try: + # Ensure we can import noxfile_config in the project's directory. + sys.path.append('.') + from noxfile_config import TEST_CONFIG_OVERRIDE +except ImportError as e: + print("No user noxfile_config found: detail: {}".format(e)) + TEST_CONFIG_OVERRIDE = {} + +# Update the TEST_CONFIG with the user supplied values. +TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) + + +def get_pytest_env_vars(): + """Returns a dict for pytest invocation.""" + ret = {} + + # Override the GCLOUD_PROJECT and the alias. + env_key = TEST_CONFIG['gcloud_project_env'] + # This should error out if not set. + ret['GOOGLE_CLOUD_PROJECT'] = os.environ[env_key] + + # Apply user supplied envs. + ret.update(TEST_CONFIG['envs']) + return ret + + +# DO NOT EDIT - automatically generated. +# All versions used to tested samples. +ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8"] + +# Any default versions that should be ignored. +IGNORED_VERSIONS = TEST_CONFIG['ignored_versions'] + +TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) + +INSTALL_LIBRARY_FROM_SOURCE = bool(os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False)) +# +# Style Checks +# + + +def _determine_local_import_names(start_dir): + """Determines all import names that should be considered "local". + + This is used when running the linter to insure that import order is + properly checked. + """ + file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] + return [ + basename + for basename, extension in file_ext_pairs + if extension == ".py" + or os.path.isdir(os.path.join(start_dir, basename)) + and basename not in ("__pycache__") + ] + + +# Linting with flake8. +# +# We ignore the following rules: +# E203: whitespace before ‘:’ +# E266: too many leading ‘#’ for block comment +# E501: line too long +# I202: Additional newline in a section of imports +# +# We also need to specify the rules which are ignored by default: +# ['E226', 'W504', 'E126', 'E123', 'W503', 'E24', 'E704', 'E121'] +FLAKE8_COMMON_ARGS = [ + "--show-source", + "--builtin=gettext", + "--max-complexity=20", + "--import-order-style=google", + "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", + "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", + "--max-line-length=88", +] + + +@nox.session +def lint(session): + session.install("flake8", "flake8-import-order") + + local_names = _determine_local_import_names(".") + args = FLAKE8_COMMON_ARGS + [ + "--application-import-names", + ",".join(local_names), + "." + ] + session.run("flake8", *args) + + +# +# Sample Tests +# + + +PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] + + +def _session_tests(session, post_install=None): + """Runs py.test for a particular project.""" + if os.path.exists("requirements.txt"): + session.install("-r", "requirements.txt") + + if os.path.exists("requirements-test.txt"): + session.install("-r", "requirements-test.txt") + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars() + ) + + +@nox.session(python=ALL_VERSIONS) +def py(session): + """Runs py.test for a sample using the specified version of Python.""" + if session.python in TESTED_VERSIONS: + _session_tests(session) + else: + session.skip("SKIPPED: {} tests are disabled for this sample.".format( + session.python + )) + + +# +# Readmegen +# + + +def _get_repo_root(): + """ Returns the root folder of the project. """ + # Get root of this repository. Assume we don't have directories nested deeper than 10 items. + p = Path(os.getcwd()) + for i in range(10): + if p is None: + break + if Path(p / ".git").exists(): + return str(p) + p = p.parent + raise Exception("Unable to detect repository root.") + + +GENERATED_READMES = sorted([x for x in Path(".").rglob("*.rst.in")]) + + +@nox.session +@nox.parametrize("path", GENERATED_READMES) +def readmegen(session, path): + """(Re-)generates the readme for a sample.""" + session.install("jinja2", "pyyaml") + dir_ = os.path.dirname(path) + + if os.path.exists(os.path.join(dir_, "requirements.txt")): + session.install("-r", os.path.join(dir_, "requirements.txt")) + + in_file = os.path.join(dir_, "README.rst.in") + session.run( + "python", _get_repo_root() + "/scripts/readme-gen/readme_gen.py", in_file + ) diff --git a/dataproc/snippets/pyspark_sort.py b/dataproc/snippets/pyspark_sort.py new file mode 100644 index 000000000000..0ce2350ad02b --- /dev/null +++ b/dataproc/snippets/pyspark_sort.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" Sample pyspark script to be uploaded to Cloud Storage and run on +Cloud Dataproc. + +Note this file is not intended to be run directly, but run inside a PySpark +environment. +""" + +# [START dataproc_pyspark_sort] +import pyspark + +sc = pyspark.SparkContext() +rdd = sc.parallelize(['Hello,', 'world!', 'dog', 'elephant', 'panther']) +words = sorted(rdd.collect()) +print(words) +# [END dataproc_pyspark_sort] diff --git a/dataproc/snippets/pyspark_sort_gcs.py b/dataproc/snippets/pyspark_sort_gcs.py new file mode 100644 index 000000000000..f1961c378d36 --- /dev/null +++ b/dataproc/snippets/pyspark_sort_gcs.py @@ -0,0 +1,30 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" Sample pyspark script to be uploaded to Cloud Storage and run on +Cloud Dataproc. + +Note this file is not intended to be run directly, but run inside a PySpark +environment. + +This file demonstrates how to read from a GCS bucket. See README.md for more +information. +""" + +# [START dataproc_pyspark_sort_gcs] +import pyspark + +sc = pyspark.SparkContext() +rdd = sc.textFile('gs://path-to-your-GCS-file') +print(sorted(rdd.collect())) +# [END dataproc_pyspark_sort_gcs] diff --git a/dataproc/snippets/python-api-walkthrough.md b/dataproc/snippets/python-api-walkthrough.md new file mode 100644 index 000000000000..1a8d436f7202 --- /dev/null +++ b/dataproc/snippets/python-api-walkthrough.md @@ -0,0 +1,170 @@ +# Use the Python Client Library to call Dataproc APIs + +Estimated completion time: + +## Overview + +This [Cloud Shell](https://cloud.google.com/shell/docs/) walkthrough leads you +through the steps to use the +[Google Cloud Client Libraries for Python](https://googleapis.github.io/google-cloud-python/latest/dataproc/index.html) +to programmatically interact with [Dataproc](https://cloud.google.com/dataproc/docs/). + +As you follow this walkthrough, you run Python code that calls +[Dataproc gRPC APIs](https://cloud.google.com/dataproc/docs/reference/rpc/) +to: + +* create a Dataproc cluster +* submit a small PySpark word sort job to run on the cluster +* get job status +* tear down the cluster after job completion + +## Using the walkthrough + +The `submit_job_to_cluster.py file` used in this walkthrough is opened in the +Cloud Shell editor when you launch the walkthrough. You can view +the code as your follow the walkthrough steps. + +**For more information**: See [Dataproc→Use the Python Client Library](https://cloud.google.com/dataproc/docs/tutorials/python-library-example) for +an explanation of how the code works. + +**To reload this walkthrough:** Run the following command from the +`~/python-docs-samples/dataproc` directory in Cloud Shell: + + cloudshell launch-tutorial python-api-walkthrough.md + +**To copy and run commands**: Click the "Paste in Cloud Shell" button + () + on the side of a code box, then press `Enter` to run the command. + +## Prerequisites (1) + +1. Create or select a Google Cloud Platform project to use for this tutorial. + * + +1. Click the link below to enable the Dataproc, Compute Engine, and Cloud Storage APIs + in a separate GCP console tab in your browser. + + **Note:** After you select your project and enable the APIs, return to this tutorial by clicking + on the **Cloud Shell** tab in your browser. + + * [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=dataproc,compute_component,storage-component.googleapis.com&redirect=https://console.cloud.google.com) + +## Prerequisites (2) + +1. This walkthrough uploads a PySpark file (`pyspark_sort.py`) to a + [Cloud Storage bucket](https://cloud.google.com/storage/docs/key-terms#buckets) in + your project. + * You can use the [Cloud Storage browser page](https://console.cloud.google.com/storage/browser) + in Google Cloud Platform Console to view existing buckets in your project. + +     **OR** + + * To create a new bucket, run the following command. Your bucket name must be unique. + ```bash + gsutil mb -p {{project-id}} gs://your-bucket-name + ``` + +1. Set environment variables. + + * Set the name of your bucket. + ```bash + BUCKET=your-bucket-name + ``` + +## Prerequisites (3) + +1. Set up a Python + [virtual environment](https://virtualenv.readthedocs.org/en/latest/) + in Cloud Shell. + + * Create the virtual environment. + ```bash + virtualenv ENV + ``` + * Activate the virtual environment. + ```bash + source ENV/bin/activate + ``` + +1. Install library dependencies in Cloud Shell. + ```bash + pip install -r requirements.txt + ``` + +## Create a cluster and submit a job + +1. Set a name for your new cluster. + ```bash + CLUSTER=new-cluster-name + ``` + +1. Set a [zone](https://cloud.google.com/compute/docs/regions-zones/#available) + where your new cluster will be located. You can change the + "us-central1-a" zone that is pre-set in the following command. + ```bash + ZONE=us-central1-a + ``` + +1. Run `submit_job.py` with the `--create_new_cluster` flag + to create a new cluster and submit the `pyspark_sort.py` job + to the cluster. + + ```bash + python submit_job_to_cluster.py \ + --project_id={{project-id}} \ + --cluster_name=$CLUSTER \ + --zone=$ZONE \ + --gcs_bucket=$BUCKET \ + --create_new_cluster + ``` + +## Job Output + +Job output in Cloud Shell shows cluster creation, job submission, + job completion, and then tear-down of the cluster. + + ... + Creating cluster... + Cluster created. + Uploading pyspark file to Cloud Storage. + new-cluster-name - RUNNING + Submitted job ID ... + Waiting for job to finish... + Job finished. + Downloading output file + ..... + ['Hello,', 'dog', 'elephant', 'panther', 'world!'] + ... + Tearing down cluster + ``` +## Congratulations on Completing the Walkthrough! + + +--- + +### Next Steps: + +* **View job details from the Console.** View job details by selecting the + PySpark job from the Dataproc += + [Jobs page](https://console.cloud.google.com/dataproc/jobs) + in the Google Cloud Platform Console. + +* **Delete resources used in the walkthrough.** + The `submit_job_to_cluster.py` job deletes the cluster that it created for this + walkthrough. + + If you created a bucket to use for this walkthrough, + you can run the following command to delete the + Cloud Storage bucket (the bucket must be empty). + ```bash + gsutil rb gs://$BUCKET + ``` + You can run the following command to delete the bucket **and all + objects within it. Note: the deleted objects cannot be recovered.** + ```bash + gsutil rm -r gs://$BUCKET + ``` + +* **For more information.** See the [Dataproc documentation](https://cloud.google.com/dataproc/docs/) + for API reference and product feature information. diff --git a/dataproc/snippets/quickstart/quickstart.py b/dataproc/snippets/quickstart/quickstart.py new file mode 100644 index 000000000000..4159e2815202 --- /dev/null +++ b/dataproc/snippets/quickstart/quickstart.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python + +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START dataproc_quickstart] +""" +This quickstart sample walks a user through creating a Cloud Dataproc +cluster, submitting a PySpark job from Google Cloud Storage to the +cluster, reading the output of the job and deleting the cluster, all +using the Python client library. + +Usage: + python quickstart.py --project_id --region \ + --cluster_name --job_file_path +""" + +import argparse +import time + +from google.cloud import dataproc_v1 as dataproc +from google.cloud import storage + + +def quickstart(project_id, region, cluster_name, job_file_path): + # Create the cluster client. + cluster_client = dataproc.ClusterControllerClient(client_options={ + 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) + }) + + # Create the cluster config. + cluster = { + 'project_id': project_id, + 'cluster_name': cluster_name, + 'config': { + 'master_config': { + 'num_instances': 1, + 'machine_type_uri': 'n1-standard-1' + }, + 'worker_config': { + 'num_instances': 2, + 'machine_type_uri': 'n1-standard-1' + } + } + } + + # Create the cluster. + operation = cluster_client.create_cluster(project_id, region, cluster) + result = operation.result() + + print('Cluster created successfully: {}'.format(result.cluster_name)) + + # Create the job client. + job_client = dataproc.JobControllerClient(client_options={ + 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) + }) + + # Create the job config. + job = { + 'placement': { + 'cluster_name': cluster_name + }, + 'pyspark_job': { + 'main_python_file_uri': job_file_path + } + } + + job_response = job_client.submit_job(project_id, region, job) + job_id = job_response.reference.job_id + + print('Submitted job \"{}\".'.format(job_id)) + + # Termimal states for a job. + terminal_states = { + dataproc.types.JobStatus.ERROR, + dataproc.types.JobStatus.CANCELLED, + dataproc.types.JobStatus.DONE + } + + # Create a timeout such that the job gets cancelled if not in a + # terminal state after a fixed period of time. + timeout_seconds = 600 + time_start = time.time() + + # Wait for the job to complete. + while job_response.status.state not in terminal_states: + if time.time() > time_start + timeout_seconds: + job_client.cancel_job(project_id, region, job_id) + print('Job {} timed out after threshold of {} seconds.'.format( + job_id, timeout_seconds)) + + # Poll for job termination once a second. + time.sleep(1) + job_response = job_client.get_job(project_id, region, job_id) + + # Cloud Dataproc job output gets saved to a GCS bucket allocated to it. + cluster_info = cluster_client.get_cluster( + project_id, region, cluster_name) + + storage_client = storage.Client() + bucket = storage_client.get_bucket(cluster_info.config.config_bucket) + output_blob = ( + 'google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000' + .format(cluster_info.cluster_uuid, job_id)) + output = bucket.blob(output_blob).download_as_string() + + print('Job {} finished with state {}:\n{}'.format( + job_id, + job_response.status.State.Name(job_response.status.state), + output)) + + # Delete the cluster once the job has terminated. + operation = cluster_client.delete_cluster(project_id, region, cluster_name) + operation.result() + + print('Cluster {} successfully deleted.'.format(cluster_name)) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + '--project_id', + type=str, + required=True, + help='Project to use for creating resources.') + parser.add_argument( + '--region', + type=str, + required=True, + help='Region where the resources should live.') + parser.add_argument( + '--cluster_name', + type=str, + required=True, + help='Name to use for creating a cluster.') + parser.add_argument( + '--job_file_path', + type=str, + required=True, + help='Job in GCS to execute against the cluster.') + + args = parser.parse_args() + quickstart(args.project_id, args.region, + args.cluster_name, args.job_file_path) +# [END dataproc_quickstart] diff --git a/dataproc/snippets/quickstart/quickstart_test.py b/dataproc/snippets/quickstart/quickstart_test.py new file mode 100644 index 000000000000..3e17f6fa3e57 --- /dev/null +++ b/dataproc/snippets/quickstart/quickstart_test.py @@ -0,0 +1,71 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import uuid + +from google.cloud import dataproc_v1 as dataproc +from google.cloud import storage +import pytest + +import quickstart + + +PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] +REGION = 'us-central1' +CLUSTER_NAME = 'py-qs-test-{}'.format(str(uuid.uuid4())) +STAGING_BUCKET = 'py-dataproc-qs-bucket-{}'.format(str(uuid.uuid4())) +JOB_FILE_NAME = 'sum.py' +JOB_FILE_PATH = 'gs://{}/{}'.format(STAGING_BUCKET, JOB_FILE_NAME) +SORT_CODE = ( + "import pyspark\n" + "sc = pyspark.SparkContext()\n" + "rdd = sc.parallelize((1,2,3,4,5))\n" + "sum = rdd.reduce(lambda x, y: x + y)\n" +) + + +@pytest.fixture(autouse=True) +def setup_teardown(): + storage_client = storage.Client() + bucket = storage_client.create_bucket(STAGING_BUCKET) + blob = bucket.blob(JOB_FILE_NAME) + blob.upload_from_string(SORT_CODE) + + yield + + cluster_client = dataproc.ClusterControllerClient(client_options={ + 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(REGION) + }) + + # The quickstart sample deletes the cluster, but if the test fails + # before cluster deletion occurs, it can be manually deleted here. + clusters = cluster_client.list_clusters(PROJECT_ID, REGION) + + for cluster in clusters: + if cluster.cluster_name == CLUSTER_NAME: + cluster_client.delete_cluster(PROJECT_ID, REGION, CLUSTER_NAME) + + blob.delete() + bucket.delete() + + +def test_quickstart(capsys): + quickstart.quickstart(PROJECT_ID, REGION, CLUSTER_NAME, JOB_FILE_PATH) + out, _ = capsys.readouterr() + + assert 'Cluster created successfully' in out + assert 'Submitted job' in out + assert 'finished with state DONE:' in out + assert 'successfully deleted' in out diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt new file mode 100644 index 000000000000..7e460c8c866e --- /dev/null +++ b/dataproc/snippets/requirements-test.txt @@ -0,0 +1 @@ +pytest==6.0.1 diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt new file mode 100644 index 000000000000..ebc1d8813435 --- /dev/null +++ b/dataproc/snippets/requirements.txt @@ -0,0 +1,6 @@ +grpcio==1.31.0 +google-auth==1.20.1 +google-auth-httplib2==0.0.4 +google-cloud==0.34.0 +google-cloud-storage==1.29.0 +google-cloud-dataproc==1.0.1 diff --git a/dataproc/snippets/single_job_workflow.py b/dataproc/snippets/single_job_workflow.py new file mode 100644 index 000000000000..b2754b06c1ec --- /dev/null +++ b/dataproc/snippets/single_job_workflow.py @@ -0,0 +1,209 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +r"""Sample Cloud Dataproc inline workflow to run a pyspark job on an ephermeral +cluster. +Example Usage to run the inline workflow on a managed cluster: +python single_job_workflow.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ + --cluster_name=$CLUSTER --zone=$ZONE +Example Usage to run the inline workflow on a global region managed cluster: +python submit_job_to_cluster.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ + --cluster_name=$CLUSTER --zone=$ZONE --global_region +""" + +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import argparse +import os + +from google.cloud import dataproc_v1 +from google.cloud import storage +from google.cloud.dataproc_v1.gapic.transports import ( + workflow_template_service_grpc_transport) + + +DEFAULT_FILENAME = "pyspark_sort.py" +waiting_callback = False + + +def get_pyspark_file(pyspark_file=None): + if pyspark_file: + f = open(pyspark_file, "rb") + return f, os.path.basename(pyspark_file) + else: + """Gets the PySpark file from current directory.""" + current_dir = os.path.dirname(os.path.abspath(__file__)) + f = open(os.path.join(current_dir, DEFAULT_FILENAME), "rb") + return f, DEFAULT_FILENAME + + +def get_region_from_zone(zone): + try: + region_as_list = zone.split("-")[:-1] + return "-".join(region_as_list) + except (AttributeError, IndexError, ValueError): + raise ValueError("Invalid zone provided, please check your input.") + + +def upload_pyspark_file(project, bucket_name, filename, spark_file): + """Uploads the PySpark file in this directory to the configured input + bucket.""" + print("Uploading pyspark file to Cloud Storage.") + client = storage.Client(project=project) + bucket = client.get_bucket(bucket_name) + blob = bucket.blob(filename) + blob.upload_from_file(spark_file) + + +def run_workflow(dataproc, project, region, zone, bucket_name, filename, + cluster_name): + + parent = "projects/{}/regions/{}".format(project, region) + zone_uri = ("https://www.googleapis.com/compute/v1/projects/{}/zones/{}" + .format(project, zone)) + + workflow_data = { + "placement": { + "managed_cluster": { + "cluster_name": cluster_name, + "config": { + "gce_cluster_config": {"zone_uri": zone_uri}, + "master_config": { + "num_instances": 1, + "machine_type_uri": "n1-standard-1", + }, + "worker_config": { + "num_instances": 2, + "machine_type_uri": "n1-standard-1", + }, + }, + } + }, + "jobs": [ + { + "pyspark_job": { + "main_python_file_uri": "gs://{}/{}".format( + bucket_name, filename) + }, + "step_id": "pyspark-job", + } + ], + } + + workflow = dataproc.instantiate_inline_workflow_template(parent, + workflow_data) + + workflow.add_done_callback(callback) + global waiting_callback + waiting_callback = True + + +def callback(operation_future): + # Reset global when callback returns. + global waiting_callback + waiting_callback = False + + +def wait_for_workflow_end(): + """Wait for cluster creation.""" + print("Waiting for workflow completion ...") + print("Workflow and job progress, and job driver output available from: " + "https://console.cloud.google.com/dataproc/workflows/") + + while True: + if not waiting_callback: + print("Workflow completed.") + break + + +def main( + project_id, + zone, + cluster_name, + bucket_name, + pyspark_file=None, + create_new_cluster=True, + global_region=True, +): + + # [START dataproc_get_workflow_template_client] + if global_region: + region = "global" + # Use the default gRPC global endpoints. + dataproc_workflow_client = dataproc_v1.WorkflowTemplateServiceClient() + else: + region = get_region_from_zone(zone) + # Use a regional gRPC endpoint. See: + # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints + client_transport = (workflow_template_service_grpc_transport + .WorkflowTemplateServiceGrpcTransport( + address="{}-dataproc.googleapis.com:443" + .format(region))) + dataproc_workflow_client = dataproc_v1.WorkflowTemplateServiceClient( + client_transport + ) + # [END dataproc_get_workflow_template_client] + + try: + spark_file, spark_filename = get_pyspark_file(pyspark_file) + upload_pyspark_file(project_id, bucket_name, spark_filename, + spark_file) + + run_workflow( + dataproc_workflow_client, + project_id, + region, + zone, + bucket_name, + spark_filename, + cluster_name + ) + wait_for_workflow_end() + + finally: + spark_file.close() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=(argparse + .RawDescriptionHelpFormatter)) + parser.add_argument( + "--project_id", help="Project ID you want to access.", required=True + ) + parser.add_argument( + "--zone", help="Zone to create clusters in/connect to", required=True + ) + parser.add_argument( + "--cluster_name", help="Name of the cluster to create/connect to", + required=True + ) + parser.add_argument( + "--gcs_bucket", help="Bucket to upload Pyspark file to", required=True + ) + parser.add_argument( + "--pyspark_file", help="Pyspark filename. Defaults to pyspark_sort.py" + ) + parser.add_argument("--global_region", + action="store_true", + help="If cluster is in the global region") + + args = parser.parse_args() + main( + args.project_id, + args.zone, + args.cluster_name, + args.gcs_bucket, + args.pyspark_file, + ) diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py new file mode 100644 index 000000000000..389cbec87aa3 --- /dev/null +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -0,0 +1,288 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +r"""Sample command-line program to run a pyspark job on a new or existing +cluster. + +Global region clusters are supported with --global_region flag. + +Example Usage to run the pyspark job on a new cluster: +python submit_job_to_cluster.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ + --create_new_cluster --cluster_name=$CLUSTER --zone=$ZONE + +Example Usage to run the pyspark job on an existing global region cluster: +python submit_job_to_cluster.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ + --global_region --cluster_name=$CLUSTER --zone=$ZONE + +""" + +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import argparse +import os + +from google.cloud import dataproc_v1 +from google.cloud import storage +from google.cloud.dataproc_v1.gapic.transports import ( + cluster_controller_grpc_transport) +from google.cloud.dataproc_v1.gapic.transports import ( + job_controller_grpc_transport) + + +DEFAULT_FILENAME = 'pyspark_sort.py' +waiting_callback = False + + +def get_pyspark_file(pyspark_file=None): + if pyspark_file: + f = open(pyspark_file, "rb") + return f, os.path.basename(pyspark_file) + else: + """Gets the PySpark file from current directory.""" + current_dir = os.path.dirname(os.path.abspath(__file__)) + f = open(os.path.join(current_dir, DEFAULT_FILENAME), "rb") + return f, DEFAULT_FILENAME + + +def get_region_from_zone(zone): + try: + region_as_list = zone.split('-')[:-1] + return '-'.join(region_as_list) + except (AttributeError, IndexError, ValueError): + raise ValueError('Invalid zone provided, please check your input.') + + +def upload_pyspark_file(project, bucket_name, filename, spark_file): + """Uploads the PySpark file in this directory to the configured input + bucket.""" + print('Uploading pyspark file to Cloud Storage.') + client = storage.Client(project=project) + bucket = client.get_bucket(bucket_name) + blob = bucket.blob(filename) + blob.upload_from_file(spark_file) + + +def download_output(project, cluster_id, output_bucket, job_id): + """Downloads the output file from Cloud Storage and returns it as a + string.""" + print('Downloading output file.') + client = storage.Client(project=project) + bucket = client.get_bucket(output_bucket) + output_blob = ( + ('google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000'. + format(cluster_id, job_id))) + return bucket.blob(output_blob).download_as_string() + + +# [START dataproc_create_cluster] +def create_cluster(dataproc, project, zone, region, cluster_name): + """Create the cluster.""" + print('Creating cluster...') + zone_uri = \ + 'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format( + project, zone) + cluster_data = { + 'project_id': project, + 'cluster_name': cluster_name, + 'config': { + 'gce_cluster_config': { + 'zone_uri': zone_uri + }, + 'master_config': { + 'num_instances': 1, + 'machine_type_uri': 'n1-standard-1' + }, + 'worker_config': { + 'num_instances': 2, + 'machine_type_uri': 'n1-standard-1' + } + } + } + + cluster = dataproc.create_cluster(project, region, cluster_data) + cluster.add_done_callback(callback) + global waiting_callback + waiting_callback = True +# [END dataproc_create_cluster] + + +def callback(operation_future): + # Reset global when callback returns. + global waiting_callback + waiting_callback = False + + +def wait_for_cluster_creation(): + """Wait for cluster creation.""" + print('Waiting for cluster creation...') + + while True: + if not waiting_callback: + print("Cluster created.") + break + + +# [START dataproc_list_clusters_with_detail] +def list_clusters_with_details(dataproc, project, region): + """List the details of clusters in the region.""" + for cluster in dataproc.list_clusters(project, region): + print(('{} - {}'.format(cluster.cluster_name, + cluster.status.State.Name( + cluster.status.state)))) +# [END dataproc_list_clusters_with_detail] + + +def get_cluster_id_by_name(dataproc, project_id, region, cluster_name): + """Helper function to retrieve the ID and output bucket of a cluster by + name.""" + for cluster in dataproc.list_clusters(project_id, region): + if cluster.cluster_name == cluster_name: + return cluster.cluster_uuid, cluster.config.config_bucket + + +# [START dataproc_submit_pyspark_job] +def submit_pyspark_job(dataproc, project, region, cluster_name, bucket_name, + filename): + """Submit the Pyspark job to the cluster (assumes `filename` was uploaded + to `bucket_name.""" + job_details = { + 'placement': { + 'cluster_name': cluster_name + }, + 'pyspark_job': { + 'main_python_file_uri': 'gs://{}/{}'.format(bucket_name, filename) + } + } + + result = dataproc.submit_job( + project_id=project, region=region, job=job_details) + job_id = result.reference.job_id + print('Submitted job ID {}.'.format(job_id)) + return job_id +# [END dataproc_submit_pyspark_job] + + +# [START dataproc_delete] +def delete_cluster(dataproc, project, region, cluster): + """Delete the cluster.""" + print('Tearing down cluster.') + result = dataproc.delete_cluster( + project_id=project, region=region, cluster_name=cluster) + return result +# [END dataproc_delete] + + +# [START dataproc_wait] +def wait_for_job(dataproc, project, region, job_id): + """Wait for job to complete or error out.""" + print('Waiting for job to finish...') + while True: + job = dataproc.get_job(project, region, job_id) + # Handle exceptions + if job.status.State.Name(job.status.state) == 'ERROR': + raise Exception(job.status.details) + elif job.status.State.Name(job.status.state) == 'DONE': + print('Job finished.') + return job +# [END dataproc_wait] + + +def main(project_id, + zone, + cluster_name, + bucket_name, + pyspark_file=None, + create_new_cluster=True, + global_region=True): + + # [START dataproc_get_client] + if global_region: + region = 'global' + # Use the default gRPC global endpoints. + dataproc_cluster_client = dataproc_v1.ClusterControllerClient() + dataproc_job_client = dataproc_v1.JobControllerClient() + else: + region = get_region_from_zone(zone) + # Use a regional gRPC endpoint. See: + # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints + client_transport = ( + cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address='{}-dataproc.googleapis.com:443'.format(region))) + job_transport = ( + job_controller_grpc_transport.JobControllerGrpcTransport( + address='{}-dataproc.googleapis.com:443'.format(region))) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient( + client_transport) + dataproc_job_client = dataproc_v1.JobControllerClient(job_transport) + # [END dataproc_get_client] + + try: + spark_file, spark_filename = get_pyspark_file(pyspark_file) + if create_new_cluster: + create_cluster(dataproc_cluster_client, project_id, zone, region, + cluster_name) + wait_for_cluster_creation() + upload_pyspark_file(project_id, bucket_name, spark_filename, + spark_file) + + list_clusters_with_details(dataproc_cluster_client, project_id, + region) + + (cluster_id, output_bucket) = ( + get_cluster_id_by_name(dataproc_cluster_client, project_id, + region, cluster_name)) + + # [START dataproc_call_submit_pyspark_job] + job_id = submit_pyspark_job(dataproc_job_client, project_id, region, + cluster_name, bucket_name, spark_filename) + # [END dataproc_call_submit_pyspark_job] + + wait_for_job(dataproc_job_client, project_id, region, job_id) + output = download_output(project_id, cluster_id, output_bucket, job_id) + print('Received job output {}'.format(output)) + return output + finally: + if create_new_cluster: + delete_cluster(dataproc_cluster_client, project_id, region, + cluster_name) + spark_file.close() + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse. + RawDescriptionHelpFormatter) + parser.add_argument( + '--project_id', help='Project ID you want to access.', required=True) + parser.add_argument('--zone', + help='Zone to create clusters in/connect to', + required=True) + parser.add_argument('--cluster_name', + help='Name of the cluster to create/connect to', + required=True) + parser.add_argument('--gcs_bucket', + help='Bucket to upload Pyspark file to', + required=True) + parser.add_argument('--pyspark_file', + help='Pyspark filename. Defaults to pyspark_sort.py') + parser.add_argument('--create_new_cluster', + action='store_true', + help='States if the cluster should be created') + parser.add_argument('--global_region', + action='store_true', + help='If cluster is in the global region') + + args = parser.parse_args() + main(args.project_id, args.zone, args.cluster_name, args.gcs_bucket, + args.pyspark_file, args.create_new_cluster, args.global_region) From 3c57ee2bf7f5ed3b88c0d17041aff324940e41fb Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 8 Aug 2020 02:46:02 +0200 Subject: [PATCH 002/158] chore(deps): update dependency google-cloud-storage to v1.30.0 (#68) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | minor | `==1.29.0` -> `==1.30.0` | --- ### Release Notes
googleapis/python-storage ### [`v1.30.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1300-httpswwwgithubcomgoogleapispython-storagecomparev1290v1300-2020-07-24) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.29.0...v1.30.0) ##### Features - add timeouts to Blob methods where missing ([#​185](https://www.github.com/googleapis/python-storage/issues/185)) ([6eeb855](https://www.github.com/googleapis/python-storage/commit/6eeb855aa0e6a7835d1d4f6e72951e43af22ab57)) - auto-populate standard headers for non-chunked downloads ([#​204](https://www.github.com/googleapis/python-storage/issues/204)) ([d8432cd](https://www.github.com/googleapis/python-storage/commit/d8432cd65a4e9b38eebd1ade2ff00f2f44bb0ef6)), closes [#​24](https://www.github.com/googleapis/python-storage/issues/24) - migrate to Service Account Credentials API ([#​189](https://www.github.com/googleapis/python-storage/issues/189)) ([e4990d0](https://www.github.com/googleapis/python-storage/commit/e4990d06043dbd8d1a417f3a1a67fe8746071f1c)) ##### Bug Fixes - add multiprocessing.rst to synthool excludes ([#​186](https://www.github.com/googleapis/python-storage/issues/186)) ([4d76e38](https://www.github.com/googleapis/python-storage/commit/4d76e3882210ed2818a43256265f6df31348d410)) ##### Documentation - fix indent in code blocks ([#​171](https://www.github.com/googleapis/python-storage/issues/171)) ([62d1543](https://www.github.com/googleapis/python-storage/commit/62d1543e18040b286b23464562aa6eb998074c54)), closes [#​170](https://www.github.com/googleapis/python-storage/issues/170) - remove doubled word in docstring ([#​209](https://www.github.com/googleapis/python-storage/issues/209)) ([7a4e7a5](https://www.github.com/googleapis/python-storage/commit/7a4e7a5974abedb0b7b2e110cacbfcd0a40346b6)) ##### Documentation - fix indent in code blocks ([#​171](https://www.github.com/googleapis/python-storage/issues/171)) ([62d1543](https://www.github.com/googleapis/python-storage/commit/62d1543e18040b286b23464562aa6eb998074c54)), closes [#​170](https://www.github.com/googleapis/python-storage/issues/170) - remove doubled word in docstring ([#​209](https://www.github.com/googleapis/python-storage/issues/209)) ([7a4e7a5](https://www.github.com/googleapis/python-storage/commit/7a4e7a5974abedb0b7b2e110cacbfcd0a40346b6)) ##### Dependencies - prep for grmp-1.0.0 release ([#​211](https://www.github.com/googleapis/python-storage/issues/211)) ([55bae9a](https://www.github.com/googleapis/python-storage/commit/55bae9a0e7c0db512c10c6b3b621cd2ef05c9729))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index ebc1d8813435..bd6abe878125 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.31.0 google-auth==1.20.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.29.0 +google-cloud-storage==1.30.0 google-cloud-dataproc==1.0.1 From 4c4563b028ccf0eaf64f34150e1b0e826ea6ea2c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 8 Aug 2020 02:54:10 +0200 Subject: [PATCH 003/158] chore(deps): update dependency google-cloud-dataproc to v1.1.0 (#67) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | minor | `==1.0.1` -> `==1.1.0` | --- ### Release Notes
googleapis/python-dataproc ### [`v1.1.0`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#​110-httpswwwgithubcomgoogleapispython-dataproccomparev101v110-2020-07-31) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v1.0.1...v1.1.0) ##### Features - add support for temp_bucket, endpoint_config in clusters; add preemptibility for instance group configs ([#​60](https://www.github.com/googleapis/python-dataproc/issues/60)) ([a80fc72](https://www.github.com/googleapis/python-dataproc/commit/a80fc727510c10c678caa125902c201c8280dcc1)) ##### [1.0.1](https://www.github.com/googleapis/python-dataproc/compare/v1.0.0...v1.0.1) (2020-07-16) ##### Bug Fixes - correct protobuf type for diagnose_cluster, update retry configs ([#​55](https://www.github.com/googleapis/python-dataproc/issues/55)) ([822315e](https://www.github.com/googleapis/python-dataproc/commit/822315ec3f2517ebb6ca199b72156ebd50e0518b))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index bd6abe878125..c2c958754ddc 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -3,4 +3,4 @@ google-auth==1.20.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.30.0 -google-cloud-dataproc==1.0.1 +google-cloud-dataproc==1.1.0 From 162b8526332d18c022c75a7731b5ddd5c046b7dc Mon Sep 17 00:00:00 2001 From: arithmetic1728 <58957152+arithmetic1728@users.noreply.github.com> Date: Mon, 10 Aug 2020 16:05:32 -0700 Subject: [PATCH 004/158] feat!: migrate to use microgen (#71) * feat!: migrate to use microgen * update * update * update * update --- dataproc/snippets/create_cluster.py | 32 +-- dataproc/snippets/create_cluster_test.py | 20 +- dataproc/snippets/dataproc_e2e_donttest.py | 11 +- .../instantiate_inline_workflow_template.py | 68 ++--- ...stantiate_inline_workflow_template_test.py | 4 +- dataproc/snippets/list_clusters.py | 43 +-- dataproc/snippets/pyspark_sort.py | 2 +- dataproc/snippets/pyspark_sort_gcs.py | 2 +- dataproc/snippets/quickstart/quickstart.py | 129 +++++---- .../snippets/quickstart/quickstart_test.py | 38 +-- dataproc/snippets/single_job_workflow.py | 52 ++-- dataproc/snippets/submit_job_to_cluster.py | 248 ++++++++++-------- 12 files changed, 350 insertions(+), 299 deletions(-) diff --git a/dataproc/snippets/create_cluster.py b/dataproc/snippets/create_cluster.py index b4d63d2e13f5..732b349ae03e 100644 --- a/dataproc/snippets/create_cluster.py +++ b/dataproc/snippets/create_cluster.py @@ -38,38 +38,34 @@ def create_cluster(project_id, region, cluster_name): """ # Create a client with the endpoint set to the desired cluster region. - cluster_client = dataproc.ClusterControllerClient(client_options={ - 'api_endpoint': f'{region}-dataproc.googleapis.com:443', - }) + cluster_client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} + ) # Create the cluster config. cluster = { - 'project_id': project_id, - 'cluster_name': cluster_name, - 'config': { - 'master_config': { - 'num_instances': 1, - 'machine_type_uri': 'n1-standard-1' - }, - 'worker_config': { - 'num_instances': 2, - 'machine_type_uri': 'n1-standard-1' - } - } + "project_id": project_id, + "cluster_name": cluster_name, + "config": { + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + }, } # Create the cluster. - operation = cluster_client.create_cluster(project_id, region, cluster) + operation = cluster_client.create_cluster( + request={"project_id": project_id, "region": region, "cluster": cluster} + ) result = operation.result() # Output a success message. - print(f'Cluster created successfully: {result.cluster_name}') + print(f"Cluster created successfully: {result.cluster_name}") # [END dataproc_create_cluster] if __name__ == "__main__": if len(sys.argv) < 4: - sys.exit('python create_cluster.py project_id region cluster_name') + sys.exit("python create_cluster.py project_id region cluster_name") project_id = sys.argv[1] region = sys.argv[2] diff --git a/dataproc/snippets/create_cluster_test.py b/dataproc/snippets/create_cluster_test.py index 6b1d6806100e..fd391c3e0abd 100644 --- a/dataproc/snippets/create_cluster_test.py +++ b/dataproc/snippets/create_cluster_test.py @@ -21,20 +21,26 @@ import create_cluster -PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] -REGION = 'us-central1' -CLUSTER_NAME = 'py-cc-test-{}'.format(str(uuid.uuid4())) +PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"] +REGION = "us-central1" +CLUSTER_NAME = "py-cc-test-{}".format(str(uuid.uuid4())) @pytest.fixture(autouse=True) def teardown(): yield - cluster_client = dataproc.ClusterControllerClient(client_options={ - 'api_endpoint': f'{REGION}-dataproc.googleapis.com:443' - }) + cluster_client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": f"{REGION}-dataproc.googleapis.com:443"} + ) # Client library function - operation = cluster_client.delete_cluster(PROJECT_ID, REGION, CLUSTER_NAME) + operation = cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) # Wait for cluster to delete operation.result() diff --git a/dataproc/snippets/dataproc_e2e_donttest.py b/dataproc/snippets/dataproc_e2e_donttest.py index 44cc03bfd428..a92f46065f25 100644 --- a/dataproc/snippets/dataproc_e2e_donttest.py +++ b/dataproc/snippets/dataproc_e2e_donttest.py @@ -20,13 +20,12 @@ import submit_job_to_cluster -PROJECT = os.environ['GOOGLE_CLOUD_PROJECT'] -BUCKET = os.environ['CLOUD_STORAGE_BUCKET'] -CLUSTER_NAME = 'testcluster3' -ZONE = 'us-central1-b' +PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] +BUCKET = os.environ["CLOUD_STORAGE_BUCKET"] +CLUSTER_NAME = "testcluster3" +ZONE = "us-central1-b" def test_e2e(): - output = submit_job_to_cluster.main( - PROJECT, ZONE, CLUSTER_NAME, BUCKET) + output = submit_job_to_cluster.main(PROJECT, ZONE, CLUSTER_NAME, BUCKET) assert b"['Hello,', 'dog', 'elephant', 'panther', 'world!']" in output diff --git a/dataproc/snippets/instantiate_inline_workflow_template.py b/dataproc/snippets/instantiate_inline_workflow_template.py index f9358376f9f9..b3a40d133544 100644 --- a/dataproc/snippets/instantiate_inline_workflow_template.py +++ b/dataproc/snippets/instantiate_inline_workflow_template.py @@ -36,71 +36,61 @@ def instantiate_inline_workflow_template(project_id, region): # Create a client with the endpoint set to the desired region. workflow_template_client = dataproc.WorkflowTemplateServiceClient( - client_options={ - 'api_endpoint': f'{region}-dataproc.googleapis.com:443' - } + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} ) - parent = workflow_template_client.region_path(project_id, region) + parent = "projects/{}/regions/{}".format(project_id, region) template = { - 'jobs': [ + "jobs": [ { - 'hadoop_job': { - 'main_jar_file_uri': 'file:///usr/lib/hadoop-mapreduce/' - 'hadoop-mapreduce-examples.jar', - 'args': [ - 'teragen', - '1000', - 'hdfs:///gen/' - ] + "hadoop_job": { + "main_jar_file_uri": "file:///usr/lib/hadoop-mapreduce/" + "hadoop-mapreduce-examples.jar", + "args": ["teragen", "1000", "hdfs:///gen/"], }, - 'step_id': 'teragen' + "step_id": "teragen", }, { - 'hadoop_job': { - 'main_jar_file_uri': 'file:///usr/lib/hadoop-mapreduce/' - 'hadoop-mapreduce-examples.jar', - 'args': [ - 'terasort', - 'hdfs:///gen/', - 'hdfs:///sort/' - ] + "hadoop_job": { + "main_jar_file_uri": "file:///usr/lib/hadoop-mapreduce/" + "hadoop-mapreduce-examples.jar", + "args": ["terasort", "hdfs:///gen/", "hdfs:///sort/"], }, - 'step_id': 'terasort', - 'prerequisite_step_ids': [ - 'teragen' - ] - }], - 'placement': { - 'managed_cluster': { - 'cluster_name': 'my-managed-cluster', - 'config': { - 'gce_cluster_config': { + "step_id": "terasort", + "prerequisite_step_ids": ["teragen"], + }, + ], + "placement": { + "managed_cluster": { + "cluster_name": "my-managed-cluster", + "config": { + "gce_cluster_config": { # Leave 'zone_uri' empty for 'Auto Zone Placement' # 'zone_uri': '' - 'zone_uri': 'us-central1-a' + "zone_uri": "us-central1-a" } - } + }, } - } + }, } # Submit the request to instantiate the workflow from an inline template. operation = workflow_template_client.instantiate_inline_workflow_template( - parent, template + request={"parent": parent, "template": template} ) operation.result() # Output a success message. - print('Workflow ran successfully.') + print("Workflow ran successfully.") # [END dataproc_instantiate_inline_workflow_template] if __name__ == "__main__": if len(sys.argv) < 3: - sys.exit('python instantiate_inline_workflow_template.py ' - + 'project_id region') + sys.exit( + "python instantiate_inline_workflow_template.py " + "project_id region" + ) project_id = sys.argv[1] region = sys.argv[2] diff --git a/dataproc/snippets/instantiate_inline_workflow_template_test.py b/dataproc/snippets/instantiate_inline_workflow_template_test.py index 22673e4ee086..ef4f31a564c9 100644 --- a/dataproc/snippets/instantiate_inline_workflow_template_test.py +++ b/dataproc/snippets/instantiate_inline_workflow_template_test.py @@ -17,8 +17,8 @@ import instantiate_inline_workflow_template -PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] -REGION = 'us-central1' +PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"] +REGION = "us-central1" def test_workflows(capsys): diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py index 1639c4134685..f0e7bac348e8 100644 --- a/dataproc/snippets/list_clusters.py +++ b/dataproc/snippets/list_clusters.py @@ -19,45 +19,50 @@ import argparse from google.cloud import dataproc_v1 -from google.cloud.dataproc_v1.gapic.transports import ( - cluster_controller_grpc_transport) +from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport # [START dataproc_list_clusters] def list_clusters(dataproc, project, region): """List the details of clusters in the region.""" - for cluster in dataproc.list_clusters(project, region): - print(('{} - {}'.format(cluster.cluster_name, - cluster.status.State.Name( - cluster.status.state)))) + for cluster in dataproc.list_clusters( + request={"project_id": project, "region": region} + ): + print( + ( + "{} - {}".format( + cluster.cluster_name, + cluster.status.State.Name(cluster.status.state), + ) + ) + ) + + # [END dataproc_list_clusters] def main(project_id, region): - if region == 'global': + if region == "global": # Use the default gRPC global endpoints. dataproc_cluster_client = dataproc_v1.ClusterControllerClient() else: # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = ( - cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address='{}-dataproc.googleapis.com:443'.format(region))) - dataproc_cluster_client = dataproc_v1.ClusterControllerClient( - client_transport) + client_transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient(client_transport) list_clusters(dataproc_cluster_client, project_id, region) -if __name__ == '__main__': +if __name__ == "__main__": parser = argparse.ArgumentParser( - description=__doc__, formatter_class=( - argparse.RawDescriptionHelpFormatter)) - parser.add_argument( - '--project_id', help='Project ID to access.', required=True) - parser.add_argument( - '--region', help='Region of clusters to list.', required=True) + description=__doc__, formatter_class=(argparse.RawDescriptionHelpFormatter) + ) + parser.add_argument("--project_id", help="Project ID to access.", required=True) + parser.add_argument("--region", help="Region of clusters to list.", required=True) args = parser.parse_args() main(args.project_id, args.region) diff --git a/dataproc/snippets/pyspark_sort.py b/dataproc/snippets/pyspark_sort.py index 0ce2350ad02b..e43d57f13186 100644 --- a/dataproc/snippets/pyspark_sort.py +++ b/dataproc/snippets/pyspark_sort.py @@ -22,7 +22,7 @@ import pyspark sc = pyspark.SparkContext() -rdd = sc.parallelize(['Hello,', 'world!', 'dog', 'elephant', 'panther']) +rdd = sc.parallelize(["Hello,", "world!", "dog", "elephant", "panther"]) words = sorted(rdd.collect()) print(words) # [END dataproc_pyspark_sort] diff --git a/dataproc/snippets/pyspark_sort_gcs.py b/dataproc/snippets/pyspark_sort_gcs.py index f1961c378d36..4b1e7b2dca4a 100644 --- a/dataproc/snippets/pyspark_sort_gcs.py +++ b/dataproc/snippets/pyspark_sort_gcs.py @@ -25,6 +25,6 @@ import pyspark sc = pyspark.SparkContext() -rdd = sc.textFile('gs://path-to-your-GCS-file') +rdd = sc.textFile("gs://path-to-your-GCS-file") print(sorted(rdd.collect())) # [END dataproc_pyspark_sort_gcs] diff --git a/dataproc/snippets/quickstart/quickstart.py b/dataproc/snippets/quickstart/quickstart.py index 4159e2815202..68f0bdf4ffdb 100644 --- a/dataproc/snippets/quickstart/quickstart.py +++ b/dataproc/snippets/quickstart/quickstart.py @@ -35,57 +35,51 @@ def quickstart(project_id, region, cluster_name, job_file_path): # Create the cluster client. - cluster_client = dataproc.ClusterControllerClient(client_options={ - 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) - }) + cluster_client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} + ) # Create the cluster config. cluster = { - 'project_id': project_id, - 'cluster_name': cluster_name, - 'config': { - 'master_config': { - 'num_instances': 1, - 'machine_type_uri': 'n1-standard-1' - }, - 'worker_config': { - 'num_instances': 2, - 'machine_type_uri': 'n1-standard-1' - } - } + "project_id": project_id, + "cluster_name": cluster_name, + "config": { + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + }, } # Create the cluster. - operation = cluster_client.create_cluster(project_id, region, cluster) + operation = cluster_client.create_cluster( + request={"project_id": project_id, "region": region, "cluster": cluster} + ) result = operation.result() - print('Cluster created successfully: {}'.format(result.cluster_name)) + print("Cluster created successfully: {}".format(result.cluster_name)) # Create the job client. - job_client = dataproc.JobControllerClient(client_options={ - 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) - }) + job_client = dataproc.JobControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} + ) # Create the job config. job = { - 'placement': { - 'cluster_name': cluster_name - }, - 'pyspark_job': { - 'main_python_file_uri': job_file_path - } + "placement": {"cluster_name": cluster_name}, + "pyspark_job": {"main_python_file_uri": job_file_path}, } - job_response = job_client.submit_job(project_id, region, job) + job_response = job_client.submit_job( + request={"project_id": project_id, "region": region, "job": job} + ) job_id = job_response.reference.job_id - print('Submitted job \"{}\".'.format(job_id)) + print('Submitted job "{}".'.format(job_id)) # Termimal states for a job. terminal_states = { - dataproc.types.JobStatus.ERROR, - dataproc.types.JobStatus.CANCELLED, - dataproc.types.JobStatus.DONE + dataproc.JobStatus.State.ERROR, + dataproc.JobStatus.State.CANCELLED, + dataproc.JobStatus.State.DONE, } # Create a timeout such that the job gets cancelled if not in a @@ -96,64 +90,85 @@ def quickstart(project_id, region, cluster_name, job_file_path): # Wait for the job to complete. while job_response.status.state not in terminal_states: if time.time() > time_start + timeout_seconds: - job_client.cancel_job(project_id, region, job_id) - print('Job {} timed out after threshold of {} seconds.'.format( - job_id, timeout_seconds)) + job_client.cancel_job( + request={"project_id": project_id, "region": region, "job_id": job_id} + ) + print( + "Job {} timed out after threshold of {} seconds.".format( + job_id, timeout_seconds + ) + ) # Poll for job termination once a second. time.sleep(1) - job_response = job_client.get_job(project_id, region, job_id) + job_response = job_client.get_job( + request={"project_id": project_id, "region": region, "job_id": job_id} + ) # Cloud Dataproc job output gets saved to a GCS bucket allocated to it. cluster_info = cluster_client.get_cluster( - project_id, region, cluster_name) + request={ + "project_id": project_id, + "region": region, + "cluster_name": cluster_name, + } + ) storage_client = storage.Client() bucket = storage_client.get_bucket(cluster_info.config.config_bucket) - output_blob = ( - 'google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000' - .format(cluster_info.cluster_uuid, job_id)) + output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( + cluster_info.cluster_uuid, job_id + ) output = bucket.blob(output_blob).download_as_string() - print('Job {} finished with state {}:\n{}'.format( - job_id, - job_response.status.State.Name(job_response.status.state), - output)) + print( + "Job {} finished with state {}:\n{}".format( + job_id, job_response.status.state.name, output + ) + ) # Delete the cluster once the job has terminated. - operation = cluster_client.delete_cluster(project_id, region, cluster_name) + operation = cluster_client.delete_cluster( + request={ + "project_id": project_id, + "region": region, + "cluster_name": cluster_name, + } + ) operation.result() - print('Cluster {} successfully deleted.'.format(cluster_name)) + print("Cluster {} successfully deleted.".format(cluster_name)) if __name__ == "__main__": parser = argparse.ArgumentParser( - description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter, + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter, ) parser.add_argument( - '--project_id', + "--project_id", type=str, required=True, - help='Project to use for creating resources.') + help="Project to use for creating resources.", + ) parser.add_argument( - '--region', + "--region", type=str, required=True, - help='Region where the resources should live.') + help="Region where the resources should live.", + ) parser.add_argument( - '--cluster_name', + "--cluster_name", type=str, required=True, - help='Name to use for creating a cluster.') + help="Name to use for creating a cluster.", + ) parser.add_argument( - '--job_file_path', + "--job_file_path", type=str, required=True, - help='Job in GCS to execute against the cluster.') + help="Job in GCS to execute against the cluster.", + ) args = parser.parse_args() - quickstart(args.project_id, args.region, - args.cluster_name, args.job_file_path) + quickstart(args.project_id, args.region, args.cluster_name, args.job_file_path) # [END dataproc_quickstart] diff --git a/dataproc/snippets/quickstart/quickstart_test.py b/dataproc/snippets/quickstart/quickstart_test.py index 3e17f6fa3e57..9ea46cd19c5a 100644 --- a/dataproc/snippets/quickstart/quickstart_test.py +++ b/dataproc/snippets/quickstart/quickstart_test.py @@ -22,12 +22,12 @@ import quickstart -PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] -REGION = 'us-central1' -CLUSTER_NAME = 'py-qs-test-{}'.format(str(uuid.uuid4())) -STAGING_BUCKET = 'py-dataproc-qs-bucket-{}'.format(str(uuid.uuid4())) -JOB_FILE_NAME = 'sum.py' -JOB_FILE_PATH = 'gs://{}/{}'.format(STAGING_BUCKET, JOB_FILE_NAME) +PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"] +REGION = "us-central1" +CLUSTER_NAME = "py-qs-test-{}".format(str(uuid.uuid4())) +STAGING_BUCKET = "py-dataproc-qs-bucket-{}".format(str(uuid.uuid4())) +JOB_FILE_NAME = "sum.py" +JOB_FILE_PATH = "gs://{}/{}".format(STAGING_BUCKET, JOB_FILE_NAME) SORT_CODE = ( "import pyspark\n" "sc = pyspark.SparkContext()\n" @@ -45,17 +45,25 @@ def setup_teardown(): yield - cluster_client = dataproc.ClusterControllerClient(client_options={ - 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(REGION) - }) + cluster_client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION)} + ) # The quickstart sample deletes the cluster, but if the test fails # before cluster deletion occurs, it can be manually deleted here. - clusters = cluster_client.list_clusters(PROJECT_ID, REGION) + clusters = cluster_client.list_clusters( + request={"project_id": PROJECT_ID, "region": REGION} + ) for cluster in clusters: if cluster.cluster_name == CLUSTER_NAME: - cluster_client.delete_cluster(PROJECT_ID, REGION, CLUSTER_NAME) + cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) blob.delete() bucket.delete() @@ -65,7 +73,7 @@ def test_quickstart(capsys): quickstart.quickstart(PROJECT_ID, REGION, CLUSTER_NAME, JOB_FILE_PATH) out, _ = capsys.readouterr() - assert 'Cluster created successfully' in out - assert 'Submitted job' in out - assert 'finished with state DONE:' in out - assert 'successfully deleted' in out + assert "Cluster created successfully" in out + assert "Submitted job" in out + assert "finished with state DONE:" in out + assert "successfully deleted" in out diff --git a/dataproc/snippets/single_job_workflow.py b/dataproc/snippets/single_job_workflow.py index b2754b06c1ec..04f73ea9f282 100644 --- a/dataproc/snippets/single_job_workflow.py +++ b/dataproc/snippets/single_job_workflow.py @@ -30,7 +30,8 @@ from google.cloud import dataproc_v1 from google.cloud import storage from google.cloud.dataproc_v1.gapic.transports import ( - workflow_template_service_grpc_transport) + workflow_template_service_grpc_transport, +) DEFAULT_FILENAME = "pyspark_sort.py" @@ -66,12 +67,12 @@ def upload_pyspark_file(project, bucket_name, filename, spark_file): blob.upload_from_file(spark_file) -def run_workflow(dataproc, project, region, zone, bucket_name, filename, - cluster_name): +def run_workflow(dataproc, project, region, zone, bucket_name, filename, cluster_name): parent = "projects/{}/regions/{}".format(project, region) - zone_uri = ("https://www.googleapis.com/compute/v1/projects/{}/zones/{}" - .format(project, zone)) + zone_uri = "https://www.googleapis.com/compute/v1/projects/{}/zones/{}".format( + project, zone + ) workflow_data = { "placement": { @@ -93,16 +94,16 @@ def run_workflow(dataproc, project, region, zone, bucket_name, filename, "jobs": [ { "pyspark_job": { - "main_python_file_uri": "gs://{}/{}".format( - bucket_name, filename) + "main_python_file_uri": "gs://{}/{}".format(bucket_name, filename) }, "step_id": "pyspark-job", } ], } - workflow = dataproc.instantiate_inline_workflow_template(parent, - workflow_data) + workflow = dataproc.instantiate_inline_workflow_template( + request={"parent": parent, "template": workflow_data} + ) workflow.add_done_callback(callback) global waiting_callback @@ -118,8 +119,10 @@ def callback(operation_future): def wait_for_workflow_end(): """Wait for cluster creation.""" print("Waiting for workflow completion ...") - print("Workflow and job progress, and job driver output available from: " - "https://console.cloud.google.com/dataproc/workflows/") + print( + "Workflow and job progress, and job driver output available from: " + "https://console.cloud.google.com/dataproc/workflows/" + ) while True: if not waiting_callback: @@ -146,10 +149,9 @@ def main( region = get_region_from_zone(zone) # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = (workflow_template_service_grpc_transport - .WorkflowTemplateServiceGrpcTransport( - address="{}-dataproc.googleapis.com:443" - .format(region))) + client_transport = workflow_template_service_grpc_transport.WorkflowTemplateServiceGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) dataproc_workflow_client = dataproc_v1.WorkflowTemplateServiceClient( client_transport ) @@ -157,8 +159,7 @@ def main( try: spark_file, spark_filename = get_pyspark_file(pyspark_file) - upload_pyspark_file(project_id, bucket_name, spark_filename, - spark_file) + upload_pyspark_file(project_id, bucket_name, spark_filename, spark_file) run_workflow( dataproc_workflow_client, @@ -167,7 +168,7 @@ def main( zone, bucket_name, spark_filename, - cluster_name + cluster_name, ) wait_for_workflow_end() @@ -177,8 +178,8 @@ def main( if __name__ == "__main__": parser = argparse.ArgumentParser( - description=__doc__, formatter_class=(argparse - .RawDescriptionHelpFormatter)) + description=__doc__, formatter_class=(argparse.RawDescriptionHelpFormatter) + ) parser.add_argument( "--project_id", help="Project ID you want to access.", required=True ) @@ -186,8 +187,7 @@ def main( "--zone", help="Zone to create clusters in/connect to", required=True ) parser.add_argument( - "--cluster_name", help="Name of the cluster to create/connect to", - required=True + "--cluster_name", help="Name of the cluster to create/connect to", required=True ) parser.add_argument( "--gcs_bucket", help="Bucket to upload Pyspark file to", required=True @@ -195,9 +195,11 @@ def main( parser.add_argument( "--pyspark_file", help="Pyspark filename. Defaults to pyspark_sort.py" ) - parser.add_argument("--global_region", - action="store_true", - help="If cluster is in the global region") + parser.add_argument( + "--global_region", + action="store_true", + help="If cluster is in the global region", + ) args = parser.parse_args() main( diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index 389cbec87aa3..d613cf5b8609 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -34,13 +34,11 @@ from google.cloud import dataproc_v1 from google.cloud import storage -from google.cloud.dataproc_v1.gapic.transports import ( - cluster_controller_grpc_transport) -from google.cloud.dataproc_v1.gapic.transports import ( - job_controller_grpc_transport) +from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport +from google.cloud.dataproc_v1.gapic.transports import job_controller_grpc_transport -DEFAULT_FILENAME = 'pyspark_sort.py' +DEFAULT_FILENAME = "pyspark_sort.py" waiting_callback = False @@ -57,16 +55,16 @@ def get_pyspark_file(pyspark_file=None): def get_region_from_zone(zone): try: - region_as_list = zone.split('-')[:-1] - return '-'.join(region_as_list) + region_as_list = zone.split("-")[:-1] + return "-".join(region_as_list) except (AttributeError, IndexError, ValueError): - raise ValueError('Invalid zone provided, please check your input.') + raise ValueError("Invalid zone provided, please check your input.") def upload_pyspark_file(project, bucket_name, filename, spark_file): """Uploads the PySpark file in this directory to the configured input bucket.""" - print('Uploading pyspark file to Cloud Storage.') + print("Uploading pyspark file to Cloud Storage.") client = storage.Client(project=project) bucket = client.get_bucket(bucket_name) blob = bucket.blob(filename) @@ -76,44 +74,40 @@ def upload_pyspark_file(project, bucket_name, filename, spark_file): def download_output(project, cluster_id, output_bucket, job_id): """Downloads the output file from Cloud Storage and returns it as a string.""" - print('Downloading output file.') + print("Downloading output file.") client = storage.Client(project=project) bucket = client.get_bucket(output_bucket) - output_blob = ( - ('google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000'. - format(cluster_id, job_id))) + output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( + cluster_id, job_id + ) return bucket.blob(output_blob).download_as_string() # [START dataproc_create_cluster] def create_cluster(dataproc, project, zone, region, cluster_name): """Create the cluster.""" - print('Creating cluster...') - zone_uri = \ - 'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format( - project, zone) + print("Creating cluster...") + zone_uri = "https://www.googleapis.com/compute/v1/projects/{}/zones/{}".format( + project, zone + ) cluster_data = { - 'project_id': project, - 'cluster_name': cluster_name, - 'config': { - 'gce_cluster_config': { - 'zone_uri': zone_uri - }, - 'master_config': { - 'num_instances': 1, - 'machine_type_uri': 'n1-standard-1' - }, - 'worker_config': { - 'num_instances': 2, - 'machine_type_uri': 'n1-standard-1' - } - } + "project_id": project, + "cluster_name": cluster_name, + "config": { + "gce_cluster_config": {"zone_uri": zone_uri}, + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + }, } - cluster = dataproc.create_cluster(project, region, cluster_data) + cluster = dataproc.create_cluster( + request={"project_id": project, "region": region, "cluster": cluster_data} + ) cluster.add_done_callback(callback) global waiting_callback waiting_callback = True + + # [END dataproc_create_cluster] @@ -125,7 +119,7 @@ def callback(operation_future): def wait_for_cluster_creation(): """Wait for cluster creation.""" - print('Waiting for cluster creation...') + print("Waiting for cluster creation...") while True: if not waiting_callback: @@ -136,79 +130,99 @@ def wait_for_cluster_creation(): # [START dataproc_list_clusters_with_detail] def list_clusters_with_details(dataproc, project, region): """List the details of clusters in the region.""" - for cluster in dataproc.list_clusters(project, region): - print(('{} - {}'.format(cluster.cluster_name, - cluster.status.State.Name( - cluster.status.state)))) + for cluster in dataproc.list_clusters( + request={"project_id": project, "region": region} + ): + print( + ( + "{} - {}".format( + cluster.cluster_name, + cluster.status.State.Name(cluster.status.state), + ) + ) + ) + + # [END dataproc_list_clusters_with_detail] def get_cluster_id_by_name(dataproc, project_id, region, cluster_name): """Helper function to retrieve the ID and output bucket of a cluster by name.""" - for cluster in dataproc.list_clusters(project_id, region): + for cluster in dataproc.list_clusters( + request={"project_id": project_id, "region": region} + ): if cluster.cluster_name == cluster_name: return cluster.cluster_uuid, cluster.config.config_bucket # [START dataproc_submit_pyspark_job] -def submit_pyspark_job(dataproc, project, region, cluster_name, bucket_name, - filename): +def submit_pyspark_job(dataproc, project, region, cluster_name, bucket_name, filename): """Submit the Pyspark job to the cluster (assumes `filename` was uploaded to `bucket_name.""" job_details = { - 'placement': { - 'cluster_name': cluster_name + "placement": {"cluster_name": cluster_name}, + "pyspark_job": { + "main_python_file_uri": "gs://{}/{}".format(bucket_name, filename) }, - 'pyspark_job': { - 'main_python_file_uri': 'gs://{}/{}'.format(bucket_name, filename) - } } result = dataproc.submit_job( - project_id=project, region=region, job=job_details) + request={"project_id": project, "region": region, "job": job_details} + ) job_id = result.reference.job_id - print('Submitted job ID {}.'.format(job_id)) + print("Submitted job ID {}.".format(job_id)) return job_id + + # [END dataproc_submit_pyspark_job] # [START dataproc_delete] def delete_cluster(dataproc, project, region, cluster): """Delete the cluster.""" - print('Tearing down cluster.') + print("Tearing down cluster.") result = dataproc.delete_cluster( - project_id=project, region=region, cluster_name=cluster) + request={"project_id": project, "region": region, "cluster_name": cluster} + ) return result + + # [END dataproc_delete] # [START dataproc_wait] def wait_for_job(dataproc, project, region, job_id): """Wait for job to complete or error out.""" - print('Waiting for job to finish...') + print("Waiting for job to finish...") while True: - job = dataproc.get_job(project, region, job_id) + job = dataproc.get_job( + request={"project_id": project, "region": region, "job_id": job_id} + ) # Handle exceptions - if job.status.State.Name(job.status.state) == 'ERROR': + if job.status.State.Name(job.status.state) == "ERROR": raise Exception(job.status.details) - elif job.status.State.Name(job.status.state) == 'DONE': - print('Job finished.') + elif job.status.State.Name(job.status.state) == "DONE": + print("Job finished.") return job + + # [END dataproc_wait] -def main(project_id, - zone, - cluster_name, - bucket_name, - pyspark_file=None, - create_new_cluster=True, - global_region=True): +def main( + project_id, + zone, + cluster_name, + bucket_name, + pyspark_file=None, + create_new_cluster=True, + global_region=True, +): # [START dataproc_get_client] if global_region: - region = 'global' + region = "global" # Use the default gRPC global endpoints. dataproc_cluster_client = dataproc_v1.ClusterControllerClient() dataproc_job_client = dataproc_v1.JobControllerClient() @@ -216,73 +230,89 @@ def main(project_id, region = get_region_from_zone(zone) # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = ( - cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address='{}-dataproc.googleapis.com:443'.format(region))) - job_transport = ( - job_controller_grpc_transport.JobControllerGrpcTransport( - address='{}-dataproc.googleapis.com:443'.format(region))) - dataproc_cluster_client = dataproc_v1.ClusterControllerClient( - client_transport) + client_transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) + job_transport = job_controller_grpc_transport.JobControllerGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient(client_transport) dataproc_job_client = dataproc_v1.JobControllerClient(job_transport) # [END dataproc_get_client] try: spark_file, spark_filename = get_pyspark_file(pyspark_file) if create_new_cluster: - create_cluster(dataproc_cluster_client, project_id, zone, region, - cluster_name) + create_cluster( + dataproc_cluster_client, project_id, zone, region, cluster_name + ) wait_for_cluster_creation() - upload_pyspark_file(project_id, bucket_name, spark_filename, - spark_file) + upload_pyspark_file(project_id, bucket_name, spark_filename, spark_file) - list_clusters_with_details(dataproc_cluster_client, project_id, - region) + list_clusters_with_details(dataproc_cluster_client, project_id, region) - (cluster_id, output_bucket) = ( - get_cluster_id_by_name(dataproc_cluster_client, project_id, - region, cluster_name)) + (cluster_id, output_bucket) = get_cluster_id_by_name( + dataproc_cluster_client, project_id, region, cluster_name + ) # [START dataproc_call_submit_pyspark_job] - job_id = submit_pyspark_job(dataproc_job_client, project_id, region, - cluster_name, bucket_name, spark_filename) + job_id = submit_pyspark_job( + dataproc_job_client, + project_id, + region, + cluster_name, + bucket_name, + spark_filename, + ) # [END dataproc_call_submit_pyspark_job] wait_for_job(dataproc_job_client, project_id, region, job_id) output = download_output(project_id, cluster_id, output_bucket, job_id) - print('Received job output {}'.format(output)) + print("Received job output {}".format(output)) return output finally: if create_new_cluster: - delete_cluster(dataproc_cluster_client, project_id, region, - cluster_name) + delete_cluster(dataproc_cluster_client, project_id, region, cluster_name) spark_file.close() -if __name__ == '__main__': - parser = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse. - RawDescriptionHelpFormatter) +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument( + "--project_id", help="Project ID you want to access.", required=True + ) + parser.add_argument( + "--zone", help="Zone to create clusters in/connect to", required=True + ) + parser.add_argument( + "--cluster_name", help="Name of the cluster to create/connect to", required=True + ) + parser.add_argument( + "--gcs_bucket", help="Bucket to upload Pyspark file to", required=True + ) + parser.add_argument( + "--pyspark_file", help="Pyspark filename. Defaults to pyspark_sort.py" + ) + parser.add_argument( + "--create_new_cluster", + action="store_true", + help="States if the cluster should be created", + ) parser.add_argument( - '--project_id', help='Project ID you want to access.', required=True) - parser.add_argument('--zone', - help='Zone to create clusters in/connect to', - required=True) - parser.add_argument('--cluster_name', - help='Name of the cluster to create/connect to', - required=True) - parser.add_argument('--gcs_bucket', - help='Bucket to upload Pyspark file to', - required=True) - parser.add_argument('--pyspark_file', - help='Pyspark filename. Defaults to pyspark_sort.py') - parser.add_argument('--create_new_cluster', - action='store_true', - help='States if the cluster should be created') - parser.add_argument('--global_region', - action='store_true', - help='If cluster is in the global region') + "--global_region", + action="store_true", + help="If cluster is in the global region", + ) args = parser.parse_args() - main(args.project_id, args.zone, args.cluster_name, args.gcs_bucket, - args.pyspark_file, args.create_new_cluster, args.global_region) + main( + args.project_id, + args.zone, + args.cluster_name, + args.gcs_bucket, + args.pyspark_file, + args.create_new_cluster, + args.global_region, + ) From c0b587b15e2f7437ca41e055bf934d4fc4fca2c5 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 16 Sep 2020 04:05:23 +0200 Subject: [PATCH 005/158] chore(deps): update dependency google-auth to v1.21.2 (#84) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index c2c958754ddc..737119944830 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.31.0 -google-auth==1.20.1 +google-auth==1.21.2 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.30.0 From ef0ea6ccfd6eef43c5dadc4b191a7db5674fdec8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 16 Sep 2020 05:40:02 +0200 Subject: [PATCH 006/158] chore(deps): update dependency grpcio to v1.32.0 (#89) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio](https://grpc.io) | minor | `==1.31.0` -> `==1.32.0` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 737119944830..a36272e7db37 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.31.0 +grpcio==1.32.0 google-auth==1.21.2 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 7704b720ca8bdfbb444bc4d40acc0222009d840f Mon Sep 17 00:00:00 2001 From: Brad Miro Date: Wed, 16 Sep 2020 15:53:38 -0400 Subject: [PATCH 007/158] feat: adding submit_job samples (#88) Adding submit_job samples and updating quickstart samples. --- dataproc/snippets/quickstart/quickstart.py | 65 +++------------ .../snippets/quickstart/quickstart_test.py | 3 +- dataproc/snippets/submit_job.py | 80 +++++++++++++++++++ dataproc/snippets/submit_job_test.py | 66 +++++++++++++++ 4 files changed, 159 insertions(+), 55 deletions(-) create mode 100644 dataproc/snippets/submit_job.py create mode 100644 dataproc/snippets/submit_job_test.py diff --git a/dataproc/snippets/quickstart/quickstart.py b/dataproc/snippets/quickstart/quickstart.py index 68f0bdf4ffdb..45e7f348cacf 100644 --- a/dataproc/snippets/quickstart/quickstart.py +++ b/dataproc/snippets/quickstart/quickstart.py @@ -27,7 +27,7 @@ """ import argparse -import time +import re from google.cloud import dataproc_v1 as dataproc from google.cloud import storage @@ -68,64 +68,23 @@ def quickstart(project_id, region, cluster_name, job_file_path): "pyspark_job": {"main_python_file_uri": job_file_path}, } - job_response = job_client.submit_job( + operation = job_client.submit_job_as_operation( request={"project_id": project_id, "region": region, "job": job} ) - job_id = job_response.reference.job_id + response = operation.result() - print('Submitted job "{}".'.format(job_id)) + # Dataproc job output gets saved to the Google Cloud Storage bucket + # allocated to the job. Use a regex to obtain the bucket and blob info. + matches = re.match("gs://(.*?)/(.*)", response.driver_output_resource_uri) - # Termimal states for a job. - terminal_states = { - dataproc.JobStatus.State.ERROR, - dataproc.JobStatus.State.CANCELLED, - dataproc.JobStatus.State.DONE, - } - - # Create a timeout such that the job gets cancelled if not in a - # terminal state after a fixed period of time. - timeout_seconds = 600 - time_start = time.time() - - # Wait for the job to complete. - while job_response.status.state not in terminal_states: - if time.time() > time_start + timeout_seconds: - job_client.cancel_job( - request={"project_id": project_id, "region": region, "job_id": job_id} - ) - print( - "Job {} timed out after threshold of {} seconds.".format( - job_id, timeout_seconds - ) - ) - - # Poll for job termination once a second. - time.sleep(1) - job_response = job_client.get_job( - request={"project_id": project_id, "region": region, "job_id": job_id} - ) - - # Cloud Dataproc job output gets saved to a GCS bucket allocated to it. - cluster_info = cluster_client.get_cluster( - request={ - "project_id": project_id, - "region": region, - "cluster_name": cluster_name, - } + output = ( + storage.Client() + .get_bucket(matches.group(1)) + .blob(f"{matches.group(2)}.000000000") + .download_as_string() ) - storage_client = storage.Client() - bucket = storage_client.get_bucket(cluster_info.config.config_bucket) - output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( - cluster_info.cluster_uuid, job_id - ) - output = bucket.blob(output_blob).download_as_string() - - print( - "Job {} finished with state {}:\n{}".format( - job_id, job_response.status.state.name, output - ) - ) + print(f"Job finished successfully: {output}") # Delete the cluster once the job has terminated. operation = cluster_client.delete_cluster( diff --git a/dataproc/snippets/quickstart/quickstart_test.py b/dataproc/snippets/quickstart/quickstart_test.py index 9ea46cd19c5a..38ae8b894173 100644 --- a/dataproc/snippets/quickstart/quickstart_test.py +++ b/dataproc/snippets/quickstart/quickstart_test.py @@ -74,6 +74,5 @@ def test_quickstart(capsys): out, _ = capsys.readouterr() assert "Cluster created successfully" in out - assert "Submitted job" in out - assert "finished with state DONE:" in out + assert "Job finished successfully" in out assert "successfully deleted" in out diff --git a/dataproc/snippets/submit_job.py b/dataproc/snippets/submit_job.py new file mode 100644 index 000000000000..b70348c3220c --- /dev/null +++ b/dataproc/snippets/submit_job.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This sample walks a user through submitting a Spark job using the Dataproc +# client library. + +# Usage: +# python submit_job.py --project_id --region \ +# --cluster_name + +# [START dataproc_submit_job] +import re +# [END dataproc_submit_job] +import sys +# [START dataproc_submit_job] + +from google.cloud import dataproc_v1 as dataproc +from google.cloud import storage + + +def submit_job(project_id, region, cluster_name): + # Create the job client. + job_client = dataproc.JobControllerClient(client_options={ + 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) + }) + + # Create the job config. 'main_jar_file_uri' can also be a + # Google Cloud Storage URL. + job = { + 'placement': { + 'cluster_name': cluster_name + }, + 'spark_job': { + 'main_class': 'org.apache.spark.examples.SparkPi', + 'jar_file_uris': ['file:///usr/lib/spark/examples/jars/spark-examples.jar'], + 'args': ['1000'] + } + } + + operation = job_client.submit_job_as_operation( + request={"project_id": project_id, "region": region, "job": job} + ) + response = operation.result() + + # Dataproc job output gets saved to the Google Cloud Storage bucket + # allocated to the job. Use a regex to obtain the bucket and blob info. + matches = re.match("gs://(.*?)/(.*)", response.driver_output_resource_uri) + + output = ( + storage.Client() + .get_bucket(matches.group(1)) + .blob(f"{matches.group(2)}.000000000") + .download_as_string() + ) + + print(f"Job finished successfully: {output}") +# [END dataproc_submit_job] + + +if __name__ == "__main__": + if len(sys.argv) < 3: + sys.exit('python submit_job.py project_id region cluster_name') + + project_id = sys.argv[1] + region = sys.argv[2] + cluster_name = sys.argv[3] + submit_job(project_id, region, cluster_name) diff --git a/dataproc/snippets/submit_job_test.py b/dataproc/snippets/submit_job_test.py new file mode 100644 index 000000000000..0cad6248bf18 --- /dev/null +++ b/dataproc/snippets/submit_job_test.py @@ -0,0 +1,66 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import uuid + +from google.cloud import dataproc_v1 as dataproc +import pytest + +import submit_job + + +PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] +REGION = 'us-central1' +CLUSTER_NAME = 'py-sj-test-{}'.format(str(uuid.uuid4())) +CLUSTER = { + 'project_id': PROJECT_ID, + 'cluster_name': CLUSTER_NAME, + 'config': { + 'master_config': { + 'num_instances': 1, + 'machine_type_uri': 'n1-standard-1' + }, + 'worker_config': { + 'num_instances': 2, + 'machine_type_uri': 'n1-standard-1' + } + } +} + + +@pytest.fixture(autouse=True) +def setup_teardown(): + cluster_client = dataproc.ClusterControllerClient(client_options={ + 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(REGION) + }) + + # Create the cluster. + operation = cluster_client.create_cluster( + request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} + ) + operation.result() + + yield + + cluster_client.delete_cluster(request={ + "project_id": PROJECT_ID, "region": REGION, "cluster_name": CLUSTER_NAME + }) + + +def test_submit_job(capsys): + submit_job.submit_job(PROJECT_ID, REGION, CLUSTER_NAME) + out, _ = capsys.readouterr() + + assert 'Job finished successfully' in out From 35ffc736cc270086defaef02f2b287dc7dd4475d Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 23 Sep 2020 22:18:36 +0200 Subject: [PATCH 008/158] chore(deps): update dependency google-auth to v1.21.3 (#93) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a36272e7db37..c8232609f090 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.32.0 -google-auth==1.21.2 +google-auth==1.21.3 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.30.0 From e84dcd8ef6cea5c28a67f16bb7f009d9dd68c02f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 29 Sep 2020 01:34:23 +0200 Subject: [PATCH 009/158] chore(deps): update dependency google-auth to v1.22.0 (#94) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index c8232609f090..3a2afa28770c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.32.0 -google-auth==1.21.3 +google-auth==1.22.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.30.0 From 53d2a8de43eb0ec6f3004ef209aa4e06531be1ce Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 13 Oct 2020 22:06:40 +0200 Subject: [PATCH 010/158] chore(deps): update dependency google-auth to v1.22.1 (#95) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 3a2afa28770c..60e499934c96 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.32.0 -google-auth==1.22.0 +google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.30.0 From e57d060b4264690f1d4e47258e874d389f564455 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 13 Oct 2020 22:43:32 +0200 Subject: [PATCH 011/158] chore(deps): update dependency google-cloud-storage to v1.31.2 (#83) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 60e499934c96..70d2c9438161 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.32.0 google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.30.0 +google-cloud-storage==1.31.2 google-cloud-dataproc==1.1.0 From 299f8df9c3ea3ea6d3fcfe8192a2341502014919 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 16 Oct 2020 21:32:36 +0200 Subject: [PATCH 012/158] chore(deps): update dependency google-cloud-dataproc to v1.1.1 (#81) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 70d2c9438161..91de9aa7593e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -3,4 +3,4 @@ google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.31.2 -google-cloud-dataproc==1.1.0 +google-cloud-dataproc==1.1.1 From 67194221ad2ce2d419a4a71508d7c9205bc8eef0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 16 Oct 2020 22:24:15 +0200 Subject: [PATCH 013/158] chore(deps): update dependency google-cloud-dataproc to v2 (#82) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 91de9aa7593e..514e68d24036 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -3,4 +3,4 @@ google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.31.2 -google-cloud-dataproc==1.1.1 +google-cloud-dataproc==2.0.2 From b2c00a3693cbdf68569d950757d84a7a80138efd Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 23 Oct 2020 20:47:32 +0200 Subject: [PATCH 014/158] chore(deps): update dependency grpcio to v1.33.1 (#97) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 514e68d24036..9c8ba2456965 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.32.0 +grpcio==1.33.1 google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 1ae94dfb85b834a605a93ca3e1cd20c17192f945 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 28 Oct 2020 19:16:05 +0100 Subject: [PATCH 015/158] chore(deps): update dependency grpcio to v1.33.2 (#98) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio](https://grpc.io) | patch | `==1.33.1` -> `==1.33.2` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 9c8ba2456965..87d5f20cd097 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.33.1 +grpcio==1.33.2 google-auth==1.22.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 5bf266647d0f6311bd044e9033b576a3c7ef5eed Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 30 Oct 2020 00:47:52 +0100 Subject: [PATCH 016/158] chore(deps): update dependency google-auth to v1.23.0 (#99) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 87d5f20cd097..d0634d45636c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.33.2 -google-auth==1.22.1 +google-auth==1.23.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.31.2 From 991e0d5ac67666c2fc9baa0809fe7590a955da1e Mon Sep 17 00:00:00 2001 From: Yoshi Automation Bot Date: Mon, 16 Nov 2020 10:15:26 -0800 Subject: [PATCH 017/158] feat: add common resource paths, expose client transport (#87) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * changes without context autosynth cannot find the source of changes triggered by earlier changes in this repository, or by version upgrades to tools such as linters. * chore(py-library): enable snippet-bot Co-authored-by: Benjamin E. Coe Source-Author: Takashi Matsuo Source-Date: Tue Sep 1 17:14:08 2020 +0000 Source-Repo: googleapis/synthtool Source-Sha: d91dd8aac77f7a9c5506c238038a26fa4f9e361e Source-Link: https://github.com/googleapis/synthtool/commit/d91dd8aac77f7a9c5506c238038a26fa4f9e361e * chore(py-library): update decrypt secrets file * chore(py-library): update decrypt secrets file From https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/scripts/decrypt-secrets.sh * docs: explain conditional Co-authored-by: Jeffrey Rennie Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Tue Sep 8 11:35:59 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: d302f93d7f47e2852e585ac35ab2d15585717ec0 Source-Link: https://github.com/googleapis/synthtool/commit/d302f93d7f47e2852e585ac35ab2d15585717ec0 * chore(python-library): use sphinx 1.5.5 for the docfx job Originally tested at: https://github.com/googleapis/python-texttospeech/pull/89 This change will fix the missing docstring in the yaml files. Source-Author: Takashi Matsuo Source-Date: Thu Sep 10 04:12:14 2020 +0000 Source-Repo: googleapis/synthtool Source-Sha: ffcee7952b74f647cbb3ef021d95422f10816fca Source-Link: https://github.com/googleapis/synthtool/commit/ffcee7952b74f647cbb3ef021d95422f10816fca * build(python): use release-publish app for notifying GitHub of releas… * build(python): use release-publish app for notifying GitHub of release status * fix: re-add pypi password Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Wed Sep 16 08:46:42 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: 257fda18168bedb76985024bd198ed1725485488 Source-Link: https://github.com/googleapis/synthtool/commit/257fda18168bedb76985024bd198ed1725485488 * build(python): add secret manager in kokoro Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Wed Sep 16 10:24:40 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: dba48bb9bc6959c232bec9150ac6313b608fe7bd Source-Link: https://github.com/googleapis/synthtool/commit/dba48bb9bc6959c232bec9150ac6313b608fe7bd * chore(python): add sphinx doctest extension Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Mon Sep 21 13:09:57 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: 27f4406999b1eee29e04b09b2423a8e4646c7e24 Source-Link: https://github.com/googleapis/synthtool/commit/27f4406999b1eee29e04b09b2423a8e4646c7e24 * chore(python): remove note about editable installs `pip install -e .` is supported and is how we install the library for tests. Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Tue Sep 22 12:06:12 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: a651c5fb763c69a921aecdd3e1d8dc51dbf20f8d Source-Link: https://github.com/googleapis/synthtool/commit/a651c5fb763c69a921aecdd3e1d8dc51dbf20f8d * chore(python): use BUILD_SPECIFIC_GCLOUD_PROJECT for samples https://github.com/googleapis/python-talent/blob/ef045e8eb348db36d7a2a611e6f26b11530d273b/samples/snippets/noxfile_config.py#L27-L32 `BUILD_SPECIFIC_GCLOUD_PROJECT` is an alternate project used for sample tests that do poorly with concurrent runs on the same project. Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Wed Sep 30 13:06:03 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: 9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4 Source-Link: https://github.com/googleapis/synthtool/commit/9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4 * chore(python): use 'setup.py' to detect repo root Closes #792 Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Fri Oct 9 15:06:33 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: e0ae456852bf22f38796deb79cff30b516fde244 Source-Link: https://github.com/googleapis/synthtool/commit/e0ae456852bf22f38796deb79cff30b516fde244 * build(python): samples tests should pass if no samples exist Source-Author: Daniel Sanche Source-Date: Wed Oct 14 08:00:06 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: 477764cc4ee6db346d3febef2bb1ea0abf27de52 Source-Link: https://github.com/googleapis/synthtool/commit/477764cc4ee6db346d3febef2bb1ea0abf27de52 * chore(python_library): change the docs bucket name Source-Author: Takashi Matsuo Source-Date: Fri Oct 16 09:58:05 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: da5c6050d13b4950c82666a81d8acd25157664ae Source-Link: https://github.com/googleapis/synthtool/commit/da5c6050d13b4950c82666a81d8acd25157664ae * chore(docs): update code of conduct of synthtool and templates Source-Author: Christopher Wilcox Source-Date: Thu Oct 22 14:22:01 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: 5f6ef0ec5501d33c4667885b37a7685a30d41a76 Source-Link: https://github.com/googleapis/synthtool/commit/5f6ef0ec5501d33c4667885b37a7685a30d41a76 * docs: add proto-plus to intersphinx mapping Source-Author: Tim Swast Source-Date: Tue Oct 27 12:01:14 2020 -0500 Source-Repo: googleapis/synthtool Source-Sha: ea52b8a0bd560f72f376efcf45197fb7c8869120 Source-Link: https://github.com/googleapis/synthtool/commit/ea52b8a0bd560f72f376efcf45197fb7c8869120 * fix(python_library): fix external unit test dependencies I recently submitted https://github.com/googleapis/synthtool/pull/811/files, allowing external dependencies for unit tests. This fixes a small missing comma bug Source-Author: Daniel Sanche Source-Date: Thu Oct 29 16:58:01 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: 6542bd723403513626f61642fc02ddca528409aa Source-Link: https://github.com/googleapis/synthtool/commit/6542bd723403513626f61642fc02ddca528409aa * chore: add type hint check Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Source-Date: Wed Nov 4 17:36:32 2020 -0800 Source-Repo: googleapis/synthtool Source-Sha: 3d3e94c4e02370f307a9a200b0c743c3d8d19f29 Source-Link: https://github.com/googleapis/synthtool/commit/3d3e94c4e02370f307a9a200b0c743c3d8d19f29 * chore: add blacken to template Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Source-Date: Thu Nov 5 15:22:03 2020 -0800 Source-Repo: googleapis/synthtool Source-Sha: 1f1148d3c7a7a52f0c98077f976bd9b3c948ee2b Source-Link: https://github.com/googleapis/synthtool/commit/1f1148d3c7a7a52f0c98077f976bd9b3c948ee2b * fix: address lint issues Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Source-Date: Thu Nov 12 11:30:49 2020 -0800 Source-Repo: googleapis/synthtool Source-Sha: e89175cf074dccc4babb4eca66ae913696e47a71 Source-Link: https://github.com/googleapis/synthtool/commit/e89175cf074dccc4babb4eca66ae913696e47a71 --- dataproc/snippets/noxfile.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index ba55d7ce53ca..b90eef00f2d9 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -39,6 +39,10 @@ # You can opt out from the test for specific Python versions. 'ignored_versions': ["2.7"], + # Old samples are opted out of enforcing Python type hints + # All new samples should feature them + 'enforce_type_hints': False, + # An envvar key for determining the project id to use. Change it # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a # build specific Cloud project. You can also use your own string @@ -132,7 +136,10 @@ def _determine_local_import_names(start_dir): @nox.session def lint(session): - session.install("flake8", "flake8-import-order") + if not TEST_CONFIG['enforce_type_hints']: + session.install("flake8", "flake8-import-order") + else: + session.install("flake8", "flake8-import-order", "flake8-annotations") local_names = _determine_local_import_names(".") args = FLAKE8_COMMON_ARGS + [ @@ -141,8 +148,18 @@ def lint(session): "." ] session.run("flake8", *args) +# +# Black +# +@nox.session +def blacken(session): + session.install("black") + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + session.run("black", *python_files) + # # Sample Tests # @@ -201,6 +218,11 @@ def _get_repo_root(): break if Path(p / ".git").exists(): return str(p) + # .git is not available in repos cloned via Cloud Build + # setup.py is always in the library's root, so use that instead + # https://github.com/googleapis/synthtool/issues/792 + if Path(p / "setup.py").exists(): + return str(p) p = p.parent raise Exception("Unable to detect repository root.") From d8bc905a14028aca2225f7dbfc2118a45c4c04d9 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 16 Nov 2020 19:34:04 +0100 Subject: [PATCH 018/158] chore(deps): update dependency google-cloud-storage to v1.32.0 (#96) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | minor | `==1.31.2` -> `==1.32.0` | --- ### Release Notes
googleapis/python-storage ### [`v1.32.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1320-httpswwwgithubcomgoogleapispython-storagecomparev1312v1320-2020-10-16) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.31.2...v1.32.0) ##### Features - retry API calls with exponential backoff ([#​287](https://www.github.com/googleapis/python-storage/issues/287)) ([fbe5d9c](https://www.github.com/googleapis/python-storage/commit/fbe5d9ca8684c6a992dcdee977fc8dd012a96a5c)) ##### Bug Fixes - field policy return string ([#​282](https://www.github.com/googleapis/python-storage/issues/282)) ([c356b84](https://www.github.com/googleapis/python-storage/commit/c356b8484a758548d5f4823a495ab70c798cfaaf)) - self-upload files for Unicode system test ([#​296](https://www.github.com/googleapis/python-storage/issues/296)) ([6f865d9](https://www.github.com/googleapis/python-storage/commit/6f865d97a19278884356055dfeeaae92f7c63cc1)) - use version.py for versioning, avoid issues with discovering version via get_distribution ([#​288](https://www.github.com/googleapis/python-storage/issues/288)) ([fcd1c4f](https://www.github.com/googleapis/python-storage/commit/fcd1c4f7c947eb95d6937783fd69670a570f145e)) ##### [1.31.2](https://www.github.com/googleapis/python-storage/compare/v1.31.1...v1.31.2) (2020-09-23) ##### Documentation - fix docstring example for 'blob.generate_signed_url' ([#​278](https://www.github.com/googleapis/python-storage/issues/278)) ([2dc91c9](https://www.github.com/googleapis/python-storage/commit/2dc91c947e3693023b4478a15c460693808ea2d9)) ##### [1.31.1](https://www.github.com/googleapis/python-storage/compare/v1.31.0...v1.31.1) (2020-09-16) ##### Bug Fixes - add requests as a dependency ([#​271](https://www.github.com/googleapis/python-storage/issues/271)) ([ec52b38](https://www.github.com/googleapis/python-storage/commit/ec52b38df211fad18a86d7e16d83db79de59d5f5)) - preserve existing blob hashes when 'X-Goog-Hash header' is not present ([#​267](https://www.github.com/googleapis/python-storage/issues/267)) ([277afb8](https://www.github.com/googleapis/python-storage/commit/277afb83f464d77b163f2722272092df4180411e)) - **blob:** base64 includes additional characters ([#​258](https://www.github.com/googleapis/python-storage/issues/258)) ([cf0774a](https://www.github.com/googleapis/python-storage/commit/cf0774aa8ffd45d340aff9a7d2236d8d65c8ae93)) ##### Documentation - add docs signed_url expiration take default utc ([#​250](https://www.github.com/googleapis/python-storage/issues/250)) ([944ab18](https://www.github.com/googleapis/python-storage/commit/944ab1827b3ca0bd1d3aafc2829245290e9bde59))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d0634d45636c..6ba2d5d838dd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.33.2 google-auth==1.23.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.31.2 +google-cloud-storage==1.32.0 google-cloud-dataproc==2.0.2 From 67542be1f1900b9c5477b9cc09c58d7c09f797a5 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 16 Nov 2020 20:47:13 +0100 Subject: [PATCH 019/158] chore(deps): update dependency google-cloud-dataproc to v2.2.0 (#102) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 6ba2d5d838dd..982833f3f712 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -3,4 +3,4 @@ google-auth==1.23.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.32.0 -google-cloud-dataproc==2.0.2 +google-cloud-dataproc==2.2.0 From 72624c1b84a38e0b8b2a670c9b589cf4ea8fb59a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 18 Nov 2020 01:14:01 +0100 Subject: [PATCH 020/158] chore(deps): update dependency google-cloud-storage to v1.33.0 (#103) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 982833f3f712..bf62ea36f6e0 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.33.2 google-auth==1.23.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.32.0 +google-cloud-storage==1.33.0 google-cloud-dataproc==2.2.0 From 16dacfe3ee04b3373d2bcf6e7f5e37fa18af44ef Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 5 Dec 2020 06:36:56 +0100 Subject: [PATCH 021/158] chore(deps): update dependency grpcio to v1.34.0 (#109) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index bf62ea36f6e0..dc4982780bcd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.33.2 +grpcio==1.34.0 google-auth==1.23.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 4ccf22a80231a581a4b71491447128116576675c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 15 Dec 2020 22:34:12 +0100 Subject: [PATCH 022/158] chore(deps): update dependency google-auth to v1.24.0 (#114) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index dc4982780bcd..af0257c8a7d2 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.34.0 -google-auth==1.23.0 +google-auth==1.24.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.33.0 From 522f2b1fbc9644cd1f43e6e082133a1c949e1066 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 15 Dec 2020 23:02:17 +0100 Subject: [PATCH 023/158] chore(deps): update dependency google-cloud-storage to v1.35.0 (#113) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index af0257c8a7d2..148c2c2a2683 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.34.0 google-auth==1.24.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.33.0 +google-cloud-storage==1.35.0 google-cloud-dataproc==2.2.0 From 89b26dd46d32a123a86a1b9732dab7265a059dab Mon Sep 17 00:00:00 2001 From: Yoshi Automation Bot Date: Tue, 29 Dec 2020 08:46:09 -0800 Subject: [PATCH 024/158] chore: update templates (#118) --- dataproc/snippets/noxfile.py | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index b90eef00f2d9..bca0522ec4d9 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -17,6 +17,7 @@ import os from pathlib import Path import sys +from typing import Callable, Dict, List, Optional import nox @@ -68,7 +69,7 @@ TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) -def get_pytest_env_vars(): +def get_pytest_env_vars() -> Dict[str, str]: """Returns a dict for pytest invocation.""" ret = {} @@ -97,7 +98,7 @@ def get_pytest_env_vars(): # -def _determine_local_import_names(start_dir): +def _determine_local_import_names(start_dir: str) -> List[str]: """Determines all import names that should be considered "local". This is used when running the linter to insure that import order is @@ -135,7 +136,7 @@ def _determine_local_import_names(start_dir): @nox.session -def lint(session): +def lint(session: nox.sessions.Session) -> None: if not TEST_CONFIG['enforce_type_hints']: session.install("flake8", "flake8-import-order") else: @@ -154,7 +155,7 @@ def lint(session): @nox.session -def blacken(session): +def blacken(session: nox.sessions.Session) -> None: session.install("black") python_files = [path for path in os.listdir(".") if path.endswith(".py")] @@ -168,7 +169,7 @@ def blacken(session): PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] -def _session_tests(session, post_install=None): +def _session_tests(session: nox.sessions.Session, post_install: Callable = None) -> None: """Runs py.test for a particular project.""" if os.path.exists("requirements.txt"): session.install("-r", "requirements.txt") @@ -194,7 +195,7 @@ def _session_tests(session, post_install=None): @nox.session(python=ALL_VERSIONS) -def py(session): +def py(session: nox.sessions.Session) -> None: """Runs py.test for a sample using the specified version of Python.""" if session.python in TESTED_VERSIONS: _session_tests(session) @@ -209,7 +210,7 @@ def py(session): # -def _get_repo_root(): +def _get_repo_root() -> Optional[str]: """ Returns the root folder of the project. """ # Get root of this repository. Assume we don't have directories nested deeper than 10 items. p = Path(os.getcwd()) @@ -232,7 +233,7 @@ def _get_repo_root(): @nox.session @nox.parametrize("path", GENERATED_READMES) -def readmegen(session, path): +def readmegen(session: nox.sessions.Session, path: str) -> None: """(Re-)generates the readme for a sample.""" session.install("jinja2", "pyyaml") dir_ = os.path.dirname(path) From ef8f87cf926a463d04a73be29cc51ec089987d95 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 15 Jan 2021 20:53:24 +0100 Subject: [PATCH 025/158] chore(deps): update dependency grpcio to v1.34.1 (#124) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 148c2c2a2683..f185eed09a9c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.34.0 +grpcio==1.34.1 google-auth==1.24.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 49a965aa335c936d10661d7b8e96cf8a7aaea5ac Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 12 Feb 2021 07:26:24 +0100 Subject: [PATCH 026/158] chore(deps): update dependency google-auth to v1.26.1 (#132) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index f185eed09a9c..1e5456e8be5e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.34.1 -google-auth==1.24.0 +google-auth==1.26.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.35.0 From 88828d946b13acede8820ca08c6275d0a56c730f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 12 Feb 2021 07:26:45 +0100 Subject: [PATCH 027/158] chore(deps): update dependency google-cloud-storage to v1.36.0 (#131) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 1e5456e8be5e..fe1a9c5be93f 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.34.1 google-auth==1.26.1 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.35.0 +google-cloud-storage==1.36.0 google-cloud-dataproc==2.2.0 From c9df8a01834aa099a2f200ba6f81778aa6507393 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 20 Feb 2021 06:58:32 +0100 Subject: [PATCH 028/158] chore(deps): update dependency google-auth to v1.27.0 (#134) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index fe1a9c5be93f..33758576a0dd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ grpcio==1.34.1 -google-auth==1.26.1 +google-auth==1.27.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 google-cloud-storage==1.36.0 From e378716ea990a522632c228e2f9bcc9fc03b3c5f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 22 Feb 2021 17:04:38 +0100 Subject: [PATCH 029/158] chore(deps): update dependency grpcio to v1.35.0 (#127) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 33758576a0dd..1841063ba67e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.34.1 +grpcio==1.35.0 google-auth==1.27.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From 41b9dafa00a5750aa57e55de0db0f5a672ef7346 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 24 Feb 2021 05:40:10 +0100 Subject: [PATCH 030/158] chore(deps): update dependency google-cloud-storage to v1.36.1 (#136) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 1841063ba67e..e14db7e1e9ee 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,5 +2,5 @@ grpcio==1.35.0 google-auth==1.27.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 -google-cloud-storage==1.36.0 +google-cloud-storage==1.36.1 google-cloud-dataproc==2.2.0 From b86ebf80fa8be3121c388c249fa1a144616bc077 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 1 Mar 2021 17:36:19 +0100 Subject: [PATCH 031/158] chore(deps): update dependency grpcio to v1.36.0 (#137) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e14db7e1e9ee..5b924fb19ce2 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.35.0 +grpcio==1.36.0 google-auth==1.27.0 google-auth-httplib2==0.0.4 google-cloud==0.34.0 From ba80c32f64d332ce8fd748ed44adc3d29a1c0686 Mon Sep 17 00:00:00 2001 From: Brad Miro Date: Tue, 2 Mar 2021 18:31:40 -0500 Subject: [PATCH 032/158] docs: adding backoff to quickstart test (#135) * adding backoff to quickstart test * fixing lint --- dataproc/snippets/quickstart/quickstart_test.py | 11 ++++++++++- dataproc/snippets/requirements.txt | 2 ++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/dataproc/snippets/quickstart/quickstart_test.py b/dataproc/snippets/quickstart/quickstart_test.py index 38ae8b894173..19385353ccf5 100644 --- a/dataproc/snippets/quickstart/quickstart_test.py +++ b/dataproc/snippets/quickstart/quickstart_test.py @@ -15,6 +15,8 @@ import os import uuid +import backoff +from google.api_core.exceptions import ServiceUnavailable from google.cloud import dataproc_v1 as dataproc from google.cloud import storage import pytest @@ -39,7 +41,14 @@ @pytest.fixture(autouse=True) def setup_teardown(): storage_client = storage.Client() - bucket = storage_client.create_bucket(STAGING_BUCKET) + + @backoff.on_exception(backoff.expo, + ServiceUnavailable, + max_tries=5) + def create_bucket(): + return storage_client.create_bucket(STAGING_BUCKET) + + bucket = create_bucket() blob = bucket.blob(JOB_FILE_NAME) blob.upload_from_string(SORT_CODE) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 5b924fb19ce2..3682210a040a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,3 +1,5 @@ + +backoff==1.10.0 grpcio==1.36.0 google-auth==1.27.0 google-auth-httplib2==0.0.4 From d6fe3c7f47d29216ed5337986506843ca101a4b8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 5 Mar 2021 03:42:30 +0100 Subject: [PATCH 033/158] chore(deps): update dependency google-auth-httplib2 to v0.1.0 (#142) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 3682210a040a..74c031854a03 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -2,7 +2,7 @@ backoff==1.10.0 grpcio==1.36.0 google-auth==1.27.0 -google-auth-httplib2==0.0.4 +google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.36.1 google-cloud-dataproc==2.2.0 From 11c266e771b7203772ce4d4ce645b92ed3cc771c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 5 Mar 2021 03:44:33 +0100 Subject: [PATCH 034/158] chore(deps): update dependency grpcio to v1.36.1 (#139) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 74c031854a03..353f3bbf1a8b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.10.0 -grpcio==1.36.0 +grpcio==1.36.1 google-auth==1.27.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 797f7b7aa600418373ca859c9a71607f6dfdb6d3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 5 Mar 2021 07:40:25 +0100 Subject: [PATCH 035/158] chore(deps): update dependency google-auth to v1.27.1 (#141) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 353f3bbf1a8b..a3ce15d68e19 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.36.1 -google-auth==1.27.0 +google-auth==1.27.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.36.1 From 1a99aab0337f2ee6a1779f012c256a7da0351616 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 5 Mar 2021 08:06:02 +0100 Subject: [PATCH 036/158] chore(deps): update dependency google-cloud-dataproc to v2.3.0 (#143) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | `==2.2.0` -> `==2.3.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/2.3.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/2.3.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/2.3.0/compatibility-slim/2.2.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/2.3.0/confidence-slim/2.2.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-dataproc ### [`v2.3.0`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#​230-httpswwwgithubcomgoogleapispython-dataproccomparev220v230-2021-03-01) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v2.2.0...v2.3.0) ##### Features - **v1beta1:** BREAKING CHANGE: remove DOCKER/FLINK from Component enum; adds HBASE ([#​108](https://www.github.com/googleapis/python-dataproc/issues/108)) ([ee093a8](https://www.github.com/googleapis/python-dataproc/commit/ee093a88841c7f9c9ea41b066993e56b4abe267d)) ##### Bug Fixes - remove gRPC send/recv limits; expose client transport ([#​117](https://www.github.com/googleapis/python-dataproc/issues/117)) ([6f27109](https://www.github.com/googleapis/python-dataproc/commit/6f27109faf03dd13f25294e57960f0d9e1a9fa27))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a3ce15d68e19..4bf59759d642 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==1.27.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.36.1 -google-cloud-dataproc==2.2.0 +google-cloud-dataproc==2.3.0 From e402a9298e4baf40345ce7e4baf0a23f1e9f865a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 11 Mar 2021 22:56:44 +0100 Subject: [PATCH 037/158] chore(deps): update dependency google-cloud-storage to v1.36.2 (#144) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 4bf59759d642..e20f7f0c07bc 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.36.1 google-auth==1.27.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.36.1 +google-cloud-storage==1.36.2 google-cloud-dataproc==2.3.0 From 84489af2c97540fef3d635a33ac7a5b0aaed83b3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 17 Mar 2021 18:17:58 +0100 Subject: [PATCH 038/158] chore(deps): update dependency google-auth to v1.28.0 (#146) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e20f7f0c07bc..6812b60dd72d 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.36.1 -google-auth==1.27.1 +google-auth==1.28.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.36.2 From 063c5c9cd374147858fee99547d9bb04e0e09876 Mon Sep 17 00:00:00 2001 From: Brad Miro Date: Fri, 26 Mar 2021 01:54:03 -0400 Subject: [PATCH 039/158] fix: (samples) fixing samples for new machine types (#150) Dataproc 2.0 is the new default image which does not support n1-standard-1. Fixes https://github.com/googleapis/python-dataproc/issues/147, https://github.com/googleapis/python-dataproc/issues/148, https://github.com/googleapis/python-dataproc/issues/149 --- dataproc/snippets/create_cluster.py | 4 ++-- dataproc/snippets/quickstart/quickstart.py | 4 ++-- dataproc/snippets/submit_job_test.py | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/dataproc/snippets/create_cluster.py b/dataproc/snippets/create_cluster.py index 732b349ae03e..f4fee7d8f90e 100644 --- a/dataproc/snippets/create_cluster.py +++ b/dataproc/snippets/create_cluster.py @@ -47,8 +47,8 @@ def create_cluster(project_id, region, cluster_name): "project_id": project_id, "cluster_name": cluster_name, "config": { - "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, - "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-2"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-2"}, }, } diff --git a/dataproc/snippets/quickstart/quickstart.py b/dataproc/snippets/quickstart/quickstart.py index 45e7f348cacf..c9e73002665d 100644 --- a/dataproc/snippets/quickstart/quickstart.py +++ b/dataproc/snippets/quickstart/quickstart.py @@ -44,8 +44,8 @@ def quickstart(project_id, region, cluster_name, job_file_path): "project_id": project_id, "cluster_name": cluster_name, "config": { - "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, - "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-2"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-2"}, }, } diff --git a/dataproc/snippets/submit_job_test.py b/dataproc/snippets/submit_job_test.py index 0cad6248bf18..326b38d5a301 100644 --- a/dataproc/snippets/submit_job_test.py +++ b/dataproc/snippets/submit_job_test.py @@ -30,11 +30,11 @@ 'config': { 'master_config': { 'num_instances': 1, - 'machine_type_uri': 'n1-standard-1' + 'machine_type_uri': 'n1-standard-2' }, 'worker_config': { 'num_instances': 2, - 'machine_type_uri': 'n1-standard-1' + 'machine_type_uri': 'n1-standard-2' } } } From 69833e044a946b2788540773bdc53eca5d06afde Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 27 Mar 2021 05:07:05 +0100 Subject: [PATCH 040/158] chore(deps): update dependency google-cloud-storage to v1.37.0 (#153) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 6812b60dd72d..73831643d340 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.36.1 google-auth==1.28.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.36.2 +google-cloud-storage==1.37.0 google-cloud-dataproc==2.3.0 From 2b11b22d4bc82adf1b934407a3a0cf5a16d53635 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 2 Apr 2021 07:34:17 +0200 Subject: [PATCH 041/158] chore(deps): update dependency google-cloud-dataproc to v2.3.1 (#154) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 73831643d340..16c2047b1d69 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==1.28.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.37.0 -google-cloud-dataproc==2.3.0 +google-cloud-dataproc==2.3.1 From f2fb618c949a7458ae7160771fe5e5f673c729d3 Mon Sep 17 00:00:00 2001 From: Yoshi Automation Bot Date: Mon, 5 Apr 2021 11:43:30 -0700 Subject: [PATCH 042/158] fix: use correct retry deadlines (#122) feat: add 'from_service_account_info' factory to clients fix: fix sphinx identifiers --- dataproc/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index bca0522ec4d9..97bf7da80e39 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -85,7 +85,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to tested samples. -ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8"] +ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8", "3.9"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG['ignored_versions'] From 7280790fe1d5f629d012ef73186193d4737776bf Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 8 Apr 2021 20:32:05 +0200 Subject: [PATCH 043/158] chore(deps): update dependency grpcio to v1.37.0 (#163) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 16c2047b1d69..87accec00229 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.10.0 -grpcio==1.36.1 +grpcio==1.37.0 google-auth==1.28.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 1e64f91d99523f08fa06d43731a0486d95263e45 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 8 Apr 2021 20:34:11 +0200 Subject: [PATCH 044/158] chore(deps): update dependency google-cloud-storage to v1.37.1 (#159) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 87accec00229..95510171555e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.37.0 google-auth==1.28.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.37.0 +google-cloud-storage==1.37.1 google-cloud-dataproc==2.3.1 From 2a3ee6ff16584000f6369032d01b7700680b2f35 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 10 Apr 2021 07:42:25 +0200 Subject: [PATCH 045/158] chore(deps): update dependency google-auth to v1.28.1 (#165) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 95510171555e..d0ab77d99bc1 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.37.0 -google-auth==1.28.0 +google-auth==1.28.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.37.1 From d14892195a4aa49b92021a5c58a9086a37fd85ff Mon Sep 17 00:00:00 2001 From: Anthonios Partheniou Date: Fri, 23 Apr 2021 12:10:03 -0400 Subject: [PATCH 046/158] chore: migrate to owl bot (#171) --- dataproc/snippets/noxfile.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 97bf7da80e39..956cdf4f9250 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -172,10 +172,16 @@ def blacken(session: nox.sessions.Session) -> None: def _session_tests(session: nox.sessions.Session, post_install: Callable = None) -> None: """Runs py.test for a particular project.""" if os.path.exists("requirements.txt"): - session.install("-r", "requirements.txt") + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") if os.path.exists("requirements-test.txt"): - session.install("-r", "requirements-test.txt") + if os.path.exists("constraints-test.txt"): + session.install("-r", "requirements-test.txt", "-c", "constraints-test.txt") + else: + session.install("-r", "requirements-test.txt") if INSTALL_LIBRARY_FROM_SOURCE: session.install("-e", _get_repo_root()) From 1355a07e2d3848aaeb64cd6ee7a93aca36f26fe8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 28 Apr 2021 19:55:53 +0200 Subject: [PATCH 047/158] chore(deps): update dependency google-auth to v1.30.0 (#169) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d0ab77d99bc1..d8463063edc1 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.37.0 -google-auth==1.28.1 +google-auth==1.30.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.37.1 From 3e8c65c8772d9dfeba0d298e49e4aa8ced15ee6f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 14 May 2021 03:06:27 +0200 Subject: [PATCH 048/158] chore(deps): update dependency grpcio to v1.37.1 (#182) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d8463063edc1..53ace4407cf0 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.10.0 -grpcio==1.37.0 +grpcio==1.37.1 google-auth==1.30.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From d3f6998fb90cea454de22f05ce346496a4e6e658 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 14 May 2021 03:25:38 +0200 Subject: [PATCH 049/158] chore(deps): update dependency pytest to v6.2.4 (#172) --- dataproc/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 7e460c8c866e..95ea1e6a02b0 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1 +1 @@ -pytest==6.0.1 +pytest==6.2.4 From 417756ea58b559c2443d34df595ebe91cfe3431a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 14 May 2021 07:15:19 +0200 Subject: [PATCH 050/158] chore(deps): update dependency google-cloud-storage to v1.38.0 (#174) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 53ace4407cf0..e68ba05fe4ea 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.37.1 google-auth==1.30.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.37.1 +google-cloud-storage==1.38.0 google-cloud-dataproc==2.3.1 From cae55bf1bbe0b55a854c2b289b1716dcefb5668c Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Sat, 22 May 2021 09:18:08 +0000 Subject: [PATCH 051/158] chore: new owl bot post processor docker image (#193) gcr.io/repo-automation-bots/owlbot-python:latest@sha256:3c3a445b3ddc99ccd5d31edc4b4519729635d20693900db32c4f587ed51f7479 --- dataproc/snippets/noxfile.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 956cdf4f9250..5ff9e1db5808 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -50,7 +50,10 @@ # to use your own Cloud project. 'gcloud_project_env': 'GOOGLE_CLOUD_PROJECT', # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', - + # If you need to use a specific version of pip, + # change pip_version_override to the string representation + # of the version number, for example, "20.2.4" + "pip_version_override": None, # A dictionary you want to inject into your test. Don't put any # secrets here. These values will override predefined values. 'envs': {}, @@ -170,6 +173,9 @@ def blacken(session: nox.sessions.Session) -> None: def _session_tests(session: nox.sessions.Session, post_install: Callable = None) -> None: + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") """Runs py.test for a particular project.""" if os.path.exists("requirements.txt"): if os.path.exists("constraints.txt"): From 9500ba72978bd64d8c36ad56315719d5b9891948 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 26 May 2021 20:11:54 +0200 Subject: [PATCH 052/158] chore(deps): update dependency google-auth to v1.30.1 (#195) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e68ba05fe4ea..87176d291b21 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.37.1 -google-auth==1.30.0 +google-auth==1.30.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.38.0 From 122e30300389043eb88cd80dacfd7a2c7490b9f6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 26 May 2021 20:12:35 +0200 Subject: [PATCH 053/158] chore(deps): update dependency google-cloud-dataproc to v2.4.0 (#191) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 87176d291b21..50ac2320dfe3 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==1.30.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.38.0 -google-cloud-dataproc==2.3.1 +google-cloud-dataproc==2.4.0 From c608499393eda31282c4cc6b0eb62b7a7240c330 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 9 Jun 2021 07:12:07 +0200 Subject: [PATCH 054/158] chore(deps): update dependency google-auth to v1.30.2 (#197) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 50ac2320dfe3..766478b46f2b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.37.1 -google-auth==1.30.1 +google-auth==1.30.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.38.0 From 658993b6587dfcd519e27d03f4e07353a0c68cb8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 16 Jun 2021 13:44:01 +0200 Subject: [PATCH 055/158] chore(deps): update dependency grpcio to v1.38.0 (#190) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [grpcio](https://grpc.io) | `==1.37.1` -> `==1.38.0` | [![age](https://badges.renovateapi.com/packages/pypi/grpcio/1.38.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/grpcio/1.38.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/grpcio/1.38.0/compatibility-slim/1.37.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/grpcio/1.38.0/confidence-slim/1.37.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻️ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 766478b46f2b..6080ae3ee43d 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.10.0 -grpcio==1.37.1 +grpcio==1.38.0 google-auth==1.30.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From eb24de00bdb2331771db33eada488412d98ebbc8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 16 Jun 2021 14:44:02 +0200 Subject: [PATCH 056/158] chore(deps): update dependency google-auth to v1.31.0 (#198) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | `==1.30.2` -> `==1.31.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-auth/1.31.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-auth/1.31.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-auth/1.31.0/compatibility-slim/1.30.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-auth/1.31.0/confidence-slim/1.30.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.31.0`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1310-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1302v1310-2021-06-09) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.30.2...v1.31.0) ##### Features - define useful properties on `google.auth.external_account.Credentials` ([#​770](https://www.github.com/googleapis/google-auth-library-python/issues/770)) ([f97499c](https://www.github.com/googleapis/google-auth-library-python/commit/f97499c718af70d17c17e0c58d6381273eceabcd)) ##### Bug Fixes - avoid deleting items while iterating ([#​772](https://www.github.com/googleapis/google-auth-library-python/issues/772)) ([a5e6b65](https://www.github.com/googleapis/google-auth-library-python/commit/a5e6b651aa8ad407ce087fe32f40b46925bae527)) ##### [1.30.2](https://www.github.com/googleapis/google-auth-library-python/compare/v1.30.1...v1.30.2) (2021-06-03) ##### Bug Fixes - **dependencies:** add urllib3 and requests to aiohttp extra ([#​755](https://www.github.com/googleapis/google-auth-library-python/issues/755)) ([a923442](https://www.github.com/googleapis/google-auth-library-python/commit/a9234423cb2b69068fc0d30a5a0ee86a599ab8b7)) - enforce constraints during unit tests ([#​760](https://www.github.com/googleapis/google-auth-library-python/issues/760)) ([1a6496a](https://www.github.com/googleapis/google-auth-library-python/commit/1a6496abfc17ab781bfa485dc74d0f7dbbe0c44b)), closes [#​759](https://www.github.com/googleapis/google-auth-library-python/issues/759) - session object was never used in aiohttp request ([#​700](https://www.github.com/googleapis/google-auth-library-python/issues/700)) ([#​701](https://www.github.com/googleapis/google-auth-library-python/issues/701)) ([09e0389](https://www.github.com/googleapis/google-auth-library-python/commit/09e0389db72cc9d6c5dde34864cb54d717dc0b92)) ##### [1.30.1](https://www.github.com/googleapis/google-auth-library-python/compare/v1.30.0...v1.30.1) (2021-05-20) ##### Bug Fixes - allow user to customize context aware metadata path in \_mtls_helper ([#​754](https://www.github.com/googleapis/google-auth-library-python/issues/754)) ([e697687](https://www.github.com/googleapis/google-auth-library-python/commit/e6976879b392508c022610ab3ea2ea55c7089c63)) - fix function name in signing error message ([#​751](https://www.github.com/googleapis/google-auth-library-python/issues/751)) ([e9ca25f](https://www.github.com/googleapis/google-auth-library-python/commit/e9ca25fa39a112cc1a376388ab47a4e1b3ea746c))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 6080ae3ee43d..c814c49f2d16 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.38.0 -google-auth==1.30.2 +google-auth==1.31.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.38.0 From 3904a24d88a8bd155d4c0c32fa2dc784d8349e8b Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 23 Jun 2021 18:29:24 +0200 Subject: [PATCH 057/158] chore(deps): update dependency google-cloud-storage to v1.39.0 (#210) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index c814c49f2d16..2ddf2ec442f2 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.38.0 google-auth==1.31.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.38.0 +google-cloud-storage==1.39.0 google-cloud-dataproc==2.4.0 From a390a5cad27a7af59e6d1468c35120e77a781a73 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 23 Jun 2021 18:32:24 +0200 Subject: [PATCH 058/158] chore(deps): update dependency grpcio to v1.38.1 (#208) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 2ddf2ec442f2..e7a9796f9780 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.10.0 -grpcio==1.38.0 +grpcio==1.38.1 google-auth==1.31.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From bba3d48a7959253512f58c96528fb73f404d0f8c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 26 Jun 2021 01:30:27 +0200 Subject: [PATCH 059/158] chore(deps): update dependency google-auth to v1.32.0 (#207) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | `==1.31.0` -> `==1.32.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-auth/1.32.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-auth/1.32.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-auth/1.32.0/compatibility-slim/1.31.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-auth/1.32.0/confidence-slim/1.31.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.32.0`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1320-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1310v1320-2021-06-16) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.31.0...v1.32.0) ##### Features - allow scopes for self signed jwt ([#​776](https://www.github.com/googleapis/google-auth-library-python/issues/776)) ([2cfe655](https://www.github.com/googleapis/google-auth-library-python/commit/2cfe655bba837170abc07701557a1a5e0fe3294e))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e7a9796f9780..158859b08e14 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.38.1 -google-auth==1.31.0 +google-auth==1.32.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.39.0 From 5dec9779f8970b6ff453e8969e5d5b4980156111 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 1 Jul 2021 07:55:11 +0200 Subject: [PATCH 060/158] chore(deps): update dependency google-auth to v1.32.1 (#217) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 158859b08e14..50dd477bceaa 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.10.0 grpcio==1.38.1 -google-auth==1.32.0 +google-auth==1.32.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.39.0 From 531d5c3ad6d9ac285855251d686ebf10a87b80c8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 1 Jul 2021 11:20:27 +0200 Subject: [PATCH 061/158] chore(deps): update dependency google-cloud-storage to v1.40.0 (#216) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.39.0` -> `==1.40.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.40.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.40.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.40.0/compatibility-slim/1.39.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.40.0/confidence-slim/1.39.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-storage ### [`v1.40.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1400-httpswwwgithubcomgoogleapispython-storagecomparev1390v1400-2021-06-30) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.39.0...v1.40.0) ##### Features - add preconditions and retry configuration to blob.create_resumable_upload_session ([#​484](https://www.github.com/googleapis/python-storage/issues/484)) ([0ae35ee](https://www.github.com/googleapis/python-storage/commit/0ae35eef0fe82fe60bc095c4b183102bb1dabeeb)) - add public access prevention to bucket IAM configuration ([#​304](https://www.github.com/googleapis/python-storage/issues/304)) ([e3e57a9](https://www.github.com/googleapis/python-storage/commit/e3e57a9c779d6b87852063787f19e27c76b1bb14)) ##### Bug Fixes - replace default retry for upload operations ([#​480](https://www.github.com/googleapis/python-storage/issues/480)) ([c027ccf](https://www.github.com/googleapis/python-storage/commit/c027ccf4279fb05e041754294f10744b7d81beea))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 50dd477bceaa..e025ed1dfd67 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.38.1 google-auth==1.32.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.39.0 +google-cloud-storage==1.40.0 google-cloud-dataproc==2.4.0 From 5516b72cb1b44f2cc3b8a0a1e809ebb2dd4ea5a2 Mon Sep 17 00:00:00 2001 From: vikrant-sinha <63613816+vikrant-sinha@users.noreply.github.com> Date: Sat, 3 Jul 2021 00:09:58 +0530 Subject: [PATCH 062/158] fix: Attribute error Name while executing the sample code (#205) * Fix attribute error `Name` while getting the status of cluster `cluster.status.State.Name(cluster.status.state),` this code will throw an attribute error when executed because it's trying to access State.Name which is not a function of class ClusterStatus.State A better approach will be to use the `cluster.status.state.name` as State is of enum type * fix:attribute error Name while getting the status of cluster --- dataproc/snippets/list_clusters.py | 2 +- dataproc/snippets/submit_job_to_cluster.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py index f0e7bac348e8..916f1a541f9f 100644 --- a/dataproc/snippets/list_clusters.py +++ b/dataproc/snippets/list_clusters.py @@ -32,7 +32,7 @@ def list_clusters(dataproc, project, region): ( "{} - {}".format( cluster.cluster_name, - cluster.status.State.Name(cluster.status.state), + cluster.status.state.name ) ) ) diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index d613cf5b8609..35d329c550aa 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -137,7 +137,7 @@ def list_clusters_with_details(dataproc, project, region): ( "{} - {}".format( cluster.cluster_name, - cluster.status.State.Name(cluster.status.state), + cluster.status.state.name, ) ) ) From f16f4391321df19e663868d9ff056973a88773af Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 12 Jul 2021 21:42:11 +0200 Subject: [PATCH 063/158] chore(deps): update dependency backoff to v1.11.0 (#220) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [backoff](https://togithub.com/litl/backoff) | `==1.10.0` -> `==1.11.0` | [![age](https://badges.renovateapi.com/packages/pypi/backoff/1.11.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/backoff/1.11.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/backoff/1.11.0/compatibility-slim/1.10.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/backoff/1.11.0/confidence-slim/1.10.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
litl/backoff ### [`v1.11.0`](https://togithub.com/litl/backoff/blob/master/CHANGELOG.md#v1110-2021-07-12) [Compare Source](https://togithub.com/litl/backoff/compare/v1.10.0...v1.11.0) ##### Changed - Configurable logging levels for backoff and giveup events - Minor documentation fixes ##### NOTE THIS WILL BE THE FINAL PYTHON 2.7 COMPATIBLE RELEASE.
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e025ed1dfd67..5badeceb29e6 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ -backoff==1.10.0 +backoff==1.11.0 grpcio==1.38.1 google-auth==1.32.1 google-auth-httplib2==0.1.0 From 83f7a08de0989d943ac9170aafa8a8fce599acc8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 14 Jul 2021 23:56:37 +0200 Subject: [PATCH 064/158] chore(deps): update dependency google-auth to v1.33.0 (#226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | `==1.32.1` -> `==1.33.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-auth/1.33.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-auth/1.33.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-auth/1.33.0/compatibility-slim/1.32.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-auth/1.33.0/confidence-slim/1.32.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-auth-library-python ### [`v1.33.0`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​1330-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1321v1330-2021-07-14) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.32.1...v1.33.0) ##### Features - define `CredentialAccessBoundary` classes ([#​793](https://www.github.com/googleapis/google-auth-library-python/issues/793)) ([d883921](https://www.github.com/googleapis/google-auth-library-python/commit/d883921ae8fdc92b2c2cf1b3a5cd389e1287eb60)) - define `google.auth.downscoped.Credentials` class ([#​801](https://www.github.com/googleapis/google-auth-library-python/issues/801)) ([2f5c3a6](https://www.github.com/googleapis/google-auth-library-python/commit/2f5c3a636192c20cf4c92c3831d1f485031d24d2)) - service account is able to use a private token endpoint ([#​784](https://www.github.com/googleapis/google-auth-library-python/issues/784)) ([0e26409](https://www.github.com/googleapis/google-auth-library-python/commit/0e264092e35ac02ad68d5d91424ecba5397daa41)) ##### Bug Fixes - fix fetch_id_token credential lookup order to match adc ([#​748](https://www.github.com/googleapis/google-auth-library-python/issues/748)) ([c34452e](https://www.github.com/googleapis/google-auth-library-python/commit/c34452ef450c42cfef37a1b0c548bb422302dd5d)) ##### Documentation - fix code block formatting in 'user-guide.rst' ([#​794](https://www.github.com/googleapis/google-auth-library-python/issues/794)) ([4fd84bd](https://www.github.com/googleapis/google-auth-library-python/commit/4fd84bdf43694af5107dc8c8b443c06ba2f61d2c)) ##### [1.32.1](https://www.github.com/googleapis/google-auth-library-python/compare/v1.32.0...v1.32.1) (2021-06-30) ##### Bug Fixes - avoid leaking sub-session created for '\_auth_request' ([#​789](https://www.github.com/googleapis/google-auth-library-python/issues/789)) ([2079ab5](https://www.github.com/googleapis/google-auth-library-python/commit/2079ab5e1db464f502248ae4f9e424deeef87fb2))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 5badeceb29e6..b94615752332 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.0 grpcio==1.38.1 -google-auth==1.32.1 +google-auth==1.33.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.40.0 From 90323b8886f23dda3cfdd7791ccf6af88d0c44cf Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 15 Jul 2021 15:22:22 +0200 Subject: [PATCH 065/158] chore(deps): update dependency google-cloud-storage to v1.41.0 (#222) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.40.0` -> `==1.41.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.41.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.41.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.41.0/compatibility-slim/1.40.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.41.0/confidence-slim/1.40.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-storage ### [`v1.41.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1410-httpswwwgithubcomgoogleapispython-storagecomparev1400v1410-2021-07-13) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.40.0...v1.41.0) ##### Features - add support for Etag headers on reads ([#​489](https://www.github.com/googleapis/python-storage/issues/489)) ([741d3fd](https://www.github.com/googleapis/python-storage/commit/741d3fda4e4280022cede29ebeb7c2ea09e73b6f)) ##### Bug Fixes - **deps:** update minimum dependency versions to pick up bugfixes ([#​496](https://www.github.com/googleapis/python-storage/issues/496)) ([92251a5](https://www.github.com/googleapis/python-storage/commit/92251a5c8ea4d663773506eb1c630201a657aa69)), closes [#​494](https://www.github.com/googleapis/python-storage/issues/494) - populate etag / generation / metageneration properties during download ([#​488](https://www.github.com/googleapis/python-storage/issues/488)) ([49ba14c](https://www.github.com/googleapis/python-storage/commit/49ba14c9c47dbe6bc2bb45d53bbe5621c131fbcb)) - revise and rename is_etag_in_json(data) ([#​483](https://www.github.com/googleapis/python-storage/issues/483)) ([0a52546](https://www.github.com/googleapis/python-storage/commit/0a5254647bf1155874fe48f3891bcc34a76b0b81))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index b94615752332..0b1232c50e8e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.38.1 google-auth==1.33.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.40.0 +google-cloud-storage==1.41.0 google-cloud-dataproc==2.4.0 From dcb2ca1785243e43cf8b61206f602ceecfff6d27 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 16 Jul 2021 15:12:08 +0200 Subject: [PATCH 066/158] chore(deps): update dependency backoff to v1.11.1 (#224) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [backoff](https://togithub.com/litl/backoff) | `==1.11.0` -> `==1.11.1` | [![age](https://badges.renovateapi.com/packages/pypi/backoff/1.11.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/backoff/1.11.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/backoff/1.11.1/compatibility-slim/1.11.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/backoff/1.11.1/confidence-slim/1.11.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
litl/backoff ### [`v1.11.1`](https://togithub.com/litl/backoff/blob/master/CHANGELOG.md#v1111-2021-07-14) [Compare Source](https://togithub.com/litl/backoff/compare/v1.11.0...v1.11.1) ##### Changed - Update **version** in backoff module
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 0b1232c50e8e..d11b78dff530 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ -backoff==1.11.0 +backoff==1.11.1 grpcio==1.38.1 google-auth==1.33.0 google-auth-httplib2==0.1.0 From 38a4b34e8f8187583cb4467d79eabb98867642b4 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 21 Jul 2021 19:36:51 +0200 Subject: [PATCH 067/158] chore(deps): update dependency google-cloud-storage to v1.41.1 (#230) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d11b78dff530..1262db7aea5a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.38.1 google-auth==1.33.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.41.0 +google-cloud-storage==1.41.1 google-cloud-dataproc==2.4.0 From 5f5b2b750f1c0308ca9d69ed0f86a0e3baae7478 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 21 Jul 2021 19:39:00 +0200 Subject: [PATCH 068/158] chore(deps): update dependency google-auth to v1.33.1 (#229) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 1262db7aea5a..cbf58ee8ffbb 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.38.1 -google-auth==1.33.0 +google-auth==1.33.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.41.1 From 309bddd81a079da4ad658aeb2ff87a84b5186820 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Thu, 22 Jul 2021 13:50:27 +0000 Subject: [PATCH 069/158] feat: add Samples section to CONTRIBUTING.rst (#228) Source-Link: https://github.com/googleapis/synthtool/commit/52e4e46eff2a0b70e3ff5506a02929d089d077d4 Post-Processor: gcr.io/repo-automation-bots/owlbot-python:latest@sha256:6186535cbdbf6b9fe61f00294929221d060634dae4a0795c1cefdbc995b2d605 --- dataproc/snippets/noxfile.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 5ff9e1db5808..6a8ccdae22c9 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -28,8 +28,9 @@ # WARNING - WARNING - WARNING - WARNING - WARNING # WARNING - WARNING - WARNING - WARNING - WARNING -# Copy `noxfile_config.py` to your directory and modify it instead. +BLACK_VERSION = "black==19.10b0" +# Copy `noxfile_config.py` to your directory and modify it instead. # `TEST_CONFIG` dict is a configuration hook that allows users to # modify the test configurations. The values here should be in sync @@ -159,7 +160,7 @@ def lint(session: nox.sessions.Session) -> None: @nox.session def blacken(session: nox.sessions.Session) -> None: - session.install("black") + session.install(BLACK_VERSION) python_files = [path for path in os.listdir(".") if path.endswith(".py")] session.run("black", *python_files) From 130035660055a1b714e3b58177152378a0eba1f7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 22 Jul 2021 22:07:29 +0200 Subject: [PATCH 070/158] chore(deps): update dependency grpcio to v1.39.0 (#231) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index cbf58ee8ffbb..5e45fca3bf08 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.11.1 -grpcio==1.38.1 +grpcio==1.39.0 google-auth==1.33.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 82685988525788abb9a656f6c86360e33dbe1556 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 27 Jul 2021 22:24:21 +0200 Subject: [PATCH 071/158] chore(deps): update dependency google-auth to v1.34.0 (#235) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 5e45fca3bf08..072f957c40b8 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.39.0 -google-auth==1.33.1 +google-auth==1.34.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.41.1 From a8c21b2368a075820c12cff753e84bdea4abd5b0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 28 Jul 2021 01:23:10 +0200 Subject: [PATCH 072/158] chore(deps): update dependency google-cloud-dataproc to v2.5.0 (#234) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 072f957c40b8..6b6cf3a344f8 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==1.34.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.41.1 -google-cloud-dataproc==2.4.0 +google-cloud-dataproc==2.5.0 From a40774cf34424b6d2c37b6b1f9bc4ebb130041e1 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 11 Aug 2021 16:32:33 +0000 Subject: [PATCH 073/158] chore: fix INSTALL_LIBRARY_FROM_SOURCE in noxfile.py (#240) Source-Link: https://github.com/googleapis/synthtool/commit/6252f2cd074c38f37b44abe5e96d128733eb1b61 Post-Processor: gcr.io/repo-automation-bots/owlbot-python:latest@sha256:50e35228649c47b6ca82aa0be3ff9eb2afce51c82b66c4a03fe4afeb5ff6c0fc --- dataproc/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 6a8ccdae22c9..125bb619cc49 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -96,7 +96,7 @@ def get_pytest_env_vars() -> Dict[str, str]: TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) -INSTALL_LIBRARY_FROM_SOURCE = bool(os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False)) +INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ("True", "true") # # Style Checks # From c8ba31b82b65bc0346dd31d2ca877c19c02fc4ae Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 12 Aug 2021 16:57:20 +0200 Subject: [PATCH 074/158] chore(deps): update dependency google-cloud-storage to v1.42.0 (#242) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 6b6cf3a344f8..0679e224ce44 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.39.0 google-auth==1.34.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.41.1 +google-cloud-storage==1.42.0 google-cloud-dataproc==2.5.0 From 14cf1254a2e70a635e3e79ecb62af808e14f04a7 Mon Sep 17 00:00:00 2001 From: Lo Ferris <50979514+loferris@users.noreply.github.com> Date: Thu, 12 Aug 2021 14:00:31 -0700 Subject: [PATCH 075/158] docs: update cluster sample (#218) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * first steps in adding sample * consistent formatting with create_cluster.py * test first draft * update_cluster sample complete * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * docs: add update cluster sample - fixing formatting * Update samples/snippets/update_cluster.py Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> * Update samples/snippets/update_cluster.py Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> * updated test, still fine-tuning * added get_cluster to test * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * another attempt at writing test * new test pattern * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * updated static for new_num_instances and fixed linting error Co-authored-by: Owl Bot Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> --- dataproc/snippets/create_cluster.py | 10 +-- .../instantiate_inline_workflow_template.py | 8 +- dataproc/snippets/list_clusters.py | 6 +- dataproc/snippets/submit_job.py | 24 +++--- dataproc/snippets/submit_job_test.py | 42 +++++----- dataproc/snippets/submit_job_to_cluster.py | 12 ++- dataproc/snippets/update_cluster.py | 78 ++++++++++++++++++ dataproc/snippets/update_cluster_test.py | 80 +++++++++++++++++++ 8 files changed, 212 insertions(+), 48 deletions(-) create mode 100644 dataproc/snippets/update_cluster.py create mode 100644 dataproc/snippets/update_cluster_test.py diff --git a/dataproc/snippets/create_cluster.py b/dataproc/snippets/create_cluster.py index f4fee7d8f90e..633b59e8cf4c 100644 --- a/dataproc/snippets/create_cluster.py +++ b/dataproc/snippets/create_cluster.py @@ -29,12 +29,12 @@ def create_cluster(project_id, region, cluster_name): """This sample walks a user through creating a Cloud Dataproc cluster - using the Python client library. + using the Python client library. - Args: - project_id (string): Project to use for creating resources. - region (string): Region where the resources should live. - cluster_name (string): Name to use for creating a cluster. + Args: + project_id (string): Project to use for creating resources. + region (string): Region where the resources should live. + cluster_name (string): Name to use for creating a cluster. """ # Create a client with the endpoint set to the desired cluster region. diff --git a/dataproc/snippets/instantiate_inline_workflow_template.py b/dataproc/snippets/instantiate_inline_workflow_template.py index b3a40d133544..cbb1a2186e2d 100644 --- a/dataproc/snippets/instantiate_inline_workflow_template.py +++ b/dataproc/snippets/instantiate_inline_workflow_template.py @@ -27,11 +27,11 @@ def instantiate_inline_workflow_template(project_id, region): """This sample walks a user through submitting a workflow - for a Cloud Dataproc using the Python client library. + for a Cloud Dataproc using the Python client library. - Args: - project_id (string): Project to use for running the workflow. - region (string): Region where the workflow resources should live. + Args: + project_id (string): Project to use for running the workflow. + region (string): Region where the workflow resources should live. """ # Create a client with the endpoint set to the desired region. diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py index 916f1a541f9f..3ecb8aebc6db 100644 --- a/dataproc/snippets/list_clusters.py +++ b/dataproc/snippets/list_clusters.py @@ -49,8 +49,10 @@ def main(project_id, region): else: # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address="{}-dataproc.googleapis.com:443".format(region) + client_transport = ( + cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) ) dataproc_cluster_client = dataproc_v1.ClusterControllerClient(client_transport) diff --git a/dataproc/snippets/submit_job.py b/dataproc/snippets/submit_job.py index b70348c3220c..d7761b734c8c 100644 --- a/dataproc/snippets/submit_job.py +++ b/dataproc/snippets/submit_job.py @@ -23,8 +23,10 @@ # [START dataproc_submit_job] import re + # [END dataproc_submit_job] import sys + # [START dataproc_submit_job] from google.cloud import dataproc_v1 as dataproc @@ -33,21 +35,19 @@ def submit_job(project_id, region, cluster_name): # Create the job client. - job_client = dataproc.JobControllerClient(client_options={ - 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(region) - }) + job_client = dataproc.JobControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} + ) # Create the job config. 'main_jar_file_uri' can also be a # Google Cloud Storage URL. job = { - 'placement': { - 'cluster_name': cluster_name + "placement": {"cluster_name": cluster_name}, + "spark_job": { + "main_class": "org.apache.spark.examples.SparkPi", + "jar_file_uris": ["file:///usr/lib/spark/examples/jars/spark-examples.jar"], + "args": ["1000"], }, - 'spark_job': { - 'main_class': 'org.apache.spark.examples.SparkPi', - 'jar_file_uris': ['file:///usr/lib/spark/examples/jars/spark-examples.jar'], - 'args': ['1000'] - } } operation = job_client.submit_job_as_operation( @@ -67,12 +67,14 @@ def submit_job(project_id, region, cluster_name): ) print(f"Job finished successfully: {output}") + + # [END dataproc_submit_job] if __name__ == "__main__": if len(sys.argv) < 3: - sys.exit('python submit_job.py project_id region cluster_name') + sys.exit("python submit_job.py project_id region cluster_name") project_id = sys.argv[1] region = sys.argv[2] diff --git a/dataproc/snippets/submit_job_test.py b/dataproc/snippets/submit_job_test.py index 326b38d5a301..6827916fbff9 100644 --- a/dataproc/snippets/submit_job_test.py +++ b/dataproc/snippets/submit_job_test.py @@ -21,30 +21,24 @@ import submit_job -PROJECT_ID = os.environ['GOOGLE_CLOUD_PROJECT'] -REGION = 'us-central1' -CLUSTER_NAME = 'py-sj-test-{}'.format(str(uuid.uuid4())) +PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"] +REGION = "us-central1" +CLUSTER_NAME = "py-sj-test-{}".format(str(uuid.uuid4())) CLUSTER = { - 'project_id': PROJECT_ID, - 'cluster_name': CLUSTER_NAME, - 'config': { - 'master_config': { - 'num_instances': 1, - 'machine_type_uri': 'n1-standard-2' - }, - 'worker_config': { - 'num_instances': 2, - 'machine_type_uri': 'n1-standard-2' - } - } + "project_id": PROJECT_ID, + "cluster_name": CLUSTER_NAME, + "config": { + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-2"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-2"}, + }, } @pytest.fixture(autouse=True) def setup_teardown(): - cluster_client = dataproc.ClusterControllerClient(client_options={ - 'api_endpoint': '{}-dataproc.googleapis.com:443'.format(REGION) - }) + cluster_client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION)} + ) # Create the cluster. operation = cluster_client.create_cluster( @@ -54,13 +48,17 @@ def setup_teardown(): yield - cluster_client.delete_cluster(request={ - "project_id": PROJECT_ID, "region": REGION, "cluster_name": CLUSTER_NAME - }) + cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) def test_submit_job(capsys): submit_job.submit_job(PROJECT_ID, REGION, CLUSTER_NAME) out, _ = capsys.readouterr() - assert 'Job finished successfully' in out + assert "Job finished successfully" in out diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index 35d329c550aa..68a547c481fd 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -77,8 +77,10 @@ def download_output(project, cluster_id, output_bucket, job_id): print("Downloading output file.") client = storage.Client(project=project) bucket = client.get_bucket(output_bucket) - output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( - cluster_id, job_id + output_blob = ( + "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( + cluster_id, job_id + ) ) return bucket.blob(output_blob).download_as_string() @@ -230,8 +232,10 @@ def main( region = get_region_from_zone(zone) # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address="{}-dataproc.googleapis.com:443".format(region) + client_transport = ( + cluster_controller_grpc_transport.ClusterControllerGrpcTransport( + address="{}-dataproc.googleapis.com:443".format(region) + ) ) job_transport = job_controller_grpc_transport.JobControllerGrpcTransport( address="{}-dataproc.googleapis.com:443".format(region) diff --git a/dataproc/snippets/update_cluster.py b/dataproc/snippets/update_cluster.py new file mode 100644 index 000000000000..f452022426cf --- /dev/null +++ b/dataproc/snippets/update_cluster.py @@ -0,0 +1,78 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This sample walks a user through updating the number of clusters using the Dataproc +# client library. + +# Usage: +# python update_cluster.py --project_id --region --cluster_name + +import sys + +# [START dataproc_update_cluster] +from google.cloud import dataproc_v1 as dataproc + + +def update_cluster(project_id, region, cluster_name, new_num_instances): + """This sample walks a user through updating a Cloud Dataproc cluster + using the Python client library. + + Args: + project_id (str): Project to use for creating resources. + region (str): Region where the resources should live. + cluster_name (str): Name to use for creating a cluster. + """ + + # Create a client with the endpoint set to the desired cluster region. + client = dataproc.ClusterControllerClient( + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} + ) + + # Get cluster you wish to update. + cluster = client.get_cluster( + project_id=project_id, region=region, cluster_name=cluster_name + ) + + # Update number of clusters + mask = {"paths": {"config.worker_config.num_instances": str(new_num_instances)}} + + # Update cluster config + cluster.config.worker_config.num_instances = new_num_instances + + # Update cluster + operation = client.update_cluster( + project_id=project_id, + region=region, + cluster=cluster, + cluster_name=cluster_name, + update_mask=mask, + ) + + # Output a success message. + updated_cluster = operation.result() + print(f"Cluster was updated successfully: {updated_cluster.cluster_name}") + + +# [END dataproc_update_cluster] + + +if __name__ == "__main__": + if len(sys.argv) < 5: + sys.exit("python update_cluster.py project_id region cluster_name") + + project_id = sys.argv[1] + region = sys.argv[2] + cluster_name = sys.argv[3] + new_num_instances = sys.argv[4] + update_cluster(project_id, region, cluster_name) diff --git a/dataproc/snippets/update_cluster_test.py b/dataproc/snippets/update_cluster_test.py new file mode 100644 index 000000000000..9c608ac43525 --- /dev/null +++ b/dataproc/snippets/update_cluster_test.py @@ -0,0 +1,80 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This sample walks a user through updating the number of clusters using the Dataproc +# client library. + + +import os +import uuid + +from google.cloud.dataproc_v1.services.cluster_controller.client import ( + ClusterControllerClient, +) +import pytest + +import update_cluster + + +PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"] +REGION = "us-central1" +CLUSTER_NAME = f"py-cc-test-{str(uuid.uuid4())}" +NEW_NUM_INSTANCES = 5 +CLUSTER = { + "project_id": PROJECT_ID, + "cluster_name": CLUSTER_NAME, + "config": { + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-2"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-2"}, + }, +} + + +@pytest.fixture(autouse=True) +def setup_teardown(cluster_client): + # Create the cluster. + operation = cluster_client.create_cluster( + request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} + ) + operation.result() + + yield + + cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) + + +@pytest.fixture +def cluster_client(): + cluster_client = ClusterControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION)} + ) + return cluster_client + + +def test_update_cluster(capsys, cluster_client: ClusterControllerClient): + # Wrapper function for client library function + update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES) + new_num_cluster = cluster_client.get_cluster( + project_id=PROJECT_ID, region=REGION, cluster_name=CLUSTER_NAME + ) + + out, _ = capsys.readouterr() + assert CLUSTER_NAME in out + assert new_num_cluster.config.worker_config.num_instances == NEW_NUM_INSTANCES From 2a4f2fd121f4f27230055c1cf22e05ab019f254c Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 13 Aug 2021 15:26:13 +0000 Subject: [PATCH 076/158] chore: drop mention of Python 2.7 from templates (#244) Source-Link: https://github.com/googleapis/synthtool/commit/facee4cc1ea096cd8bcc008bb85929daa7c414c0 Post-Processor: gcr.io/repo-automation-bots/owlbot-python:latest@sha256:9743664022bd63a8084be67f144898314c7ca12f0a03e422ac17c733c129d803 chore(python): avoid .nox directories when building docs --- dataproc/snippets/noxfile.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 125bb619cc49..e73436a15626 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -39,7 +39,7 @@ TEST_CONFIG = { # You can opt out from the test for specific Python versions. - 'ignored_versions': ["2.7"], + 'ignored_versions': [], # Old samples are opted out of enforcing Python type hints # All new samples should feature them @@ -88,8 +88,8 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. -# All versions used to tested samples. -ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8", "3.9"] +# All versions used to test samples. +ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG['ignored_versions'] From 4e95279398402632f68435757437475072a882e1 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 17 Aug 2021 17:58:25 +0200 Subject: [PATCH 077/158] chore(deps): update dependency google-auth to v1.35.0 (#245) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 0679e224ce44..88afd46970ca 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.39.0 -google-auth==1.34.0 +google-auth==1.35.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.0 From 2a425457269436961c42018a1d25c6b1ccbeaaa8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 24 Aug 2021 19:33:32 +0200 Subject: [PATCH 078/158] chore(deps): update dependency google-auth to v2 (#246) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 88afd46970ca..8a6fe1c2f842 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.39.0 -google-auth==1.35.0 +google-auth==2.0.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.0 From 5e5ffd7223875629cf34c82e79df3c89d6ea7d78 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 31 Aug 2021 13:01:29 +0200 Subject: [PATCH 079/158] chore(deps): update dependency pytest to v6.2.5 (#252) --- dataproc/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 95ea1e6a02b0..927094516e65 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1 +1 @@ -pytest==6.2.4 +pytest==6.2.5 From 424862cf55dcbd4b55393f4650b4255f8be71b88 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 1 Sep 2021 12:58:21 +0200 Subject: [PATCH 080/158] chore(deps): update dependency google-auth to v2.0.2 (#254) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 8a6fe1c2f842..8bb60b956a08 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.39.0 -google-auth==2.0.1 +google-auth==2.0.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.0 From 9584f0c7475adc2b5500dda0825e87187325391a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 8 Sep 2021 16:54:25 +0200 Subject: [PATCH 081/158] chore(deps): update dependency grpcio to v1.40.0 (#263) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 8bb60b956a08..776ad46d0fb2 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.11.1 -grpcio==1.39.0 +grpcio==1.40.0 google-auth==2.0.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From a74fc45c774df9d23c211398c90d701653778d48 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 9 Sep 2021 16:42:08 +0200 Subject: [PATCH 082/158] chore(deps): update dependency google-cloud-storage to v1.42.1 (#264) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 776ad46d0fb2..a00c04bab2f3 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.40.0 google-auth==2.0.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.42.0 +google-cloud-storage==1.42.1 google-cloud-dataproc==2.5.0 From d834761aef8d96761cec490f0adea798c7b23c8f Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 17 Sep 2021 16:30:12 +0000 Subject: [PATCH 083/158] chore: blacken samples noxfile template (#266) --- dataproc/snippets/noxfile.py | 44 ++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index e73436a15626..b008613f03ff 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -39,17 +39,15 @@ TEST_CONFIG = { # You can opt out from the test for specific Python versions. - 'ignored_versions': [], - + "ignored_versions": [], # Old samples are opted out of enforcing Python type hints # All new samples should feature them - 'enforce_type_hints': False, - + "enforce_type_hints": False, # An envvar key for determining the project id to use. Change it # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a # build specific Cloud project. You can also use your own string # to use your own Cloud project. - 'gcloud_project_env': 'GOOGLE_CLOUD_PROJECT', + "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', # If you need to use a specific version of pip, # change pip_version_override to the string representation @@ -57,13 +55,13 @@ "pip_version_override": None, # A dictionary you want to inject into your test. Don't put any # secrets here. These values will override predefined values. - 'envs': {}, + "envs": {}, } try: # Ensure we can import noxfile_config in the project's directory. - sys.path.append('.') + sys.path.append(".") from noxfile_config import TEST_CONFIG_OVERRIDE except ImportError as e: print("No user noxfile_config found: detail: {}".format(e)) @@ -78,12 +76,12 @@ def get_pytest_env_vars() -> Dict[str, str]: ret = {} # Override the GCLOUD_PROJECT and the alias. - env_key = TEST_CONFIG['gcloud_project_env'] + env_key = TEST_CONFIG["gcloud_project_env"] # This should error out if not set. - ret['GOOGLE_CLOUD_PROJECT'] = os.environ[env_key] + ret["GOOGLE_CLOUD_PROJECT"] = os.environ[env_key] # Apply user supplied envs. - ret.update(TEST_CONFIG['envs']) + ret.update(TEST_CONFIG["envs"]) return ret @@ -92,11 +90,14 @@ def get_pytest_env_vars() -> Dict[str, str]: ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9"] # Any default versions that should be ignored. -IGNORED_VERSIONS = TEST_CONFIG['ignored_versions'] +IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) -INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ("True", "true") +INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ( + "True", + "true", +) # # Style Checks # @@ -141,7 +142,7 @@ def _determine_local_import_names(start_dir: str) -> List[str]: @nox.session def lint(session: nox.sessions.Session) -> None: - if not TEST_CONFIG['enforce_type_hints']: + if not TEST_CONFIG["enforce_type_hints"]: session.install("flake8", "flake8-import-order") else: session.install("flake8", "flake8-import-order", "flake8-annotations") @@ -150,9 +151,11 @@ def lint(session: nox.sessions.Session) -> None: args = FLAKE8_COMMON_ARGS + [ "--application-import-names", ",".join(local_names), - "." + ".", ] session.run("flake8", *args) + + # # Black # @@ -165,6 +168,7 @@ def blacken(session: nox.sessions.Session) -> None: session.run("black", *python_files) + # # Sample Tests # @@ -173,7 +177,9 @@ def blacken(session: nox.sessions.Session) -> None: PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] -def _session_tests(session: nox.sessions.Session, post_install: Callable = None) -> None: +def _session_tests( + session: nox.sessions.Session, post_install: Callable = None +) -> None: if TEST_CONFIG["pip_version_override"]: pip_version = TEST_CONFIG["pip_version_override"] session.install(f"pip=={pip_version}") @@ -203,7 +209,7 @@ def _session_tests(session: nox.sessions.Session, post_install: Callable = None) # on travis where slow and flaky tests are excluded. # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html success_codes=[0, 5], - env=get_pytest_env_vars() + env=get_pytest_env_vars(), ) @@ -213,9 +219,9 @@ def py(session: nox.sessions.Session) -> None: if session.python in TESTED_VERSIONS: _session_tests(session) else: - session.skip("SKIPPED: {} tests are disabled for this sample.".format( - session.python - )) + session.skip( + "SKIPPED: {} tests are disabled for this sample.".format(session.python) + ) # From 7b5cfcc5f78967b01cad2459a6c008966a5106d6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 20 Sep 2021 21:04:09 +0200 Subject: [PATCH 084/158] chore(deps): update all dependencies (#265) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | `==2.0.2` -> `==2.1.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-auth/2.1.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-auth/2.1.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-auth/2.1.0/compatibility-slim/2.0.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-auth/2.1.0/confidence-slim/2.0.2)](https://docs.renovatebot.com/merge-confidence/) | | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.42.1` -> `==1.42.2` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/compatibility-slim/1.42.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/confidence-slim/1.42.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-auth-library-python ### [`v2.1.0`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​210-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev202v210-2021-09-10) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v2.0.2...v2.1.0) ##### Features - Improve handling of clock skew ([#​858](https://www.togithub.com/googleapis/google-auth-library-python/issues/858)) ([45c4491](https://www.github.com/googleapis/google-auth-library-python/commit/45c4491fb971c9edf590b27b9e271b7a23a1bba6)) ##### Bug Fixes - add SAML challenge to reauth ([#​819](https://www.togithub.com/googleapis/google-auth-library-python/issues/819)) ([13aed5f](https://www.github.com/googleapis/google-auth-library-python/commit/13aed5ffe3ba435004ab48202462452f04d7cb29)) - disable warning if quota project id provided to auth.default() ([#​856](https://www.togithub.com/googleapis/google-auth-library-python/issues/856)) ([11ebaeb](https://www.github.com/googleapis/google-auth-library-python/commit/11ebaeb9d7c0862916154cfb810238574507629a)) - rename CLOCK_SKEW and separate client/server user case ([#​863](https://www.togithub.com/googleapis/google-auth-library-python/issues/863)) ([738611b](https://www.github.com/googleapis/google-auth-library-python/commit/738611bd2914f0fd5fa8b49b65f56ef321829c85)) ##### [2.0.2](https://www.github.com/googleapis/google-auth-library-python/compare/v2.0.1...v2.0.2) (2021-08-25) ##### Bug Fixes - use 'int.to_bytes' rather than deprecated crypto wrapper ([#​848](https://www.togithub.com/googleapis/google-auth-library-python/issues/848)) ([b79b554](https://www.github.com/googleapis/google-auth-library-python/commit/b79b55407b31933c9a8fe6de01478fa00a33fa2b)) - use int.from_bytes ([#​846](https://www.togithub.com/googleapis/google-auth-library-python/issues/846)) ([466aed9](https://www.github.com/googleapis/google-auth-library-python/commit/466aed99f5c2ba15d2036fa21cc83b3f0fc22639)) ##### [2.0.1](https://www.github.com/googleapis/google-auth-library-python/compare/v2.0.0...v2.0.1) (2021-08-17) ##### Bug Fixes - normalize AWS paths correctly on windows ([#​842](https://www.togithub.com/googleapis/google-auth-library-python/issues/842)) ([4e0fb1c](https://www.github.com/googleapis/google-auth-library-python/commit/4e0fb1cee78ee56b878b6e12be3b3c58df242b05))
googleapis/python-storage ### [`v1.42.2`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1422-httpswwwgithubcomgoogleapispython-storagecomparev1421v1422-2021-09-16) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.42.1...v1.42.2)
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a00c04bab2f3..354210794d6a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==1.11.1 grpcio==1.40.0 -google-auth==2.0.2 +google-auth==2.1.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.42.1 +google-cloud-storage==1.42.2 google-cloud-dataproc==2.5.0 From 19e0bbed78e84dc713819e46c70ef0ebca4013c7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 27 Sep 2021 19:50:16 +0200 Subject: [PATCH 085/158] chore(deps): update dependency google-auth to v2.2.0 (#274) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 354210794d6a..4bc4bf30886e 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.40.0 -google-auth==2.1.0 +google-auth==2.2.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.2 From 4c8765feee496f9f6275d4b20e527f95a77043b2 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 28 Sep 2021 16:43:03 +0200 Subject: [PATCH 086/158] chore(deps): update dependency grpcio to v1.41.0 (#275) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 4bc4bf30886e..6d56f58ba52b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.11.1 -grpcio==1.40.0 +grpcio==1.41.0 google-auth==2.2.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 6b064519a1f85ebc9c7b01b75df5379045034ac3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 29 Sep 2021 16:54:05 +0200 Subject: [PATCH 087/158] chore(deps): update dependency google-auth to v2.2.1 (#276) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 6d56f58ba52b..51b3360f732d 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.41.0 -google-auth==2.2.0 +google-auth==2.2.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.2 From 52affd7307cd141163769fb401ce682ff5f58180 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Thu, 30 Sep 2021 15:52:11 +0000 Subject: [PATCH 088/158] chore: fail samples nox session if python version is missing (#279) --- dataproc/snippets/noxfile.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index b008613f03ff..1fd8956fbf01 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -98,6 +98,10 @@ def get_pytest_env_vars() -> Dict[str, str]: "True", "true", ) + +# Error if a python version is missing +nox.options.error_on_missing_interpreters = True + # # Style Checks # From 46fb7eceada62f8c20b2839cb0ed6836409dbdde Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 1 Oct 2021 12:18:47 +0200 Subject: [PATCH 089/158] chore(deps): update dependency google-cloud-storage to v1.42.3 (#280) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 51b3360f732d..34decddd6293 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.41.0 google-auth==2.2.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.42.2 +google-cloud-storage==1.42.3 google-cloud-dataproc==2.5.0 From 1c550e711e16e35be66213313abff1e2dee0995f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 5 Oct 2021 18:18:03 +0200 Subject: [PATCH 090/158] chore(deps): update dependency google-cloud-dataproc to v3 (#282) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 34decddd6293..e6433fb18ae8 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.2.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 -google-cloud-dataproc==2.5.0 +google-cloud-dataproc==3.0.0 From 3de36a677a69c73ef3f6c11ff67e240a1eed2ff0 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 8 Oct 2021 17:26:18 +0000 Subject: [PATCH 091/158] chore(python): Add kokoro configs for python 3.10 samples testing (#287) --- dataproc/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 1fd8956fbf01..93a9122cc457 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -87,7 +87,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9"] +ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9", "3.10"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From ba18b32fcb1891609969cdda3e1008e312594e69 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 10 Oct 2021 00:04:55 +0200 Subject: [PATCH 092/158] chore(deps): update dependency google-auth to v2.3.0 (#286) Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e6433fb18ae8..133ac5f1745f 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.41.0 -google-auth==2.2.1 +google-auth==2.3.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 From 1b17a1aa3ac7580c5f6400c67206642bfd5e5cd0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 26 Oct 2021 18:15:19 +0200 Subject: [PATCH 093/158] chore(deps): update all dependencies (#292) Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 133ac5f1745f..515f8a1e47cd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 -grpcio==1.41.0 -google-auth==2.3.0 +grpcio==1.41.1 +google-auth==2.3.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 From ea2cfe117bde83158342628f42d5335a8d8630a5 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 26 Oct 2021 22:24:13 +0200 Subject: [PATCH 094/158] chore(deps): update all dependencies (#295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | `==2.3.1` -> `==2.3.2` | [![age](https://badges.renovateapi.com/packages/pypi/google-auth/2.3.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-auth/2.3.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-auth/2.3.2/compatibility-slim/2.3.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-auth/2.3.2/confidence-slim/2.3.1)](https://docs.renovatebot.com/merge-confidence/) | | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | `==3.0.0` -> `==3.1.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/3.1.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/3.1.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/3.1.0/compatibility-slim/3.0.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-dataproc/3.1.0/confidence-slim/3.0.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-auth-library-python ### [`v2.3.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#​232-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev231v232-2021-10-26) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v2.3.1...v2.3.2)
googleapis/python-dataproc ### [`v3.1.0`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#​310-httpswwwgithubcomgoogleapispython-dataproccomparev300v310-2021-10-26) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v3.0.0...v3.1.0) ##### Features - add context manager support in client ([#​285](https://www.togithub.com/googleapis/python-dataproc/issues/285)) ([b54fb76](https://www.github.com/googleapis/python-dataproc/commit/b54fb7647deaea64fe6ad553514c9d0ad62a0cbc)) - add Dataproc Serverless for Spark Batches API ([#​290](https://www.togithub.com/googleapis/python-dataproc/issues/290)) ([f0ed26c](https://www.github.com/googleapis/python-dataproc/commit/f0ed26c6ccd2e9f438d1d5f31c5512761b0e20b9)) - Add support for dataproc BatchController service ([#​291](https://www.togithub.com/googleapis/python-dataproc/issues/291)) ([24a6f7d](https://www.github.com/googleapis/python-dataproc/commit/24a6f7defee1e0fd2d195f934c004769d8f1a2b7)) - add support for python 3.10 ([#​289](https://www.togithub.com/googleapis/python-dataproc/issues/289)) ([229f919](https://www.github.com/googleapis/python-dataproc/commit/229f919e31c39bc028cd2e6062437b0a8d061556))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-dataproc). --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 515f8a1e47cd..de7d9f10802b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==1.11.1 grpcio==1.41.1 -google-auth==2.3.1 +google-auth==2.3.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 -google-cloud-dataproc==3.0.0 +google-cloud-dataproc==3.1.0 From 435ef7913a3ab8b271b58b815a97a66d1a9c92fc Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 1 Nov 2021 22:30:12 +0100 Subject: [PATCH 095/158] chore(deps): update dependency google-auth to v2.3.3 (#298) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index de7d9f10802b..d22c8b7a225b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.41.1 -google-auth==2.3.2 +google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 From 101a4403b56774c8b94e28c71e17fa2142ec5e40 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 3 Nov 2021 12:30:18 +0100 Subject: [PATCH 096/158] chore(deps): update dependency google-cloud-dataproc to v3.1.1 (#299) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d22c8b7a225b..a361f04ebfa5 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==1.42.3 -google-cloud-dataproc==3.1.0 +google-cloud-dataproc==3.1.1 From 007a381d78ab578a1b77687d66e6b52a7bb8542e Mon Sep 17 00:00:00 2001 From: aman-ebay Date: Fri, 12 Nov 2021 09:34:34 -0800 Subject: [PATCH 097/158] chore: correct region tag in submit_job_to_cluster.py (#304) Change region tag to make it unique. The previous tag was used in another create cluster file and caused problems with automation tools. --- dataproc/snippets/submit_job_to_cluster.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index 68a547c481fd..93a9f66c7fe8 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -85,7 +85,7 @@ def download_output(project, cluster_id, output_bucket, job_id): return bucket.blob(output_blob).download_as_string() -# [START dataproc_create_cluster] +# [START dataproc_submit_job_create_cluster] def create_cluster(dataproc, project, zone, region, cluster_name): """Create the cluster.""" print("Creating cluster...") @@ -110,7 +110,7 @@ def create_cluster(dataproc, project, zone, region, cluster_name): waiting_callback = True -# [END dataproc_create_cluster] +# [END dataproc_submit_job_create_cluster] def callback(operation_future): From 50c18c59055acc027f7caa80e375493c1c8beaa0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 19 Nov 2021 12:13:26 +0100 Subject: [PATCH 098/158] chore(deps): update all dependencies (#306) --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a361f04ebfa5..f998ceae5891 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==1.11.1 -grpcio==1.41.1 +grpcio==1.42.0 google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.42.3 +google-cloud-storage==1.43.0 google-cloud-dataproc==3.1.1 From ab7df1fbf203bb4aa17972d2b2b575d362740b83 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 9 Jan 2022 23:22:11 +0100 Subject: [PATCH 099/158] chore(deps): update all dependencies (#313) Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index f998ceae5891..2f1e08160697 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==1.11.1 -grpcio==1.42.0 +grpcio==1.43.0 google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.43.0 +google-cloud-storage==1.44.0 google-cloud-dataproc==3.1.1 From d00d307661ab45f56a55345b9d626012f7a077e5 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 11 Jan 2022 10:18:25 -0500 Subject: [PATCH 100/158] chore(samples): Add check for tests in directory (#323) Source-Link: https://github.com/googleapis/synthtool/commit/52aef91f8d25223d9dbdb4aebd94ba8eea2101f3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:36a95b8f494e4674dc9eee9af98961293b51b86b3649942aac800ae6c1f796d4 Co-authored-by: Owl Bot --- dataproc/snippets/noxfile.py | 70 ++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 93a9122cc457..3bbef5d54f44 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -14,6 +14,7 @@ from __future__ import print_function +import glob import os from pathlib import Path import sys @@ -184,37 +185,44 @@ def blacken(session: nox.sessions.Session) -> None: def _session_tests( session: nox.sessions.Session, post_install: Callable = None ) -> None: - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install("-r", "requirements-test.txt", "-c", "constraints-test.txt") - else: - session.install("-r", "requirements-test.txt") - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) + # check for presence of tests + test_list = glob.glob("*_test.py") + glob.glob("test_*.py") + if len(test_list) == 0: + print("No tests found, skipping directory.") + else: + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install( + "-r", "requirements-test.txt", "-c", "constraints-test.txt" + ) + else: + session.install("-r", "requirements-test.txt") + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) @nox.session(python=ALL_VERSIONS) From 350f252e96257be5025b48930a82fccff36cd53c Mon Sep 17 00:00:00 2001 From: aman-ebay Date: Thu, 13 Jan 2022 10:17:28 -0800 Subject: [PATCH 101/158] docs(samples): update python-api-walkthrough.md (#308) Enable APIs using gcloud in the Cloud Shell terminal. Update product and links names. Co-authored-by: meredithslota Co-authored-by: Anthonios Partheniou --- dataproc/snippets/python-api-walkthrough.md | 29 ++++++++++++--------- 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/dataproc/snippets/python-api-walkthrough.md b/dataproc/snippets/python-api-walkthrough.md index 1a8d436f7202..29161bbacef5 100644 --- a/dataproc/snippets/python-api-walkthrough.md +++ b/dataproc/snippets/python-api-walkthrough.md @@ -6,7 +6,7 @@ Estimated completion time: - -1. Click the link below to enable the Dataproc, Compute Engine, and Cloud Storage APIs - in a separate GCP console tab in your browser. - - **Note:** After you select your project and enable the APIs, return to this tutorial by clicking - on the **Cloud Shell** tab in your browser. - - * [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=dataproc,compute_component,storage-component.googleapis.com&redirect=https://console.cloud.google.com) + + +1. Create or select a Google Cloud project to use for this +tutorial. +* + +1. Enable the Dataproc, Compute Engine, and Cloud Storage APIs in your +project. +```sh +gcloud services enable dataproc.googleapis.com \ +compute.googleapis.com \ +storage-component.googleapis.com \ +--project={{project_id}} +``` ## Prerequisites (2) From e56230f1f457d6b109631d857cbf1cb1d044ece2 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 14 Jan 2022 20:00:39 +0100 Subject: [PATCH 102/158] chore(deps): update dependency google-cloud-storage to v2 (#326) Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 2f1e08160697..7cecd031cb23 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.43.0 google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==1.44.0 +google-cloud-storage==2.0.0 google-cloud-dataproc==3.1.1 From cd6dc62bd95c345861dc038ea934942f868df394 Mon Sep 17 00:00:00 2001 From: Igor Dvorzhak Date: Mon, 17 Jan 2022 08:29:06 -0800 Subject: [PATCH 103/158] chore(samples): fix job polling (#314) * Fix job polling * Update submit_job_to_cluster.py Co-authored-by: Anthonios Partheniou --- dataproc/snippets/submit_job_to_cluster.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index 93a9f66c7fe8..b10246756166 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -202,9 +202,9 @@ def wait_for_job(dataproc, project, region, job_id): request={"project_id": project, "region": region, "job_id": job_id} ) # Handle exceptions - if job.status.State.Name(job.status.state) == "ERROR": + if job.status.State(job.status.state).name == "ERROR": raise Exception(job.status.details) - elif job.status.State.Name(job.status.state) == "DONE": + if job.status.State(job.status.state).name == "DONE": print("Job finished.") return job From afafbd42170c3938aa5e26dcb5d35df1e382658a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 18 Jan 2022 20:43:41 +0100 Subject: [PATCH 104/158] chore(deps): update dependency google-cloud-dataproc to v3.2.0 (#330) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 7cecd031cb23..a02bb2f3f802 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.0.0 -google-cloud-dataproc==3.1.1 +google-cloud-dataproc==3.2.0 From 4b3b4b7b74c6dfa83018509a5282ee9b934c72af Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 19 Jan 2022 04:37:24 -0500 Subject: [PATCH 105/158] chore(python): Noxfile recognizes that tests can live in a folder (#331) Source-Link: https://github.com/googleapis/synthtool/commit/4760d8dce1351d93658cb11d02a1b7ceb23ae5d7 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f0e4b51deef56bed74d3e2359c583fc104a8d6367da3984fc5c66938db738828 Co-authored-by: Owl Bot --- dataproc/snippets/noxfile.py | 1 + 1 file changed, 1 insertion(+) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 3bbef5d54f44..20cdfc620138 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -187,6 +187,7 @@ def _session_tests( ) -> None: # check for presence of tests test_list = glob.glob("*_test.py") + glob.glob("test_*.py") + test_list.extend(glob.glob("tests")) if len(test_list) == 0: print("No tests found, skipping directory.") else: From 66289ca983ff72fc38afba26e02976d38fc02928 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 22 Jan 2022 11:05:25 +0100 Subject: [PATCH 106/158] chore(deps): update dependency google-cloud-storage to v2.1.0 (#332) * chore(deps): update all dependencies * revert update for google-auth * add pin for google-cloud-storage for py < 3.7 Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a02bb2f3f802..7e4daca78d77 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,6 @@ grpcio==1.43.0 google-auth==2.3.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.0.0 +google-cloud-storage==2.0.0; python_version == '3.6' +google-cloud-storage==2.1.0; python_version >= '3.7' google-cloud-dataproc==3.2.0 From 266b7b5649e4e25c6d6c49c5c4ed5c488e74259a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 7 Feb 2022 17:20:52 +0100 Subject: [PATCH 107/158] chore(deps): update all dependencies (#338) --- dataproc/snippets/requirements-test.txt | 2 +- dataproc/snippets/requirements.txt | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 927094516e65..4a46ff600804 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1 +1 @@ -pytest==6.2.5 +pytest==7.0.0 diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 7e4daca78d77..40167cc30440 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,9 +1,9 @@ backoff==1.11.1 grpcio==1.43.0 -google-auth==2.3.3 +google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.0.0; python_version == '3.6' +google-cloud-storage==2.1.0; python_version == '3.6' google-cloud-storage==2.1.0; python_version >= '3.7' google-cloud-dataproc==3.2.0 From 4a3b5637dff40aabc7b2f02be538fcaf5ee75452 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 14 Feb 2022 16:53:20 +0100 Subject: [PATCH 108/158] chore(deps): update dependency pytest to v7.0.1 (#343) --- dataproc/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 4a46ff600804..c2845bffbe89 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1 +1 @@ -pytest==7.0.0 +pytest==7.0.1 From fe29c46f1eb275aa36fef3b101a8867a0bbcf177 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 18 Feb 2022 12:19:26 +0100 Subject: [PATCH 109/158] chore(deps): update dependency grpcio to v1.44.0 (#345) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 40167cc30440..e7ba6a204a96 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==1.11.1 -grpcio==1.43.0 +grpcio==1.44.0 google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 09a61566ede4766fc3984955dccc443bc7d96617 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 26 Feb 2022 01:15:43 +0100 Subject: [PATCH 110/158] chore(deps): update dependency google-cloud-dataproc to v3.3.0 (#349) * chore(deps): update dependency google-cloud-dataproc to v3.3.0 * remove pin for python 3.6 Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e7ba6a204a96..3f50eb9815d1 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,6 +4,5 @@ grpcio==1.44.0 google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.1.0; python_version == '3.6' -google-cloud-storage==2.1.0; python_version >= '3.7' -google-cloud-dataproc==3.2.0 +google-cloud-storage==2.1.0 +google-cloud-dataproc==3.3.0 From d327dea3278d351f8a486a9d745b23c36bcce7a3 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 4 Mar 2022 16:56:43 -0500 Subject: [PATCH 111/158] chore: Adding support for pytest-xdist and pytest-parallel (#358) Source-Link: https://github.com/googleapis/synthtool/commit/82f5cb283efffe96e1b6cd634738e0e7de2cd90a Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:5d8da01438ece4021d135433f2cf3227aa39ef0eaccc941d62aa35e6902832ae Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/noxfile.py | 80 +++++++++++++++++++++--------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 20cdfc620138..4c808af73ea2 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -188,42 +188,54 @@ def _session_tests( # check for presence of tests test_list = glob.glob("*_test.py") + glob.glob("test_*.py") test_list.extend(glob.glob("tests")) + if len(test_list) == 0: print("No tests found, skipping directory.") - else: - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install( - "-r", "requirements-test.txt", "-c", "constraints-test.txt" - ) - else: - session.install("-r", "requirements-test.txt") - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) + return + + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + concurrent_args = [] + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + with open("requirements.txt") as rfile: + packages = rfile.read() + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install( + "-r", "requirements-test.txt", "-c", "constraints-test.txt" + ) + else: + session.install("-r", "requirements-test.txt") + with open("requirements-test.txt") as rtfile: + packages += rtfile.read() + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + if "pytest-parallel" in packages: + concurrent_args.extend(['--workers', 'auto', '--tests-per-worker', 'auto']) + elif "pytest-xdist" in packages: + concurrent_args.extend(['-n', 'auto']) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs + concurrent_args), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) @nox.session(python=ALL_VERSIONS) From f015223cff1ebc9e061d195bca53f95f985522b7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 5 Mar 2022 18:03:56 +0100 Subject: [PATCH 112/158] chore(deps): update all dependencies (#354) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 3f50eb9815d1..ade4edf30995 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.1.0 -google-cloud-dataproc==3.3.0 +google-cloud-dataproc==4.0.0 From c3ac64a7dbcda9029d5e20fb7e8c885239807a4d Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 7 Mar 2022 23:07:10 +0100 Subject: [PATCH 113/158] chore(deps): update dependency google-cloud-dataproc to v4.0.1 (#362) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index ade4edf30995..873e05a1a4dd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.1.0 -google-cloud-dataproc==4.0.0 +google-cloud-dataproc==4.0.1 From 816fbb44ee4649de4de3f9806c4fa935e1d58670 Mon Sep 17 00:00:00 2001 From: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Date: Tue, 8 Mar 2022 09:09:06 -0700 Subject: [PATCH 114/158] test(samples): use try/finally for clusters and use pytest-xdist (#360) --- dataproc/snippets/create_cluster_test.py | 22 ++++++---- .../snippets/quickstart/quickstart_test.py | 21 +++++---- dataproc/snippets/requirements-test.txt | 1 + dataproc/snippets/submit_job_test.py | 42 +++++++++++------- dataproc/snippets/update_cluster_test.py | 44 +++++++++++-------- 5 files changed, 77 insertions(+), 53 deletions(-) diff --git a/dataproc/snippets/create_cluster_test.py b/dataproc/snippets/create_cluster_test.py index fd391c3e0abd..48941a1c5dbb 100644 --- a/dataproc/snippets/create_cluster_test.py +++ b/dataproc/snippets/create_cluster_test.py @@ -15,6 +15,7 @@ import os import uuid +from google.api_core.exceptions import NotFound from google.cloud import dataproc_v1 as dataproc import pytest @@ -34,15 +35,18 @@ def teardown(): client_options={"api_endpoint": f"{REGION}-dataproc.googleapis.com:443"} ) # Client library function - operation = cluster_client.delete_cluster( - request={ - "project_id": PROJECT_ID, - "region": REGION, - "cluster_name": CLUSTER_NAME, - } - ) - # Wait for cluster to delete - operation.result() + try: + operation = cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) + # Wait for cluster to delete + operation.result() + except NotFound: + print("Cluster already deleted") def test_cluster_create(capsys): diff --git a/dataproc/snippets/quickstart/quickstart_test.py b/dataproc/snippets/quickstart/quickstart_test.py index 19385353ccf5..4020ad80e981 100644 --- a/dataproc/snippets/quickstart/quickstart_test.py +++ b/dataproc/snippets/quickstart/quickstart_test.py @@ -39,12 +39,10 @@ @pytest.fixture(autouse=True) -def setup_teardown(): +def blob(): storage_client = storage.Client() - @backoff.on_exception(backoff.expo, - ServiceUnavailable, - max_tries=5) + @backoff.on_exception(backoff.expo, ServiceUnavailable, max_tries=5) def create_bucket(): return storage_client.create_bucket(STAGING_BUCKET) @@ -54,12 +52,20 @@ def create_bucket(): yield + blob.delete() + bucket.delete() + + +@pytest.fixture(autouse=True) +def cluster(): + yield + + # The quickstart sample deletes the cluster, but if the test fails + # before cluster deletion occurs, it can be manually deleted here. cluster_client = dataproc.ClusterControllerClient( client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION)} ) - # The quickstart sample deletes the cluster, but if the test fails - # before cluster deletion occurs, it can be manually deleted here. clusters = cluster_client.list_clusters( request={"project_id": PROJECT_ID, "region": REGION} ) @@ -74,9 +80,6 @@ def create_bucket(): } ) - blob.delete() - bucket.delete() - def test_quickstart(capsys): quickstart.quickstart(PROJECT_ID, REGION, CLUSTER_NAME, JOB_FILE_PATH) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index c2845bffbe89..54f4c3f99f3c 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1 +1,2 @@ pytest==7.0.1 +pytest-xdist==2.5.0 \ No newline at end of file diff --git a/dataproc/snippets/submit_job_test.py b/dataproc/snippets/submit_job_test.py index 6827916fbff9..c2e9bed9e7f8 100644 --- a/dataproc/snippets/submit_job_test.py +++ b/dataproc/snippets/submit_job_test.py @@ -15,6 +15,7 @@ import os import uuid +from google.api_core.exceptions import NotFound from google.cloud import dataproc_v1 as dataproc import pytest @@ -36,25 +37,34 @@ @pytest.fixture(autouse=True) def setup_teardown(): - cluster_client = dataproc.ClusterControllerClient( - client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION)} - ) + try: + cluster_client = dataproc.ClusterControllerClient( + client_options={ + "api_endpoint": "{}-dataproc.googleapis.com:443".format(REGION) + } + ) - # Create the cluster. - operation = cluster_client.create_cluster( - request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} - ) - operation.result() + # Create the cluster. + operation = cluster_client.create_cluster( + request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} + ) + operation.result() - yield + yield - cluster_client.delete_cluster( - request={ - "project_id": PROJECT_ID, - "region": REGION, - "cluster_name": CLUSTER_NAME, - } - ) + finally: + try: + operation = cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) + operation.result() + + except NotFound: + print("Cluster already deleted") def test_submit_job(capsys): diff --git a/dataproc/snippets/update_cluster_test.py b/dataproc/snippets/update_cluster_test.py index 9c608ac43525..9a9d66c97978 100644 --- a/dataproc/snippets/update_cluster_test.py +++ b/dataproc/snippets/update_cluster_test.py @@ -19,6 +19,7 @@ import os import uuid +from google.api_core.exceptions import NotFound from google.cloud.dataproc_v1.services.cluster_controller.client import ( ClusterControllerClient, ) @@ -41,25 +42,6 @@ } -@pytest.fixture(autouse=True) -def setup_teardown(cluster_client): - # Create the cluster. - operation = cluster_client.create_cluster( - request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} - ) - operation.result() - - yield - - cluster_client.delete_cluster( - request={ - "project_id": PROJECT_ID, - "region": REGION, - "cluster_name": CLUSTER_NAME, - } - ) - - @pytest.fixture def cluster_client(): cluster_client = ClusterControllerClient( @@ -68,6 +50,30 @@ def cluster_client(): return cluster_client +@pytest.fixture(autouse=True) +def setup_teardown(cluster_client): + try: + # Create the cluster. + operation = cluster_client.create_cluster( + request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER} + ) + operation.result() + + yield + finally: + try: + operation = cluster_client.delete_cluster( + request={ + "project_id": PROJECT_ID, + "region": REGION, + "cluster_name": CLUSTER_NAME, + } + ) + operation.result() + except NotFound: + print("Cluster already deleted") + + def test_update_cluster(capsys, cluster_client: ClusterControllerClient): # Wrapper function for client library function update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES) From 56d91ce422e3b80ce650d3d76b60105cdaa20459 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 13 Mar 2022 23:40:30 +0100 Subject: [PATCH 115/158] chore(deps): update dependency pytest to v7.1.0 (#366) --- dataproc/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 54f4c3f99f3c..302c0f8c6b6d 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1,2 +1,2 @@ -pytest==7.0.1 +pytest==7.1.0 pytest-xdist==2.5.0 \ No newline at end of file From 4ff6896f23212a82985a0ee16b0e451f4e312612 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 14 Mar 2022 18:55:05 +0100 Subject: [PATCH 116/158] chore(deps): update dependency google-cloud-storage to v2.2.0 (#367) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 873e05a1a4dd..04bde0093748 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.44.0 google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.1.0 +google-cloud-storage==2.2.0 google-cloud-dataproc==4.0.1 From 2b8eefa1e9fdf8c04f18d1142c3a6f34019e4b1a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 15 Mar 2022 23:43:25 +0100 Subject: [PATCH 117/158] chore(deps): update dependency google-cloud-storage to v2.2.1 (#368) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 04bde0093748..e0591ea97f1d 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.44.0 google-auth==2.6.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.2.0 +google-cloud-storage==2.2.1 google-cloud-dataproc==4.0.1 From 6cde59ec847362aefc167c0ce6036aa52b470c10 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 19 Mar 2022 11:16:03 +0100 Subject: [PATCH 118/158] chore(deps): update all dependencies (#369) --- dataproc/snippets/requirements-test.txt | 2 +- dataproc/snippets/requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 302c0f8c6b6d..cfc646faf186 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1,2 +1,2 @@ -pytest==7.1.0 +pytest==7.1.1 pytest-xdist==2.5.0 \ No newline at end of file diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e0591ea97f1d..733072cca58d 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.44.0 -google-auth==2.6.0 +google-auth==2.6.2 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.2.1 From 9c2bd8baf1b990acbf654fa4c474b312c2e5aecc Mon Sep 17 00:00:00 2001 From: Lo Ferris <50979514+loferris@users.noreply.github.com> Date: Fri, 25 Mar 2022 13:26:28 -0700 Subject: [PATCH 119/158] Fix: resource quotas (#377) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: adding a noxfile_config.py to snippets to balance load between different projects * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * disabling type hint enforcement in noxfile_config * changing the method of disabling type checking * linter on samples * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * changing transport logic for submit_job_to_cluster.py * updating transport method in list_clusters.py * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot --- dataproc/snippets/list_clusters.py | 17 ++------- dataproc/snippets/noxfile_config.py | 42 ++++++++++++++++++++++ dataproc/snippets/submit_job_to_cluster.py | 29 ++++----------- dataproc/snippets/update_cluster.py | 2 ++ 4 files changed, 54 insertions(+), 36 deletions(-) create mode 100644 dataproc/snippets/noxfile_config.py diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py index 3ecb8aebc6db..11acdf5da989 100644 --- a/dataproc/snippets/list_clusters.py +++ b/dataproc/snippets/list_clusters.py @@ -19,7 +19,6 @@ import argparse from google.cloud import dataproc_v1 -from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport # [START dataproc_list_clusters] @@ -28,14 +27,7 @@ def list_clusters(dataproc, project, region): for cluster in dataproc.list_clusters( request={"project_id": project, "region": region} ): - print( - ( - "{} - {}".format( - cluster.cluster_name, - cluster.status.state.name - ) - ) - ) + print(("{} - {}".format(cluster.cluster_name, cluster.status.state.name))) # [END dataproc_list_clusters] @@ -49,12 +41,9 @@ def main(project_id, region): else: # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = ( - cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address="{}-dataproc.googleapis.com:443".format(region) - ) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient( + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} ) - dataproc_cluster_client = dataproc_v1.ClusterControllerClient(client_transport) list_clusters(dataproc_cluster_client, project_id, region) diff --git a/dataproc/snippets/noxfile_config.py b/dataproc/snippets/noxfile_config.py new file mode 100644 index 000000000000..646d77de8543 --- /dev/null +++ b/dataproc/snippets/noxfile_config.py @@ -0,0 +1,42 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Default TEST_CONFIG_OVERRIDE for python repos. + +# You can copy this file into your directory, then it will be imported from +# the noxfile.py. + +# The source of truth: +# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/noxfile_config.py + +TEST_CONFIG_OVERRIDE = { + # You can opt out from the test for specific Python versions. + "ignored_versions": ["2.7", "3.6"], + # Old samples are opted out of enforcing Python type hints + # All new samples should feature them + # "enforce_type_hints": True, + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + # "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", + "gcloud_project_env": "BUILD_SPECIFIC_GCLOUD_PROJECT", + # If you need to use a specific version of pip, + # change pip_version_override to the string representation + # of the version number, for example, "20.2.4" + "pip_version_override": None, + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + "envs": {}, +} diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index b10246756166..3b284368a4aa 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -34,8 +34,6 @@ from google.cloud import dataproc_v1 from google.cloud import storage -from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport -from google.cloud.dataproc_v1.gapic.transports import job_controller_grpc_transport DEFAULT_FILENAME = "pyspark_sort.py" @@ -77,10 +75,8 @@ def download_output(project, cluster_id, output_bucket, job_id): print("Downloading output file.") client = storage.Client(project=project) bucket = client.get_bucket(output_bucket) - output_blob = ( - "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( - cluster_id, job_id - ) + output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( + cluster_id, job_id ) return bucket.blob(output_blob).download_as_string() @@ -135,14 +131,7 @@ def list_clusters_with_details(dataproc, project, region): for cluster in dataproc.list_clusters( request={"project_id": project, "region": region} ): - print( - ( - "{} - {}".format( - cluster.cluster_name, - cluster.status.state.name, - ) - ) - ) + print(("{} - {}".format(cluster.cluster_name, cluster.status.state.name,))) # [END dataproc_list_clusters_with_detail] @@ -232,16 +221,12 @@ def main( region = get_region_from_zone(zone) # Use a regional gRPC endpoint. See: # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - client_transport = ( - cluster_controller_grpc_transport.ClusterControllerGrpcTransport( - address="{}-dataproc.googleapis.com:443".format(region) - ) + dataproc_cluster_client = dataproc_v1.ClusterControllerClient( + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} ) - job_transport = job_controller_grpc_transport.JobControllerGrpcTransport( - address="{}-dataproc.googleapis.com:443".format(region) + dataproc_job_client = dataproc_v1.ClusterControllerClient( + client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} ) - dataproc_cluster_client = dataproc_v1.ClusterControllerClient(client_transport) - dataproc_job_client = dataproc_v1.JobControllerClient(job_transport) # [END dataproc_get_client] try: diff --git a/dataproc/snippets/update_cluster.py b/dataproc/snippets/update_cluster.py index f452022426cf..bae6eee2fd64 100644 --- a/dataproc/snippets/update_cluster.py +++ b/dataproc/snippets/update_cluster.py @@ -1,3 +1,5 @@ +#!/usr/bin/env python + # Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); From 08ad604e1d3829fa67f136e750bda678ebf8a1c5 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 29 Mar 2022 00:06:17 +0000 Subject: [PATCH 120/158] chore(python): use black==22.3.0 (#383) Source-Link: https://github.com/googleapis/synthtool/commit/6fab84af09f2cf89a031fd8671d1def6b2931b11 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:7cffbc10910c3ab1b852c05114a08d374c195a81cdec1d4a67a1d129331d0bfe --- dataproc/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 4c808af73ea2..949e0fde9ae1 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -29,7 +29,7 @@ # WARNING - WARNING - WARNING - WARNING - WARNING # WARNING - WARNING - WARNING - WARNING - WARNING -BLACK_VERSION = "black==19.10b0" +BLACK_VERSION = "black==22.3.0" # Copy `noxfile_config.py` to your directory and modify it instead. From c5f2ceea9d675e0e9b20d8cdcec333a35bfa46b7 Mon Sep 17 00:00:00 2001 From: aman-ebay Date: Thu, 31 Mar 2022 12:25:23 -0700 Subject: [PATCH 121/158] Fix: updating submit_job_to_cluster.py (#387) * Fix lint errors * Update submit_job_to_cluster.py --- dataproc/snippets/submit_job_to_cluster.py | 333 +++++---------------- 1 file changed, 83 insertions(+), 250 deletions(-) diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index 3b284368a4aa..b543f8e38992 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -1,307 +1,140 @@ #!/usr/bin/env python + +# Copyright 2022 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -r"""Sample command-line program to run a pyspark job on a new or existing -cluster. - -Global region clusters are supported with --global_region flag. - -Example Usage to run the pyspark job on a new cluster: -python submit_job_to_cluster.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ - --create_new_cluster --cluster_name=$CLUSTER --zone=$ZONE - -Example Usage to run the pyspark job on an existing global region cluster: -python submit_job_to_cluster.py --project_id=$PROJECT --gcs_bucket=$BUCKET \ - --global_region --cluster_name=$CLUSTER --zone=$ZONE +# [START dataproc_quickstart] """ +Command-line program to create a Dataproc cluster, +run a PySpark job located in Cloud Storage on the cluster, +then delete the cluster after the job completes. -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function +Usage: + python submit_job_to_cluster --project_id --region \ + --cluster_name --job_file_path +""" import argparse -import os +import re from google.cloud import dataproc_v1 from google.cloud import storage -DEFAULT_FILENAME = "pyspark_sort.py" -waiting_callback = False - - -def get_pyspark_file(pyspark_file=None): - if pyspark_file: - f = open(pyspark_file, "rb") - return f, os.path.basename(pyspark_file) - else: - """Gets the PySpark file from current directory.""" - current_dir = os.path.dirname(os.path.abspath(__file__)) - f = open(os.path.join(current_dir, DEFAULT_FILENAME), "rb") - return f, DEFAULT_FILENAME - - -def get_region_from_zone(zone): - try: - region_as_list = zone.split("-")[:-1] - return "-".join(region_as_list) - except (AttributeError, IndexError, ValueError): - raise ValueError("Invalid zone provided, please check your input.") - - -def upload_pyspark_file(project, bucket_name, filename, spark_file): - """Uploads the PySpark file in this directory to the configured input - bucket.""" - print("Uploading pyspark file to Cloud Storage.") - client = storage.Client(project=project) - bucket = client.get_bucket(bucket_name) - blob = bucket.blob(filename) - blob.upload_from_file(spark_file) - - -def download_output(project, cluster_id, output_bucket, job_id): - """Downloads the output file from Cloud Storage and returns it as a - string.""" - print("Downloading output file.") - client = storage.Client(project=project) - bucket = client.get_bucket(output_bucket) - output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( - cluster_id, job_id +# [START dataproc_create_cluster] +def quickstart(project_id, region, cluster_name, job_file_path): + # Create the cluster client. + cluster_client = dataproc_v1.ClusterControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} ) - return bucket.blob(output_blob).download_as_string() - -# [START dataproc_submit_job_create_cluster] -def create_cluster(dataproc, project, zone, region, cluster_name): - """Create the cluster.""" - print("Creating cluster...") - zone_uri = "https://www.googleapis.com/compute/v1/projects/{}/zones/{}".format( - project, zone - ) - cluster_data = { - "project_id": project, + # Create the cluster config. + cluster = { + "project_id": project_id, "cluster_name": cluster_name, "config": { - "gce_cluster_config": {"zone_uri": zone_uri}, - "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-1"}, - "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-1"}, + "master_config": {"num_instances": 1, "machine_type_uri": "n1-standard-2"}, + "worker_config": {"num_instances": 2, "machine_type_uri": "n1-standard-2"}, }, } - cluster = dataproc.create_cluster( - request={"project_id": project, "region": region, "cluster": cluster_data} + # Create the cluster. + operation = cluster_client.create_cluster( + request={"project_id": project_id, "region": region, "cluster": cluster} ) - cluster.add_done_callback(callback) - global waiting_callback - waiting_callback = True - - -# [END dataproc_submit_job_create_cluster] - - -def callback(operation_future): - # Reset global when callback returns. - global waiting_callback - waiting_callback = False - - -def wait_for_cluster_creation(): - """Wait for cluster creation.""" - print("Waiting for cluster creation...") - - while True: - if not waiting_callback: - print("Cluster created.") - break - - -# [START dataproc_list_clusters_with_detail] -def list_clusters_with_details(dataproc, project, region): - """List the details of clusters in the region.""" - for cluster in dataproc.list_clusters( - request={"project_id": project, "region": region} - ): - print(("{} - {}".format(cluster.cluster_name, cluster.status.state.name,))) - + result = operation.result() -# [END dataproc_list_clusters_with_detail] + print("Cluster created successfully: {}".format(result.cluster_name)) +# [END dataproc_create_cluster] -def get_cluster_id_by_name(dataproc, project_id, region, cluster_name): - """Helper function to retrieve the ID and output bucket of a cluster by - name.""" - for cluster in dataproc.list_clusters( - request={"project_id": project_id, "region": region} - ): - if cluster.cluster_name == cluster_name: - return cluster.cluster_uuid, cluster.config.config_bucket - +# [START dataproc_submit_job] + # Create the job client. + job_client = dataproc_v1.JobControllerClient( + client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} + ) -# [START dataproc_submit_pyspark_job] -def submit_pyspark_job(dataproc, project, region, cluster_name, bucket_name, filename): - """Submit the Pyspark job to the cluster (assumes `filename` was uploaded - to `bucket_name.""" - job_details = { + # Create the job config. + job = { "placement": {"cluster_name": cluster_name}, - "pyspark_job": { - "main_python_file_uri": "gs://{}/{}".format(bucket_name, filename) - }, + "pyspark_job": {"main_python_file_uri": job_file_path}, } - result = dataproc.submit_job( - request={"project_id": project, "region": region, "job": job_details} + operation = job_client.submit_job_as_operation( + request={"project_id": project_id, "region": region, "job": job} ) - job_id = result.reference.job_id - print("Submitted job ID {}.".format(job_id)) - return job_id - + response = operation.result() -# [END dataproc_submit_pyspark_job] + # Dataproc job output is saved to the Cloud Storage bucket + # allocated to the job. Use regex to obtain the bucket and blob info. + matches = re.match("gs://(.*?)/(.*)", response.driver_output_resource_uri) - -# [START dataproc_delete] -def delete_cluster(dataproc, project, region, cluster): - """Delete the cluster.""" - print("Tearing down cluster.") - result = dataproc.delete_cluster( - request={"project_id": project, "region": region, "cluster_name": cluster} + output = ( + storage.Client() + .get_bucket(matches.group(1)) + .blob(f"{matches.group(2)}.000000000") + .download_as_string() ) - return result - - -# [END dataproc_delete] - - -# [START dataproc_wait] -def wait_for_job(dataproc, project, region, job_id): - """Wait for job to complete or error out.""" - print("Waiting for job to finish...") - while True: - job = dataproc.get_job( - request={"project_id": project, "region": region, "job_id": job_id} - ) - # Handle exceptions - if job.status.State(job.status.state).name == "ERROR": - raise Exception(job.status.details) - if job.status.State(job.status.state).name == "DONE": - print("Job finished.") - return job - - -# [END dataproc_wait] - - -def main( - project_id, - zone, - cluster_name, - bucket_name, - pyspark_file=None, - create_new_cluster=True, - global_region=True, -): - # [START dataproc_get_client] - if global_region: - region = "global" - # Use the default gRPC global endpoints. - dataproc_cluster_client = dataproc_v1.ClusterControllerClient() - dataproc_job_client = dataproc_v1.JobControllerClient() - else: - region = get_region_from_zone(zone) - # Use a regional gRPC endpoint. See: - # https://cloud.google.com/dataproc/docs/concepts/regional-endpoints - dataproc_cluster_client = dataproc_v1.ClusterControllerClient( - client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} - ) - dataproc_job_client = dataproc_v1.ClusterControllerClient( - client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"} - ) - # [END dataproc_get_client] - - try: - spark_file, spark_filename = get_pyspark_file(pyspark_file) - if create_new_cluster: - create_cluster( - dataproc_cluster_client, project_id, zone, region, cluster_name - ) - wait_for_cluster_creation() - upload_pyspark_file(project_id, bucket_name, spark_filename, spark_file) - - list_clusters_with_details(dataproc_cluster_client, project_id, region) - - (cluster_id, output_bucket) = get_cluster_id_by_name( - dataproc_cluster_client, project_id, region, cluster_name - ) - - # [START dataproc_call_submit_pyspark_job] - job_id = submit_pyspark_job( - dataproc_job_client, - project_id, - region, - cluster_name, - bucket_name, - spark_filename, - ) - # [END dataproc_call_submit_pyspark_job] + print(f"Job finished successfully: {output}\r\n") + # [END dataproc_submit_job] + + # [START dataproc_delete_cluster] + # Delete the cluster once the job has terminated. + operation = cluster_client.delete_cluster( + request={ + "project_id": project_id, + "region": region, + "cluster_name": cluster_name, + } + ) + operation.result() - wait_for_job(dataproc_job_client, project_id, region, job_id) - output = download_output(project_id, cluster_id, output_bucket, job_id) - print("Received job output {}".format(output)) - return output - finally: - if create_new_cluster: - delete_cluster(dataproc_cluster_client, project_id, region, cluster_name) - spark_file.close() + print("Cluster {} successfully deleted.".format(cluster_name)) +# [END dataproc_delete_cluster] if __name__ == "__main__": parser = argparse.ArgumentParser( - description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter - ) - parser.add_argument( - "--project_id", help="Project ID you want to access.", required=True - ) - parser.add_argument( - "--zone", help="Zone to create clusters in/connect to", required=True + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, ) parser.add_argument( - "--cluster_name", help="Name of the cluster to create/connect to", required=True + "--project_id", + type=str, + required=True, + help="Project to use for creating resources.", ) parser.add_argument( - "--gcs_bucket", help="Bucket to upload Pyspark file to", required=True + "--region", + type=str, + required=True, + help="Region where the resources should live.", ) parser.add_argument( - "--pyspark_file", help="Pyspark filename. Defaults to pyspark_sort.py" + "--cluster_name", + type=str, + required=True, + help="Name to use for creating a cluster.", ) parser.add_argument( - "--create_new_cluster", - action="store_true", - help="States if the cluster should be created", - ) - parser.add_argument( - "--global_region", - action="store_true", - help="If cluster is in the global region", + "--job_file_path", + type=str, + required=True, + help="Job in Cloud Storage to run on the cluster.", ) args = parser.parse_args() - main( - args.project_id, - args.zone, - args.cluster_name, - args.gcs_bucket, - args.pyspark_file, - args.create_new_cluster, - args.global_region, - ) + quickstart(args.project_id, args.region, args.cluster_name, args.job_file_path) +# [END dataproc_quickstart] From 5fce9120eb2a0efc3fa3a4e6cb5d0dc77fd704a6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 8 Apr 2022 01:13:05 +0200 Subject: [PATCH 122/158] chore(deps): update all dependencies (#399) --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 733072cca58d..d82eb290f4ec 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==1.11.1 grpcio==1.44.0 -google-auth==2.6.2 +google-auth==2.6.3 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.2.1 -google-cloud-dataproc==4.0.1 +google-cloud-dataproc==4.0.2 From 4c3706270c9e22c5c9b76cf72ba08fdee25fdcee Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 13 Apr 2022 00:53:24 +0200 Subject: [PATCH 123/158] chore(deps): update dependency google-auth to v2.6.4 (#402) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d82eb290f4ec..491a42574f4b 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.44.0 -google-auth==2.6.3 +google-auth==2.6.4 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.2.1 From cca6c6a8ed7e70902693d2553c992a8f24e1e73a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 14 Apr 2022 01:53:12 +0200 Subject: [PATCH 124/158] chore(deps): update dependency google-cloud-storage to v2.3.0 (#403) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 491a42574f4b..d3c4f1b5f6d6 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.44.0 google-auth==2.6.4 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.2.1 +google-cloud-storage==2.3.0 google-cloud-dataproc==4.0.2 From d864ffa059fe4ae38e31ac7dfcc12e51c1b2827c Mon Sep 17 00:00:00 2001 From: aman-ebay Date: Thu, 14 Apr 2022 09:45:56 -0700 Subject: [PATCH 125/158] Update python-api-walkthrough.md (#398) Co-authored-by: meredithslota Co-authored-by: Lo Ferris <50979514+loferris@users.noreply.github.com> --- dataproc/snippets/python-api-walkthrough.md | 166 +++++++++----------- 1 file changed, 74 insertions(+), 92 deletions(-) diff --git a/dataproc/snippets/python-api-walkthrough.md b/dataproc/snippets/python-api-walkthrough.md index 29161bbacef5..6aec97d3bcbe 100644 --- a/dataproc/snippets/python-api-walkthrough.md +++ b/dataproc/snippets/python-api-walkthrough.md @@ -13,10 +13,9 @@ As you follow this walkthrough, you run Python code that calls [Dataproc gRPC APIs](https://cloud.google.com/dataproc/docs/reference/rpc/) to: -* create a Dataproc cluster -* submit a small PySpark word sort job to run on the cluster -* get job status -* tear down the cluster after job completion +* Create a Dataproc cluster +* Submit a PySpark word sort job to the cluster +* Delete the cluster after job completion ## Using the walkthrough @@ -32,27 +31,27 @@ an explanation of how the code works. cloudshell launch-tutorial python-api-walkthrough.md -**To copy and run commands**: Click the "Paste in Cloud Shell" button +**To copy and run commands**: Click the "Copy to Cloud Shell" button () on the side of a code box, then press `Enter` to run the command. ## Prerequisites (1) - + 1. Create or select a Google Cloud project to use for this -tutorial. -* + tutorial. + * 1. Enable the Dataproc, Compute Engine, and Cloud Storage APIs in your -project. -```sh -gcloud services enable dataproc.googleapis.com \ -compute.googleapis.com \ -storage-component.googleapis.com \ ---project={{project_id}} -``` + project. + + ```bash + gcloud services enable dataproc.googleapis.com \ + compute.googleapis.com \ + storage-component.googleapis.com \ + --project={{project_id}} + ``` ## Prerequisites (2) @@ -60,116 +59,99 @@ storage-component.googleapis.com \ [Cloud Storage bucket](https://cloud.google.com/storage/docs/key-terms#buckets) in your project. * You can use the [Cloud Storage browser page](https://console.cloud.google.com/storage/browser) - in Google Cloud Platform Console to view existing buckets in your project. + in Google Cloud Console to view existing buckets in your project. -     **OR** + **OR** * To create a new bucket, run the following command. Your bucket name must be unique. - ```bash - gsutil mb -p {{project-id}} gs://your-bucket-name - ``` -1. Set environment variables. + gsutil mb -p {{project-id}} gs://your-bucket-name + - * Set the name of your bucket. - ```bash - BUCKET=your-bucket-name - ``` +2. Set environment variables. + * Set the name of your bucket. + + BUCKET=your-bucket-name ## Prerequisites (3) 1. Set up a Python - [virtual environment](https://virtualenv.readthedocs.org/en/latest/) - in Cloud Shell. + [virtual environment](https://virtualenv.readthedocs.org/en/latest/). * Create the virtual environment. - ```bash - virtualenv ENV - ``` + + virtualenv ENV + * Activate the virtual environment. - ```bash - source ENV/bin/activate - ``` + + source ENV/bin/activate -1. Install library dependencies in Cloud Shell. - ```bash - pip install -r requirements.txt - ``` +1. Install library dependencies. + + pip install -r requirements.txt ## Create a cluster and submit a job 1. Set a name for your new cluster. - ```bash - CLUSTER=new-cluster-name - ``` -1. Set a [zone](https://cloud.google.com/compute/docs/regions-zones/#available) - where your new cluster will be located. You can change the - "us-central1-a" zone that is pre-set in the following command. - ```bash - ZONE=us-central1-a - ``` + CLUSTER=new-cluster-name -1. Run `submit_job.py` with the `--create_new_cluster` flag - to create a new cluster and submit the `pyspark_sort.py` job - to the cluster. +1. Set a [region](https://cloud.google.com/compute/docs/regions-zones/#available) + where your new cluster will be located. You can change the pre-set + "us-central1" region beforew you copy and run the following command. - ```bash - python submit_job_to_cluster.py \ - --project_id={{project-id}} \ - --cluster_name=$CLUSTER \ - --zone=$ZONE \ - --gcs_bucket=$BUCKET \ - --create_new_cluster - ``` + REGION=us-central1 + +1. Run `submit_job_to_cluster.py` to create a new cluster and run the + `pyspark_sort.py` job on the cluster. + + python submit_job_to_cluster.py \ + --project_id={{project-id}} \ + --cluster_name=$CLUSTER \ + --region=$REGION \ + --gcs_bucket=$BUCKET ## Job Output -Job output in Cloud Shell shows cluster creation, job submission, - job completion, and then tear-down of the cluster. - - ... - Creating cluster... - Cluster created. - Uploading pyspark file to Cloud Storage. - new-cluster-name - RUNNING - Submitted job ID ... - Waiting for job to finish... - Job finished. - Downloading output file - ..... - ['Hello,', 'dog', 'elephant', 'panther', 'world!'] - ... - Tearing down cluster - ``` -## Congratulations on Completing the Walkthrough! +Job output displayed in the Cloud Shell terminaL shows cluster creation, +job completion, sorted job output, and then deletion of the cluster. + +```xml +Cluster created successfully: cliuster-name. +... +Job finished successfully. +... +['Hello,', 'dog', 'elephant', 'panther', 'world!'] +... +Cluster cluster-name successfully deleted. +``` + +## Congratulations on completing the Walkthrough! --- ### Next Steps: -* **View job details from the Console.** View job details by selecting the - PySpark job from the Dataproc -= +* **View job details in the Cloud Console.** View job details by selecting the + PySpark job name on the Dataproc [Jobs page](https://console.cloud.google.com/dataproc/jobs) - in the Google Cloud Platform Console. + in the Cloud console. * **Delete resources used in the walkthrough.** - The `submit_job_to_cluster.py` job deletes the cluster that it created for this + The `submit_job_to_cluster.py` code deletes the cluster that it created for this walkthrough. - If you created a bucket to use for this walkthrough, - you can run the following command to delete the - Cloud Storage bucket (the bucket must be empty). - ```bash - gsutil rb gs://$BUCKET - ``` - You can run the following command to delete the bucket **and all - objects within it. Note: the deleted objects cannot be recovered.** - ```bash - gsutil rm -r gs://$BUCKET - ``` + If you created a Cloud Storage bucket to use for this walkthrough, + you can run the following command to delete the bucket (the bucket must be empty). + + gsutil rb gs://$BUCKET + + * You can run the following command to **delete the bucket and all + objects within it. Note: the deleted objects cannot be recovered.** + + gsutil rm -r gs://$BUCKET + * **For more information.** See the [Dataproc documentation](https://cloud.google.com/dataproc/docs/) for API reference and product feature information. From fc20d4c4ae84676edd3bbd9151a293f0dc99ecd4 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 15 Apr 2022 02:45:03 +0200 Subject: [PATCH 126/158] chore(deps): update dependency google-auth to v2.6.5 (#406) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d3c4f1b5f6d6..b9874770edb9 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.44.0 -google-auth==2.6.4 +google-auth==2.6.5 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.3.0 From 69987e363918f925b81386e2ba01f94b74693102 Mon Sep 17 00:00:00 2001 From: aman-ebay Date: Fri, 15 Apr 2022 21:13:14 -0400 Subject: [PATCH 127/158] docs: Dataproc ebay walkthrough update (#405) * Update python-api-walkthrough.md Dataproc python walkthrough update. * Update submit_job_to_cluster.py Dataproc python walkthrough update * Update submit_job_to_cluster.py docs: fix lint errors. --- dataproc/snippets/python-api-walkthrough.md | 2 +- dataproc/snippets/submit_job_to_cluster.py | 64 ++++++++++++++++++--- 2 files changed, 58 insertions(+), 8 deletions(-) diff --git a/dataproc/snippets/python-api-walkthrough.md b/dataproc/snippets/python-api-walkthrough.md index 6aec97d3bcbe..c5eb884a8f01 100644 --- a/dataproc/snippets/python-api-walkthrough.md +++ b/dataproc/snippets/python-api-walkthrough.md @@ -107,8 +107,8 @@ an explanation of how the code works. python submit_job_to_cluster.py \ --project_id={{project-id}} \ - --cluster_name=$CLUSTER \ --region=$REGION \ + --cluster_name=$CLUSTER \ --gcs_bucket=$BUCKET ## Job Output diff --git a/dataproc/snippets/submit_job_to_cluster.py b/dataproc/snippets/submit_job_to_cluster.py index b543f8e38992..506174df825f 100644 --- a/dataproc/snippets/submit_job_to_cluster.py +++ b/dataproc/snippets/submit_job_to_cluster.py @@ -26,14 +26,59 @@ """ import argparse +import os import re from google.cloud import dataproc_v1 from google.cloud import storage +DEFAULT_FILENAME = "pyspark_sort.py" +waiting_callback = False + + +def get_pyspark_file(pyspark_file=None): + if pyspark_file: + f = open(pyspark_file, "rb") + return f, os.path.basename(pyspark_file) + else: + """Gets the PySpark file from current directory.""" + current_dir = os.path.dirname(os.path.abspath(__file__)) + f = open(os.path.join(current_dir, DEFAULT_FILENAME), "rb") + return f, DEFAULT_FILENAME + + +def get_region_from_zone(zone): + try: + region_as_list = zone.split("-")[:-1] + return "-".join(region_as_list) + except (AttributeError, IndexError, ValueError): + raise ValueError("Invalid zone provided, please check your input.") + + +def upload_pyspark_file(project, bucket_name, filename, spark_file): + """Uploads the PySpark file in this directory to the configured input + bucket.""" + print("Uploading pyspark file to Cloud Storage.") + client = storage.Client(project=project) + bucket = client.get_bucket(bucket_name) + blob = bucket.blob(filename) + blob.upload_from_file(spark_file) + + +def download_output(project, cluster_id, output_bucket, job_id): + """Downloads the output file from Cloud Storage and returns it as a + string.""" + print("Downloading output file.") + client = storage.Client(project=project) + bucket = client.get_bucket(output_bucket) + output_blob = "google-cloud-dataproc-metainfo/{}/jobs/{}/driveroutput.000000000".format( + cluster_id, job_id + ) + return bucket.blob(output_blob).download_as_string() + # [START dataproc_create_cluster] -def quickstart(project_id, region, cluster_name, job_file_path): +def quickstart(project_id, region, cluster_name, gcs_bucket, pyspark_file): # Create the cluster client. cluster_client = dataproc_v1.ClusterControllerClient( client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)} @@ -59,6 +104,9 @@ def quickstart(project_id, region, cluster_name, job_file_path): # [END dataproc_create_cluster] + spark_file, spark_filename = get_pyspark_file(pyspark_file) + upload_pyspark_file(project_id, gcs_bucket, spark_filename, spark_file) + # [START dataproc_submit_job] # Create the job client. job_client = dataproc_v1.JobControllerClient( @@ -68,7 +116,7 @@ def quickstart(project_id, region, cluster_name, job_file_path): # Create the job config. job = { "placement": {"cluster_name": cluster_name}, - "pyspark_job": {"main_python_file_uri": job_file_path}, + "pyspark_job": {"main_python_file_uri": "gs://{}/{}".format(gcs_bucket, spark_filename)}, } operation = job_client.submit_job_as_operation( @@ -128,13 +176,15 @@ def quickstart(project_id, region, cluster_name, job_file_path): required=True, help="Name to use for creating a cluster.", ) + parser.add_argument( - "--job_file_path", - type=str, - required=True, - help="Job in Cloud Storage to run on the cluster.", + "--gcs_bucket", help="Bucket to upload Pyspark file to", required=True + ) + + parser.add_argument( + "--pyspark_file", help="Pyspark filename. Defaults to pyspark_sort.py" ) args = parser.parse_args() - quickstart(args.project_id, args.region, args.cluster_name, args.job_file_path) + quickstart(args.project_id, args.region, args.cluster_name, args.gcs_bucket, args.pyspark_file) # [END dataproc_quickstart] From 37da1e24b6c5037cb47b82cb99a7943f846bf051 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 20 Apr 2022 21:35:26 -0400 Subject: [PATCH 128/158] chore(python): add nox session to sort python imports (#408) Source-Link: https://github.com/googleapis/synthtool/commit/1b71c10e20de7ed3f97f692f99a0e3399b67049f Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:00c9d764fd1cd56265f12a5ef4b99a0c9e87cf261018099141e2ca5158890416 Co-authored-by: Owl Bot --- dataproc/snippets/noxfile.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 949e0fde9ae1..38bb0a572b81 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -30,6 +30,7 @@ # WARNING - WARNING - WARNING - WARNING - WARNING BLACK_VERSION = "black==22.3.0" +ISORT_VERSION = "isort==5.10.1" # Copy `noxfile_config.py` to your directory and modify it instead. @@ -168,12 +169,32 @@ def lint(session: nox.sessions.Session) -> None: @nox.session def blacken(session: nox.sessions.Session) -> None: + """Run black. Format code to uniform standard.""" session.install(BLACK_VERSION) python_files = [path for path in os.listdir(".") if path.endswith(".py")] session.run("black", *python_files) +# +# format = isort + black +# + +@nox.session +def format(session: nox.sessions.Session) -> None: + """ + Run isort to sort imports. Then run black + to format code to uniform standard. + """ + session.install(BLACK_VERSION, ISORT_VERSION) + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + # Use the --fss option to sort imports using strict alphabetical order. + # See https://pycqa.github.io/isort/docs/configuration/options.html#force-sort-within-sections + session.run("isort", "--fss", *python_files) + session.run("black", *python_files) + + # # Sample Tests # From 66b9606c1d2b9f521b11ab6999bccb58ddfadd26 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 22 Apr 2022 10:27:40 +0200 Subject: [PATCH 129/158] chore(deps): update dependency google-auth to v2.6.6 (#411) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index b9874770edb9..d2342080c1d3 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==1.11.1 grpcio==1.44.0 -google-auth==2.6.5 +google-auth==2.6.6 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.3.0 From f97520408da91414470d31a07d72eda8e13dc99a Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 25 Apr 2022 18:02:04 +0200 Subject: [PATCH 130/158] chore(deps): update dependency pytest to v7.1.2 (#412) --- dataproc/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index cfc646faf186..4370c036f859 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1,2 +1,2 @@ -pytest==7.1.1 +pytest==7.1.2 pytest-xdist==2.5.0 \ No newline at end of file From ffead3245e1edc34f4900ade3d6eb99c0e1a9099 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 26 Apr 2022 23:19:12 +0200 Subject: [PATCH 131/158] chore(deps): update dependency backoff to v2 (#413) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d2342080c1d3..047de8733c3c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ -backoff==1.11.1 +backoff==2.0.0 grpcio==1.44.0 google-auth==2.6.6 google-auth-httplib2==0.1.0 From a2c39fee41b735d0ebd1338a1479664be0060704 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 27 Apr 2022 18:10:22 +0200 Subject: [PATCH 132/158] chore(deps): update dependency backoff to v2.0.1 (#414) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 047de8733c3c..4dcb06ceda61 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ -backoff==2.0.0 +backoff==2.0.1 grpcio==1.44.0 google-auth==2.6.6 google-auth-httplib2==0.1.0 From 69539de4de6c4333f95d06301c38e379e728c074 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 5 May 2022 02:44:22 +0200 Subject: [PATCH 133/158] chore(deps): update dependency grpcio to v1.46.0 (#415) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 4dcb06ceda61..f5f2bf6f335a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.0.1 -grpcio==1.44.0 +grpcio==1.46.0 google-auth==2.6.6 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From b4f4b29ae2a95dba62d5b7ec9a6159638f4a07d0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 12 May 2022 20:17:50 +0200 Subject: [PATCH 134/158] chore(deps): update dependency grpcio to v1.46.1 (#420) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index f5f2bf6f335a..fe8343a3702a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.0.1 -grpcio==1.46.0 +grpcio==1.46.1 google-auth==2.6.6 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 61ac3526afddb871735bdaece6344708eeef20cd Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 22 May 2022 20:08:44 +0200 Subject: [PATCH 135/158] chore(deps): update dependency grpcio to v1.46.3 (#422) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index fe8343a3702a..c70fa8314d56 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.0.1 -grpcio==1.46.1 +grpcio==1.46.3 google-auth==2.6.6 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 89292c62b53402f8c3e8a38629955164b85785cd Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Sun, 10 Jul 2022 13:23:31 -0400 Subject: [PATCH 136/158] fix: require python 3.7+ (#442) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(python): drop python 3.6 Source-Link: https://github.com/googleapis/synthtool/commit/4f89b13af10d086458f9b379e56a614f9d6dab7b Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:e7bb19d47c13839fe8c147e50e02e8b6cf5da8edd1af8b82208cd6f66cc2829c * add api_description to .repo-metadata.json * require python 3.7+ in setup.py * remove python 3.6 sample configs * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * trigger CI Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 38bb0a572b81..5fcb9d7461f2 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9", "3.10"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From c9e161af6bea29a2fd7c3c81a564ea5fc47ce2b9 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 15 Jul 2022 18:07:09 +0200 Subject: [PATCH 137/158] chore(deps): update all dependencies (#428) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index c70fa8314d56..d4e44212d45c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ -backoff==2.0.1 +backoff==2.1.2 grpcio==1.46.3 -google-auth==2.6.6 +google-auth==2.7.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.3.0 -google-cloud-dataproc==4.0.2 +google-cloud-storage==2.4.0 +google-cloud-dataproc==4.0.3 From 383e5350fc63a7260f64876af42b7ddc957346e6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 2 Aug 2022 15:56:39 +0200 Subject: [PATCH 138/158] chore(deps): update all dependencies (#451) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d4e44212d45c..1d6838db83fd 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==2.1.2 -grpcio==1.46.3 -google-auth==2.7.0 +grpcio==1.48.0 +google-auth==2.9.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.4.0 -google-cloud-dataproc==4.0.3 +google-cloud-dataproc==5.0.0 From b5f9f3e9770a0206fb5cdacf3e2dc5105c71e330 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 6 Aug 2022 02:26:20 +0200 Subject: [PATCH 139/158] chore(deps): update all dependencies (#453) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 1d6838db83fd..e26f9da8f0d7 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -4,5 +4,5 @@ grpcio==1.48.0 google-auth==2.9.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 -google-cloud-storage==2.4.0 +google-cloud-storage==2.5.0 google-cloud-dataproc==5.0.0 From 180cdfd61a1cff18ccd441a7a570884124b41c15 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 6 Aug 2022 03:46:30 +0200 Subject: [PATCH 140/158] chore(deps): update all dependencies (#454) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e26f9da8f0d7..f8c1bdbb9de6 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==2.1.2 grpcio==1.48.0 -google-auth==2.9.1 +google-auth==2.10.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 From b5af4e7bdbe0d5bad0adcb14286962dd978f5ea3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 19 Aug 2022 18:32:00 +0200 Subject: [PATCH 141/158] chore(deps): update dependency google-cloud-dataproc to v5.0.1 (#460) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index f8c1bdbb9de6..c133f53ee403 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.10.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 -google-cloud-dataproc==5.0.0 +google-cloud-dataproc==5.0.1 From 99c3f19c3e662ea58fdeed3c38563e96a3a2fbfc Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 23 Aug 2022 16:19:42 +0200 Subject: [PATCH 142/158] chore(deps): update dependency google-auth to v2.11.0 (#461) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index c133f53ee403..212749126010 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==2.1.2 grpcio==1.48.0 -google-auth==2.10.0 +google-auth==2.11.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 From a88d34c2301043437786e9a34534e3791d5acd88 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 6 Sep 2022 17:49:59 +0200 Subject: [PATCH 143/158] chore(deps): update all dependencies (#473) --- dataproc/snippets/requirements-test.txt | 2 +- dataproc/snippets/requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 4370c036f859..1f2265d6ce77 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1,2 +1,2 @@ -pytest==7.1.2 +pytest==7.1.3 pytest-xdist==2.5.0 \ No newline at end of file diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 212749126010..d131f2a5f731 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.1.2 -grpcio==1.48.0 +grpcio==1.48.1 google-auth==2.11.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 0889ffea78b382069c2e1aa709907788972cfab3 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 13 Sep 2022 17:38:14 +0000 Subject: [PATCH 144/158] chore: detect samples tests in nested directories (#477) Source-Link: https://github.com/googleapis/synthtool/commit/50db768f450a50d7c1fd62513c113c9bb96fd434 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:e09366bdf0fd9c8976592988390b24d53583dd9f002d476934da43725adbb978 --- dataproc/snippets/noxfile.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py index 5fcb9d7461f2..0398d72ff690 100644 --- a/dataproc/snippets/noxfile.py +++ b/dataproc/snippets/noxfile.py @@ -207,8 +207,8 @@ def _session_tests( session: nox.sessions.Session, post_install: Callable = None ) -> None: # check for presence of tests - test_list = glob.glob("*_test.py") + glob.glob("test_*.py") - test_list.extend(glob.glob("tests")) + test_list = glob.glob("**/*_test.py", recursive=True) + glob.glob("**/test_*.py", recursive=True) + test_list.extend(glob.glob("**/tests", recursive=True)) if len(test_list) == 0: print("No tests found, skipping directory.") From 94473a2cc0ac21df43ffeae21a53870b62ea025c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 16 Sep 2022 17:53:56 +0200 Subject: [PATCH 145/158] chore(deps): update dependency grpcio to v1.49.0 (#479) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index d131f2a5f731..bbfb6638ba56 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.1.2 -grpcio==1.48.1 +grpcio==1.49.0 google-auth==2.11.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 4526cad89d00fd877ec55445206ed4c45e37ed5b Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 20 Sep 2022 13:20:46 +0200 Subject: [PATCH 146/158] chore(deps): update dependency google-auth to v2.11.1 (#480) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index bbfb6638ba56..7d1fe41dc69a 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==2.1.2 grpcio==1.49.0 -google-auth==2.11.0 +google-auth==2.11.1 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 From 0e21d8755b5aa9b4c85b2ae807452b0493e7e5ee Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 3 Oct 2022 17:39:21 +0200 Subject: [PATCH 147/158] chore(deps): update all dependencies (#483) --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 7d1fe41dc69a..8a39da83ede2 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==2.1.2 -grpcio==1.49.0 -google-auth==2.11.1 +grpcio==1.49.1 +google-auth==2.12.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 From d3ae234c9ba8b2b9e21559154224f31215b27d03 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 4 Oct 2022 15:32:42 +0200 Subject: [PATCH 148/158] chore(deps): update dependency google-cloud-dataproc to v5.0.2 (#488) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 8a39da83ede2..50d2bdbeb5bb 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -5,4 +5,4 @@ google-auth==2.12.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 -google-cloud-dataproc==5.0.1 +google-cloud-dataproc==5.0.2 From 48acea78f3bede577dac98e77d3b84e29832b6f6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 6 Oct 2022 15:40:16 +0200 Subject: [PATCH 149/158] chore(deps): update dependency backoff to v2.2.1 (#489) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 50d2bdbeb5bb..a5c953b4cd4c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,5 +1,5 @@ -backoff==2.1.2 +backoff==2.2.1 grpcio==1.49.1 google-auth==2.12.0 google-auth-httplib2==0.1.0 From fe80dcc34383d0d0e381b1d98762350479b9dda7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 18 Oct 2022 15:24:04 +0200 Subject: [PATCH 150/158] chore(deps): update all dependencies (#492) --- dataproc/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index a5c953b4cd4c..e4915aee5c1c 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,8 +1,8 @@ backoff==2.2.1 grpcio==1.49.1 -google-auth==2.12.0 +google-auth==2.13.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 -google-cloud-dataproc==5.0.2 +google-cloud-dataproc==5.0.3 From f77ae7cdb62871b12a436726d8ba06cca55b325b Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 19 Oct 2022 16:04:17 +0200 Subject: [PATCH 151/158] chore(deps): update dependency grpcio to v1.50.0 (#493) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index e4915aee5c1c..658daf9f0951 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,6 +1,6 @@ backoff==2.2.1 -grpcio==1.49.1 +grpcio==1.50.0 google-auth==2.13.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 From 6b1b7ddbf68443d67e96a4119cef4e623d974986 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 26 Oct 2022 12:56:49 +0200 Subject: [PATCH 152/158] chore(deps): update all dependencies (#494) --- dataproc/snippets/requirements-test.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dataproc/snippets/requirements-test.txt b/dataproc/snippets/requirements-test.txt index 1f2265d6ce77..0c53ad2e6787 100644 --- a/dataproc/snippets/requirements-test.txt +++ b/dataproc/snippets/requirements-test.txt @@ -1,2 +1,2 @@ -pytest==7.1.3 -pytest-xdist==2.5.0 \ No newline at end of file +pytest==7.2.0 +pytest-xdist==3.0.2 \ No newline at end of file From 0edadf4d536aa7c7a5dbf518f8730554b68bebc3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 1 Nov 2022 14:08:42 +0100 Subject: [PATCH 153/158] chore(deps): update dependency google-auth to v2.14.0 (#496) --- dataproc/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataproc/snippets/requirements.txt b/dataproc/snippets/requirements.txt index 658daf9f0951..9414a8d384a9 100644 --- a/dataproc/snippets/requirements.txt +++ b/dataproc/snippets/requirements.txt @@ -1,7 +1,7 @@ backoff==2.2.1 grpcio==1.50.0 -google-auth==2.13.0 +google-auth==2.14.0 google-auth-httplib2==0.1.0 google-cloud==0.34.0 google-cloud-storage==2.5.0 From 192d724cafc18846f5e89abe46d6bca31955b1d4 Mon Sep 17 00:00:00 2001 From: Sampath M Date: Mon, 14 Nov 2022 16:44:39 +0100 Subject: [PATCH 154/158] Remove (redundant) noxfile.py --- dataproc/snippets/noxfile.py | 312 ---------------------------- dataproc/snippets/noxfile_config.py | 42 ---- 2 files changed, 354 deletions(-) delete mode 100644 dataproc/snippets/noxfile.py delete mode 100644 dataproc/snippets/noxfile_config.py diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py deleted file mode 100644 index 0398d72ff690..000000000000 --- a/dataproc/snippets/noxfile.py +++ /dev/null @@ -1,312 +0,0 @@ -# Copyright 2019 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from __future__ import print_function - -import glob -import os -from pathlib import Path -import sys -from typing import Callable, Dict, List, Optional - -import nox - - -# WARNING - WARNING - WARNING - WARNING - WARNING -# WARNING - WARNING - WARNING - WARNING - WARNING -# DO NOT EDIT THIS FILE EVER! -# WARNING - WARNING - WARNING - WARNING - WARNING -# WARNING - WARNING - WARNING - WARNING - WARNING - -BLACK_VERSION = "black==22.3.0" -ISORT_VERSION = "isort==5.10.1" - -# Copy `noxfile_config.py` to your directory and modify it instead. - -# `TEST_CONFIG` dict is a configuration hook that allows users to -# modify the test configurations. The values here should be in sync -# with `noxfile_config.py`. Users will copy `noxfile_config.py` into -# their directory and modify it. - -TEST_CONFIG = { - # You can opt out from the test for specific Python versions. - "ignored_versions": [], - # Old samples are opted out of enforcing Python type hints - # All new samples should feature them - "enforce_type_hints": False, - # An envvar key for determining the project id to use. Change it - # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a - # build specific Cloud project. You can also use your own string - # to use your own Cloud project. - "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", - # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', - # If you need to use a specific version of pip, - # change pip_version_override to the string representation - # of the version number, for example, "20.2.4" - "pip_version_override": None, - # A dictionary you want to inject into your test. Don't put any - # secrets here. These values will override predefined values. - "envs": {}, -} - - -try: - # Ensure we can import noxfile_config in the project's directory. - sys.path.append(".") - from noxfile_config import TEST_CONFIG_OVERRIDE -except ImportError as e: - print("No user noxfile_config found: detail: {}".format(e)) - TEST_CONFIG_OVERRIDE = {} - -# Update the TEST_CONFIG with the user supplied values. -TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) - - -def get_pytest_env_vars() -> Dict[str, str]: - """Returns a dict for pytest invocation.""" - ret = {} - - # Override the GCLOUD_PROJECT and the alias. - env_key = TEST_CONFIG["gcloud_project_env"] - # This should error out if not set. - ret["GOOGLE_CLOUD_PROJECT"] = os.environ[env_key] - - # Apply user supplied envs. - ret.update(TEST_CONFIG["envs"]) - return ret - - -# DO NOT EDIT - automatically generated. -# All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] - -# Any default versions that should be ignored. -IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] - -TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) - -INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ( - "True", - "true", -) - -# Error if a python version is missing -nox.options.error_on_missing_interpreters = True - -# -# Style Checks -# - - -def _determine_local_import_names(start_dir: str) -> List[str]: - """Determines all import names that should be considered "local". - - This is used when running the linter to insure that import order is - properly checked. - """ - file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] - return [ - basename - for basename, extension in file_ext_pairs - if extension == ".py" - or os.path.isdir(os.path.join(start_dir, basename)) - and basename not in ("__pycache__") - ] - - -# Linting with flake8. -# -# We ignore the following rules: -# E203: whitespace before ‘:’ -# E266: too many leading ‘#’ for block comment -# E501: line too long -# I202: Additional newline in a section of imports -# -# We also need to specify the rules which are ignored by default: -# ['E226', 'W504', 'E126', 'E123', 'W503', 'E24', 'E704', 'E121'] -FLAKE8_COMMON_ARGS = [ - "--show-source", - "--builtin=gettext", - "--max-complexity=20", - "--import-order-style=google", - "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", - "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", - "--max-line-length=88", -] - - -@nox.session -def lint(session: nox.sessions.Session) -> None: - if not TEST_CONFIG["enforce_type_hints"]: - session.install("flake8", "flake8-import-order") - else: - session.install("flake8", "flake8-import-order", "flake8-annotations") - - local_names = _determine_local_import_names(".") - args = FLAKE8_COMMON_ARGS + [ - "--application-import-names", - ",".join(local_names), - ".", - ] - session.run("flake8", *args) - - -# -# Black -# - - -@nox.session -def blacken(session: nox.sessions.Session) -> None: - """Run black. Format code to uniform standard.""" - session.install(BLACK_VERSION) - python_files = [path for path in os.listdir(".") if path.endswith(".py")] - - session.run("black", *python_files) - - -# -# format = isort + black -# - -@nox.session -def format(session: nox.sessions.Session) -> None: - """ - Run isort to sort imports. Then run black - to format code to uniform standard. - """ - session.install(BLACK_VERSION, ISORT_VERSION) - python_files = [path for path in os.listdir(".") if path.endswith(".py")] - - # Use the --fss option to sort imports using strict alphabetical order. - # See https://pycqa.github.io/isort/docs/configuration/options.html#force-sort-within-sections - session.run("isort", "--fss", *python_files) - session.run("black", *python_files) - - -# -# Sample Tests -# - - -PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] - - -def _session_tests( - session: nox.sessions.Session, post_install: Callable = None -) -> None: - # check for presence of tests - test_list = glob.glob("**/*_test.py", recursive=True) + glob.glob("**/test_*.py", recursive=True) - test_list.extend(glob.glob("**/tests", recursive=True)) - - if len(test_list) == 0: - print("No tests found, skipping directory.") - return - - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - concurrent_args = [] - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - with open("requirements.txt") as rfile: - packages = rfile.read() - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install( - "-r", "requirements-test.txt", "-c", "constraints-test.txt" - ) - else: - session.install("-r", "requirements-test.txt") - with open("requirements-test.txt") as rtfile: - packages += rtfile.read() - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - if "pytest-parallel" in packages: - concurrent_args.extend(['--workers', 'auto', '--tests-per-worker', 'auto']) - elif "pytest-xdist" in packages: - concurrent_args.extend(['-n', 'auto']) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs + concurrent_args), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) - - -@nox.session(python=ALL_VERSIONS) -def py(session: nox.sessions.Session) -> None: - """Runs py.test for a sample using the specified version of Python.""" - if session.python in TESTED_VERSIONS: - _session_tests(session) - else: - session.skip( - "SKIPPED: {} tests are disabled for this sample.".format(session.python) - ) - - -# -# Readmegen -# - - -def _get_repo_root() -> Optional[str]: - """ Returns the root folder of the project. """ - # Get root of this repository. Assume we don't have directories nested deeper than 10 items. - p = Path(os.getcwd()) - for i in range(10): - if p is None: - break - if Path(p / ".git").exists(): - return str(p) - # .git is not available in repos cloned via Cloud Build - # setup.py is always in the library's root, so use that instead - # https://github.com/googleapis/synthtool/issues/792 - if Path(p / "setup.py").exists(): - return str(p) - p = p.parent - raise Exception("Unable to detect repository root.") - - -GENERATED_READMES = sorted([x for x in Path(".").rglob("*.rst.in")]) - - -@nox.session -@nox.parametrize("path", GENERATED_READMES) -def readmegen(session: nox.sessions.Session, path: str) -> None: - """(Re-)generates the readme for a sample.""" - session.install("jinja2", "pyyaml") - dir_ = os.path.dirname(path) - - if os.path.exists(os.path.join(dir_, "requirements.txt")): - session.install("-r", os.path.join(dir_, "requirements.txt")) - - in_file = os.path.join(dir_, "README.rst.in") - session.run( - "python", _get_repo_root() + "/scripts/readme-gen/readme_gen.py", in_file - ) diff --git a/dataproc/snippets/noxfile_config.py b/dataproc/snippets/noxfile_config.py deleted file mode 100644 index 646d77de8543..000000000000 --- a/dataproc/snippets/noxfile_config.py +++ /dev/null @@ -1,42 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Default TEST_CONFIG_OVERRIDE for python repos. - -# You can copy this file into your directory, then it will be imported from -# the noxfile.py. - -# The source of truth: -# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/noxfile_config.py - -TEST_CONFIG_OVERRIDE = { - # You can opt out from the test for specific Python versions. - "ignored_versions": ["2.7", "3.6"], - # Old samples are opted out of enforcing Python type hints - # All new samples should feature them - # "enforce_type_hints": True, - # An envvar key for determining the project id to use. Change it - # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a - # build specific Cloud project. You can also use your own string - # to use your own Cloud project. - # "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", - "gcloud_project_env": "BUILD_SPECIFIC_GCLOUD_PROJECT", - # If you need to use a specific version of pip, - # change pip_version_override to the string representation - # of the version number, for example, "20.2.4" - "pip_version_override": None, - # A dictionary you want to inject into your test. Don't put any - # secrets here. These values will override predefined values. - "envs": {}, -} From 3d5271fb97cc85efc71e4d814b6139c30c8808ec Mon Sep 17 00:00:00 2001 From: Sampath M Date: Mon, 14 Nov 2022 16:48:06 +0100 Subject: [PATCH 155/158] Update Licence Header --- dataproc/snippets/dataproc_e2e_donttest.py | 6 +++++- dataproc/snippets/list_clusters.py | 8 ++++++-- dataproc/snippets/pyspark_sort.py | 5 ++++- dataproc/snippets/pyspark_sort_gcs.py | 5 ++++- dataproc/snippets/single_job_workflow.py | 6 +++++- 5 files changed, 24 insertions(+), 6 deletions(-) diff --git a/dataproc/snippets/dataproc_e2e_donttest.py b/dataproc/snippets/dataproc_e2e_donttest.py index a92f46065f25..462a8850d0c4 100644 --- a/dataproc/snippets/dataproc_e2e_donttest.py +++ b/dataproc/snippets/dataproc_e2e_donttest.py @@ -1,8 +1,12 @@ +#!/usr/bin/env python + +# Copyright 2019 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, diff --git a/dataproc/snippets/list_clusters.py b/dataproc/snippets/list_clusters.py index 11acdf5da989..0b36ba610096 100644 --- a/dataproc/snippets/list_clusters.py +++ b/dataproc/snippets/list_clusters.py @@ -1,16 +1,20 @@ #!/usr/bin/env python + +# Copyright 2019 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""Sample command-line program to list Cloud Dataproc clusters in a region. + +""" Sample command-line program to list Cloud Dataproc clusters in a region. Example usage: python list_clusters.py --project_id=my-project-id --region=global diff --git a/dataproc/snippets/pyspark_sort.py b/dataproc/snippets/pyspark_sort.py index e43d57f13186..cecba896164a 100644 --- a/dataproc/snippets/pyspark_sort.py +++ b/dataproc/snippets/pyspark_sort.py @@ -1,9 +1,12 @@ #!/usr/bin/env python + +# Copyright 2019 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, diff --git a/dataproc/snippets/pyspark_sort_gcs.py b/dataproc/snippets/pyspark_sort_gcs.py index 4b1e7b2dca4a..1f07722e78b0 100644 --- a/dataproc/snippets/pyspark_sort_gcs.py +++ b/dataproc/snippets/pyspark_sort_gcs.py @@ -1,9 +1,12 @@ #!/usr/bin/env python + +# Copyright 2019 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, diff --git a/dataproc/snippets/single_job_workflow.py b/dataproc/snippets/single_job_workflow.py index 04f73ea9f282..6f9948f04f85 100644 --- a/dataproc/snippets/single_job_workflow.py +++ b/dataproc/snippets/single_job_workflow.py @@ -1,15 +1,19 @@ #!/usr/bin/env python + +# Copyright 2019 Google LLC +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + r"""Sample Cloud Dataproc inline workflow to run a pyspark job on an ephermeral cluster. Example Usage to run the inline workflow on a managed cluster: From 2f5e8a64fa31f0870ae46edaa2235fcc32cad32f Mon Sep 17 00:00:00 2001 From: Sampath M Date: Tue, 15 Nov 2022 23:33:29 +0100 Subject: [PATCH 156/158] Revert "Remove (redundant) noxfile.py" This reverts commit 192d724cafc18846f5e89abe46d6bca31955b1d4. --- dataproc/snippets/noxfile.py | 312 ++++++++++++++++++++++++++++ dataproc/snippets/noxfile_config.py | 42 ++++ 2 files changed, 354 insertions(+) create mode 100644 dataproc/snippets/noxfile.py create mode 100644 dataproc/snippets/noxfile_config.py diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py new file mode 100644 index 000000000000..0398d72ff690 --- /dev/null +++ b/dataproc/snippets/noxfile.py @@ -0,0 +1,312 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function + +import glob +import os +from pathlib import Path +import sys +from typing import Callable, Dict, List, Optional + +import nox + + +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING +# DO NOT EDIT THIS FILE EVER! +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING + +BLACK_VERSION = "black==22.3.0" +ISORT_VERSION = "isort==5.10.1" + +# Copy `noxfile_config.py` to your directory and modify it instead. + +# `TEST_CONFIG` dict is a configuration hook that allows users to +# modify the test configurations. The values here should be in sync +# with `noxfile_config.py`. Users will copy `noxfile_config.py` into +# their directory and modify it. + +TEST_CONFIG = { + # You can opt out from the test for specific Python versions. + "ignored_versions": [], + # Old samples are opted out of enforcing Python type hints + # All new samples should feature them + "enforce_type_hints": False, + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", + # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', + # If you need to use a specific version of pip, + # change pip_version_override to the string representation + # of the version number, for example, "20.2.4" + "pip_version_override": None, + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + "envs": {}, +} + + +try: + # Ensure we can import noxfile_config in the project's directory. + sys.path.append(".") + from noxfile_config import TEST_CONFIG_OVERRIDE +except ImportError as e: + print("No user noxfile_config found: detail: {}".format(e)) + TEST_CONFIG_OVERRIDE = {} + +# Update the TEST_CONFIG with the user supplied values. +TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) + + +def get_pytest_env_vars() -> Dict[str, str]: + """Returns a dict for pytest invocation.""" + ret = {} + + # Override the GCLOUD_PROJECT and the alias. + env_key = TEST_CONFIG["gcloud_project_env"] + # This should error out if not set. + ret["GOOGLE_CLOUD_PROJECT"] = os.environ[env_key] + + # Apply user supplied envs. + ret.update(TEST_CONFIG["envs"]) + return ret + + +# DO NOT EDIT - automatically generated. +# All versions used to test samples. +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] + +# Any default versions that should be ignored. +IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] + +TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) + +INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ( + "True", + "true", +) + +# Error if a python version is missing +nox.options.error_on_missing_interpreters = True + +# +# Style Checks +# + + +def _determine_local_import_names(start_dir: str) -> List[str]: + """Determines all import names that should be considered "local". + + This is used when running the linter to insure that import order is + properly checked. + """ + file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] + return [ + basename + for basename, extension in file_ext_pairs + if extension == ".py" + or os.path.isdir(os.path.join(start_dir, basename)) + and basename not in ("__pycache__") + ] + + +# Linting with flake8. +# +# We ignore the following rules: +# E203: whitespace before ‘:’ +# E266: too many leading ‘#’ for block comment +# E501: line too long +# I202: Additional newline in a section of imports +# +# We also need to specify the rules which are ignored by default: +# ['E226', 'W504', 'E126', 'E123', 'W503', 'E24', 'E704', 'E121'] +FLAKE8_COMMON_ARGS = [ + "--show-source", + "--builtin=gettext", + "--max-complexity=20", + "--import-order-style=google", + "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", + "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", + "--max-line-length=88", +] + + +@nox.session +def lint(session: nox.sessions.Session) -> None: + if not TEST_CONFIG["enforce_type_hints"]: + session.install("flake8", "flake8-import-order") + else: + session.install("flake8", "flake8-import-order", "flake8-annotations") + + local_names = _determine_local_import_names(".") + args = FLAKE8_COMMON_ARGS + [ + "--application-import-names", + ",".join(local_names), + ".", + ] + session.run("flake8", *args) + + +# +# Black +# + + +@nox.session +def blacken(session: nox.sessions.Session) -> None: + """Run black. Format code to uniform standard.""" + session.install(BLACK_VERSION) + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + session.run("black", *python_files) + + +# +# format = isort + black +# + +@nox.session +def format(session: nox.sessions.Session) -> None: + """ + Run isort to sort imports. Then run black + to format code to uniform standard. + """ + session.install(BLACK_VERSION, ISORT_VERSION) + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + # Use the --fss option to sort imports using strict alphabetical order. + # See https://pycqa.github.io/isort/docs/configuration/options.html#force-sort-within-sections + session.run("isort", "--fss", *python_files) + session.run("black", *python_files) + + +# +# Sample Tests +# + + +PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] + + +def _session_tests( + session: nox.sessions.Session, post_install: Callable = None +) -> None: + # check for presence of tests + test_list = glob.glob("**/*_test.py", recursive=True) + glob.glob("**/test_*.py", recursive=True) + test_list.extend(glob.glob("**/tests", recursive=True)) + + if len(test_list) == 0: + print("No tests found, skipping directory.") + return + + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + concurrent_args = [] + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + with open("requirements.txt") as rfile: + packages = rfile.read() + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install( + "-r", "requirements-test.txt", "-c", "constraints-test.txt" + ) + else: + session.install("-r", "requirements-test.txt") + with open("requirements-test.txt") as rtfile: + packages += rtfile.read() + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + if "pytest-parallel" in packages: + concurrent_args.extend(['--workers', 'auto', '--tests-per-worker', 'auto']) + elif "pytest-xdist" in packages: + concurrent_args.extend(['-n', 'auto']) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs + concurrent_args), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) + + +@nox.session(python=ALL_VERSIONS) +def py(session: nox.sessions.Session) -> None: + """Runs py.test for a sample using the specified version of Python.""" + if session.python in TESTED_VERSIONS: + _session_tests(session) + else: + session.skip( + "SKIPPED: {} tests are disabled for this sample.".format(session.python) + ) + + +# +# Readmegen +# + + +def _get_repo_root() -> Optional[str]: + """ Returns the root folder of the project. """ + # Get root of this repository. Assume we don't have directories nested deeper than 10 items. + p = Path(os.getcwd()) + for i in range(10): + if p is None: + break + if Path(p / ".git").exists(): + return str(p) + # .git is not available in repos cloned via Cloud Build + # setup.py is always in the library's root, so use that instead + # https://github.com/googleapis/synthtool/issues/792 + if Path(p / "setup.py").exists(): + return str(p) + p = p.parent + raise Exception("Unable to detect repository root.") + + +GENERATED_READMES = sorted([x for x in Path(".").rglob("*.rst.in")]) + + +@nox.session +@nox.parametrize("path", GENERATED_READMES) +def readmegen(session: nox.sessions.Session, path: str) -> None: + """(Re-)generates the readme for a sample.""" + session.install("jinja2", "pyyaml") + dir_ = os.path.dirname(path) + + if os.path.exists(os.path.join(dir_, "requirements.txt")): + session.install("-r", os.path.join(dir_, "requirements.txt")) + + in_file = os.path.join(dir_, "README.rst.in") + session.run( + "python", _get_repo_root() + "/scripts/readme-gen/readme_gen.py", in_file + ) diff --git a/dataproc/snippets/noxfile_config.py b/dataproc/snippets/noxfile_config.py new file mode 100644 index 000000000000..646d77de8543 --- /dev/null +++ b/dataproc/snippets/noxfile_config.py @@ -0,0 +1,42 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Default TEST_CONFIG_OVERRIDE for python repos. + +# You can copy this file into your directory, then it will be imported from +# the noxfile.py. + +# The source of truth: +# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/noxfile_config.py + +TEST_CONFIG_OVERRIDE = { + # You can opt out from the test for specific Python versions. + "ignored_versions": ["2.7", "3.6"], + # Old samples are opted out of enforcing Python type hints + # All new samples should feature them + # "enforce_type_hints": True, + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + # "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", + "gcloud_project_env": "BUILD_SPECIFIC_GCLOUD_PROJECT", + # If you need to use a specific version of pip, + # change pip_version_override to the string representation + # of the version number, for example, "20.2.4" + "pip_version_override": None, + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + "envs": {}, +} From 1cd824897d3d1924202ce431b6d637dac46455ba Mon Sep 17 00:00:00 2001 From: Sampath M Date: Tue, 15 Nov 2022 23:35:58 +0100 Subject: [PATCH 157/158] Removing only noxfile.py --- dataproc/snippets/noxfile.py | 312 ----------------------------------- 1 file changed, 312 deletions(-) delete mode 100644 dataproc/snippets/noxfile.py diff --git a/dataproc/snippets/noxfile.py b/dataproc/snippets/noxfile.py deleted file mode 100644 index 0398d72ff690..000000000000 --- a/dataproc/snippets/noxfile.py +++ /dev/null @@ -1,312 +0,0 @@ -# Copyright 2019 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from __future__ import print_function - -import glob -import os -from pathlib import Path -import sys -from typing import Callable, Dict, List, Optional - -import nox - - -# WARNING - WARNING - WARNING - WARNING - WARNING -# WARNING - WARNING - WARNING - WARNING - WARNING -# DO NOT EDIT THIS FILE EVER! -# WARNING - WARNING - WARNING - WARNING - WARNING -# WARNING - WARNING - WARNING - WARNING - WARNING - -BLACK_VERSION = "black==22.3.0" -ISORT_VERSION = "isort==5.10.1" - -# Copy `noxfile_config.py` to your directory and modify it instead. - -# `TEST_CONFIG` dict is a configuration hook that allows users to -# modify the test configurations. The values here should be in sync -# with `noxfile_config.py`. Users will copy `noxfile_config.py` into -# their directory and modify it. - -TEST_CONFIG = { - # You can opt out from the test for specific Python versions. - "ignored_versions": [], - # Old samples are opted out of enforcing Python type hints - # All new samples should feature them - "enforce_type_hints": False, - # An envvar key for determining the project id to use. Change it - # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a - # build specific Cloud project. You can also use your own string - # to use your own Cloud project. - "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", - # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', - # If you need to use a specific version of pip, - # change pip_version_override to the string representation - # of the version number, for example, "20.2.4" - "pip_version_override": None, - # A dictionary you want to inject into your test. Don't put any - # secrets here. These values will override predefined values. - "envs": {}, -} - - -try: - # Ensure we can import noxfile_config in the project's directory. - sys.path.append(".") - from noxfile_config import TEST_CONFIG_OVERRIDE -except ImportError as e: - print("No user noxfile_config found: detail: {}".format(e)) - TEST_CONFIG_OVERRIDE = {} - -# Update the TEST_CONFIG with the user supplied values. -TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) - - -def get_pytest_env_vars() -> Dict[str, str]: - """Returns a dict for pytest invocation.""" - ret = {} - - # Override the GCLOUD_PROJECT and the alias. - env_key = TEST_CONFIG["gcloud_project_env"] - # This should error out if not set. - ret["GOOGLE_CLOUD_PROJECT"] = os.environ[env_key] - - # Apply user supplied envs. - ret.update(TEST_CONFIG["envs"]) - return ret - - -# DO NOT EDIT - automatically generated. -# All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] - -# Any default versions that should be ignored. -IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] - -TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) - -INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ( - "True", - "true", -) - -# Error if a python version is missing -nox.options.error_on_missing_interpreters = True - -# -# Style Checks -# - - -def _determine_local_import_names(start_dir: str) -> List[str]: - """Determines all import names that should be considered "local". - - This is used when running the linter to insure that import order is - properly checked. - """ - file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] - return [ - basename - for basename, extension in file_ext_pairs - if extension == ".py" - or os.path.isdir(os.path.join(start_dir, basename)) - and basename not in ("__pycache__") - ] - - -# Linting with flake8. -# -# We ignore the following rules: -# E203: whitespace before ‘:’ -# E266: too many leading ‘#’ for block comment -# E501: line too long -# I202: Additional newline in a section of imports -# -# We also need to specify the rules which are ignored by default: -# ['E226', 'W504', 'E126', 'E123', 'W503', 'E24', 'E704', 'E121'] -FLAKE8_COMMON_ARGS = [ - "--show-source", - "--builtin=gettext", - "--max-complexity=20", - "--import-order-style=google", - "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", - "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", - "--max-line-length=88", -] - - -@nox.session -def lint(session: nox.sessions.Session) -> None: - if not TEST_CONFIG["enforce_type_hints"]: - session.install("flake8", "flake8-import-order") - else: - session.install("flake8", "flake8-import-order", "flake8-annotations") - - local_names = _determine_local_import_names(".") - args = FLAKE8_COMMON_ARGS + [ - "--application-import-names", - ",".join(local_names), - ".", - ] - session.run("flake8", *args) - - -# -# Black -# - - -@nox.session -def blacken(session: nox.sessions.Session) -> None: - """Run black. Format code to uniform standard.""" - session.install(BLACK_VERSION) - python_files = [path for path in os.listdir(".") if path.endswith(".py")] - - session.run("black", *python_files) - - -# -# format = isort + black -# - -@nox.session -def format(session: nox.sessions.Session) -> None: - """ - Run isort to sort imports. Then run black - to format code to uniform standard. - """ - session.install(BLACK_VERSION, ISORT_VERSION) - python_files = [path for path in os.listdir(".") if path.endswith(".py")] - - # Use the --fss option to sort imports using strict alphabetical order. - # See https://pycqa.github.io/isort/docs/configuration/options.html#force-sort-within-sections - session.run("isort", "--fss", *python_files) - session.run("black", *python_files) - - -# -# Sample Tests -# - - -PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] - - -def _session_tests( - session: nox.sessions.Session, post_install: Callable = None -) -> None: - # check for presence of tests - test_list = glob.glob("**/*_test.py", recursive=True) + glob.glob("**/test_*.py", recursive=True) - test_list.extend(glob.glob("**/tests", recursive=True)) - - if len(test_list) == 0: - print("No tests found, skipping directory.") - return - - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - concurrent_args = [] - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - with open("requirements.txt") as rfile: - packages = rfile.read() - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install( - "-r", "requirements-test.txt", "-c", "constraints-test.txt" - ) - else: - session.install("-r", "requirements-test.txt") - with open("requirements-test.txt") as rtfile: - packages += rtfile.read() - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - if "pytest-parallel" in packages: - concurrent_args.extend(['--workers', 'auto', '--tests-per-worker', 'auto']) - elif "pytest-xdist" in packages: - concurrent_args.extend(['-n', 'auto']) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs + concurrent_args), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) - - -@nox.session(python=ALL_VERSIONS) -def py(session: nox.sessions.Session) -> None: - """Runs py.test for a sample using the specified version of Python.""" - if session.python in TESTED_VERSIONS: - _session_tests(session) - else: - session.skip( - "SKIPPED: {} tests are disabled for this sample.".format(session.python) - ) - - -# -# Readmegen -# - - -def _get_repo_root() -> Optional[str]: - """ Returns the root folder of the project. """ - # Get root of this repository. Assume we don't have directories nested deeper than 10 items. - p = Path(os.getcwd()) - for i in range(10): - if p is None: - break - if Path(p / ".git").exists(): - return str(p) - # .git is not available in repos cloned via Cloud Build - # setup.py is always in the library's root, so use that instead - # https://github.com/googleapis/synthtool/issues/792 - if Path(p / "setup.py").exists(): - return str(p) - p = p.parent - raise Exception("Unable to detect repository root.") - - -GENERATED_READMES = sorted([x for x in Path(".").rglob("*.rst.in")]) - - -@nox.session -@nox.parametrize("path", GENERATED_READMES) -def readmegen(session: nox.sessions.Session, path: str) -> None: - """(Re-)generates the readme for a sample.""" - session.install("jinja2", "pyyaml") - dir_ = os.path.dirname(path) - - if os.path.exists(os.path.join(dir_, "requirements.txt")): - session.install("-r", os.path.join(dir_, "requirements.txt")) - - in_file = os.path.join(dir_, "README.rst.in") - session.run( - "python", _get_repo_root() + "/scripts/readme-gen/readme_gen.py", in_file - ) From 48278014f49dab393ec2c8b488dd04b1cf50a81f Mon Sep 17 00:00:00 2001 From: Sampath M Date: Wed, 16 Nov 2022 00:06:19 +0100 Subject: [PATCH 158/158] Update blunderbuss.yml and CODEOWNERS --- .github/CODEOWNERS | 3 ++- .github/blunderbuss.yml | 4 ++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 07346dc0c0c3..808f6b1db1d4 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -74,5 +74,6 @@ /talent/**/* @GoogleCloudPlatform/python-samples-reviewers /vision/**/* @GoogleCloudPlatform/python-samples-reviewers /workflows/**/* @GoogleCloudPlatform/python-samples-reviewers -/datacatalog/**/* @GoogleCloudPlatform/python-samples-reviewers +/datacatalog/**/* @GoogleCloudPlatform/python-samples-reviewers /kms/**/** @GoogleCloudPlatform/dee-infra @GoogleCloudPlatform/python-samples-reviewers +/dataproc/**/** @GoogleCloudPlatform/cloud-dpes @GoogleCloudPlatform/python-samples-reviewers \ No newline at end of file diff --git a/.github/blunderbuss.yml b/.github/blunderbuss.yml index ce3700dbaac7..c524caa7dca5 100644 --- a/.github/blunderbuss.yml +++ b/.github/blunderbuss.yml @@ -172,6 +172,10 @@ assign_prs_by: - 'api: cloudtasks' to: - GoogleCloudPlatform/infra-db-dpes +- labels: + - 'api: dataproc' + to: + - GoogleCloudPlatform/cloud-dpes assign_issues: - GoogleCloudPlatform/python-samples-owners