Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish Dataproc Serverless Batch link after it starts if batch_id was provided #41153

Merged
merged 1 commit into from
Oct 4, 2024

Conversation

rafalh
Copy link
Contributor

@rafalh rafalh commented Jul 31, 2024

Currently the link to Dataproc Serverless Batch is only available after its successful execution.
To improve user experience publish a link right after the batch starts execution if batch_id was provided. It makes it easier to observe the progress and find batch logs when it fails.
The only drawback is that if batch creation fails before it starts execution, the link will point to nowhere, but I think it is acceptable considering the gains.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jul 31, 2024
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 15, 2024
@rafalh
Copy link
Contributor Author

rafalh commented Sep 15, 2024

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

Please unstale. Hopefully in thousand years someone will look into it...

@github-actions github-actions bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 16, 2024
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes we might oversee PRs - in that case, you could always ping us at the #new-contributors channel on Slack.
I had some minor comments, and I'd be happy if you could add/update a relevant test.
Thank you!

airflow/providers/google/cloud/operators/dataproc.py Outdated Show resolved Hide resolved
airflow/providers/google/cloud/operators/dataproc.py Outdated Show resolved Hide resolved
@eladkal
Copy link
Contributor

eladkal commented Oct 4, 2024

@rafalh can you address comments and resolve conflicts?

@rafalh
Copy link
Contributor Author

rafalh commented Oct 4, 2024

Sorry for the delay. I resolved the conflicts and I think it is ready for re-review.

@eladkal eladkal requested a review from shahar1 October 4, 2024 17:07
@shahar1 shahar1 merged commit 8a255f3 into apache:main Oct 4, 2024
56 checks passed
@rafalh rafalh deleted the dataproc-batch-link branch October 8, 2024 12:17
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants