Add extra operator links for EMR Serverless #34225

dacort · 2023-09-08T22:01:05Z

Includes Dashboard UI, Spark stdout logs, S3 and CloudWatch consoles
Only shows links relevant to the job
Disables Dashboard UI and Spark stdout logs by default as the generated link is a pre-signed url
Passes in aws_conn_id from the operator hook so the proper credentials are used

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

- Includes Dashboard UI, S3 and CloudWatch consoles - Only shows links relevant to the job

airflow/providers/amazon/aws/links/emr.py

eladkal · 2023-10-24T05:42:07Z

@dacort are you still working on it?

dacort · 2023-10-24T14:50:45Z

@eladkal Yes, let me check out the failing tests.

dacort · 2023-10-24T15:33:31Z

@eladkal Ah yes, that's right, I was trying to figure out a better way to pass the operator in, but I think I'll just pass in the aws_conn_id as a string instead.

airflow/providers/amazon/aws/links/emr.py

airflow/providers/amazon/aws/operators/emr.py

airflow/providers/amazon/aws/links/emr.py

Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>

o-nikolas

Thanks for this Damon, looking good! Left some comments

docs/apache-airflow-providers-amazon/operators/emr/emr_serverless.rst

airflow/providers/amazon/aws/links/emr.py

tests/providers/amazon/aws/operators/test_emr_serverless.py

github-actions · 2024-01-07T00:13:16Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

o-nikolas · 2024-01-08T22:18:58Z

Hey @dacort, this one looks like it was close but is getting marked as stale. Do you still have time to finish it off?

dacort · 2024-01-08T22:20:02Z

Yes, let me try to resolve existing conflicts and get it over the finish line. :)

dacort · 2024-01-17T17:18:31Z

How does this work in deferrable mode?

persist_links is called before the self.defer call is triggered for EmrServerlessStartJobTrigger. I'm not too familiar with deferrable mode - would this allow the necessary link context to be populated before deferring?

~~I'll test this locally as well.~~ Tested locally and seems to work fine - the links are populated and work while the job is running.

syedahsn

Awesome work! LGTM

eladkal · 2024-01-19T21:42:30Z

@dacort @o-nikolas is anything else needed or can we merge this PR?

o-nikolas · 2024-01-22T17:20:09Z

@dacort @o-nikolas is anything else needed or can we merge this PR?

@eladkal, the last remaining item is adding some unit testing for this code (thread here)

dacort · 2024-01-22T17:39:59Z

adding some unit testing

ack - will see if I can get that pushed up today!

dacort · 2024-01-22T19:17:45Z

@eladkal @o-nikolas Added some additional unit tests to cover the various cases and merge in latest main.

Several of the DB tests are failing, but seem unrelated to my changes.

o-nikolas · 2024-01-22T21:38:14Z

Several of the DB tests are failing, but seem unrelated to my changes.

Main is broken right now, you'll at least need to wait for this PR to be merged and then rebase your branch.

dacort · 2024-01-23T01:39:38Z

Sweet, rebased and all checks have passed!

o-nikolas · 2024-01-24T18:30:11Z

Only one conversation left open that needs resolution and then I think we're ready to merge!

eladkal · 2024-02-10T09:27:16Z

@dacort can you rebase and resolve conflicts? I'll merge after

eladkal · 2024-02-12T07:30:37Z

@dacort static checks are failing due to adding D401 validation

dacort · 2024-02-12T14:49:55Z

@eladkal Ah, thank you, had missed that failure reason. Will get that addressed today!

airflow/providers/amazon/aws/links/emr.py

…ests

* Add extra operator links for EMR Serverless - Includes Dashboard UI, S3 and CloudWatch consoles - Only shows links relevant to the job * Fix imports and add context mock to tests * Move TYPE_CHECK * Remove unused variables * Pass in connection ID string instead of operator * Use mock.MagicMock * Disable application UI logs by default * Update doc lints * Update airflow/providers/amazon/aws/links/emr.py Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is> * Support dynamic task mapping * Lint/static check fixes * Update review comments * Configure get_dashboard call for EMR Serverless to only retry once * Whitespace * Add unit tests for EMRS link generation * Address D401 check * Refactor get_serverless_dashboard_url into its own method, add link tests * Fix lints --------- Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>

Add extra operator links for EMR Serverless

59cd614

- Includes Dashboard UI, S3 and CloudWatch consoles - Only shows links relevant to the job

boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Sep 8, 2023

Fix imports and add context mock to tests

93170c5

Taragolis reviewed Sep 8, 2023

View reviewed changes

airflow/providers/amazon/aws/links/emr.py Outdated Show resolved Hide resolved

airflow/providers/amazon/aws/links/emr.py Outdated Show resolved Hide resolved

Move TYPE_CHECK

906c1de

dacort added 7 commits October 24, 2023 12:46

Remove unused variables

18a6b0a

Pass in connection ID string instead of operator

e1d8970

Merge branch 'apache:main' into feature/emr-serverless-extra-links

8487f85

Use mock.MagicMock

b56a0fc

Disable application UI logs by default

987cce0

Update doc lints

76e19ac

Merge branch 'main' into feature/emr-serverless-extra-links

590f65b

dacort marked this pull request as ready for review October 25, 2023 05:00

dacort requested review from eladkal and o-nikolas as code owners October 25, 2023 05:00

Taragolis reviewed Oct 25, 2023

View reviewed changes

dacort and others added 3 commits October 30, 2023 11:47

Update airflow/providers/amazon/aws/links/emr.py

625500c

Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>

Support dynamic task mapping

7a73d32

Lint/static check fixes

e4b7efb

o-nikolas reviewed Oct 31, 2023

View reviewed changes

Update review comments

eb62bf1

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 7, 2024

Merge branch 'main' into feature/emr-serverless-extra-links

35d6e47

syedahsn approved these changes Jan 18, 2024

View reviewed changes

eladkal approved these changes Jan 19, 2024

View reviewed changes

dacort added 2 commits January 22, 2024 11:02

Add unit tests for EMRS link generation

b9acdf6

Merge branch 'apache:main' into feature/emr-serverless-extra-links

0763152

o-nikolas approved these changes Jan 22, 2024

View reviewed changes

dacort added 2 commits January 22, 2024 14:22

Merge branch 'apache:main' into feature/emr-serverless-extra-links

5b9edba

Merge branch 'apache:main' into feature/emr-serverless-extra-links

4443bbb

Merge branch 'apache:main' into feature/emr-serverless-extra-links

639f592

Taragolis reviewed Feb 12, 2024

View reviewed changes

airflow/providers/amazon/aws/links/emr.py Show resolved Hide resolved

Address D401 check

91fb733

Taragolis reviewed Feb 12, 2024

View reviewed changes

airflow/providers/amazon/aws/links/emr.py Outdated Show resolved Hide resolved

airflow/providers/amazon/aws/links/emr.py Outdated Show resolved Hide resolved

dacort added 2 commits February 12, 2024 22:34

Refactor get_serverless_dashboard_url into its own method, add link t…

fd3f378

…ests

Fix lints

12842db

Taragolis approved these changes Feb 13, 2024

View reviewed changes

Taragolis merged commit b5b452b into apache:main Feb 13, 2024
58 checks passed

dacort deleted the feature/emr-serverless-extra-links branch February 13, 2024 13:55

eladkal mentioned this pull request Feb 17, 2024

Status of testing Providers that were prepared on February 17, 2024 #37504

Closed

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add extra operator links for EMR Serverless #34225

Add extra operator links for EMR Serverless #34225

dacort commented Sep 8, 2023 •

edited

Loading

eladkal commented Oct 24, 2023

dacort commented Oct 24, 2023

dacort commented Oct 24, 2023

o-nikolas left a comment

github-actions bot commented Jan 7, 2024

o-nikolas commented Jan 8, 2024

dacort commented Jan 8, 2024

dacort commented Jan 17, 2024 •

edited

Loading

syedahsn left a comment

eladkal commented Jan 19, 2024

o-nikolas commented Jan 22, 2024

dacort commented Jan 22, 2024

dacort commented Jan 22, 2024 •

edited

Loading

o-nikolas commented Jan 22, 2024

dacort commented Jan 23, 2024

o-nikolas commented Jan 24, 2024

eladkal commented Feb 10, 2024

eladkal commented Feb 12, 2024

dacort commented Feb 12, 2024

Add extra operator links for EMR Serverless #34225

Add extra operator links for EMR Serverless #34225

Conversation

dacort commented Sep 8, 2023 • edited Loading

eladkal commented Oct 24, 2023

dacort commented Oct 24, 2023

dacort commented Oct 24, 2023

o-nikolas left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 7, 2024

o-nikolas commented Jan 8, 2024

dacort commented Jan 8, 2024

dacort commented Jan 17, 2024 • edited Loading

syedahsn left a comment

Choose a reason for hiding this comment

eladkal commented Jan 19, 2024

o-nikolas commented Jan 22, 2024

dacort commented Jan 22, 2024

dacort commented Jan 22, 2024 • edited Loading

o-nikolas commented Jan 22, 2024

dacort commented Jan 23, 2024

o-nikolas commented Jan 24, 2024

eladkal commented Feb 10, 2024

eladkal commented Feb 12, 2024

dacort commented Feb 12, 2024

dacort commented Sep 8, 2023 •

edited

Loading

dacort commented Jan 17, 2024 •

edited

Loading

dacort commented Jan 22, 2024 •

edited

Loading