Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airlock processor handles request Submission #1978

Merged
merged 12 commits into from
Jun 14, 2022
Merged

Airlock processor handles request Submission #1978

merged 12 commits into from
Jun 14, 2022

Conversation

eladiw
Copy link
Contributor

@eladiw eladiw commented Jun 7, 2022 •
edited
Loading

Fixes #1836

What is being addressed

Adding support for the StatusChangedEvent in the Airlock processor (for submission - import and export)
Consolidated the blob created queues into a single SB topic
Wired all TF scripts
Added support for the blobCreatedEvent in the AL processor

How is this addressed

  • Added code to extract the msg from the status changed queue
  • If data copy is required, extract the source and dest params, and issue a copy command
  • added unit tests

How to test:

  1. Create container in the import external storage. Give it a name: e.g. 123
  2. Put a single file
  3. Place this message in the status_changed queue

{ "data": {
"request_id" : "123",
"type":"import",
"status":"submitted" ,
"workspace_id":"124"
}
}

Check the in progress storage account , it should contain a new container with the same file.
Check the step result queue, you should find a step result event which states the request is now in-progress (transitioned from submitted)

@eladiw eladiw changed the title Feature/airlock 1836 Airlock processor handles request Submission, approval, rejection and block Jun 7, 2022
@eladiw eladiw force-pushed the feature/airlock_1836 branch 2 times, most recently from c4c37e3 to 53fece3 Compare June 7, 2022 19:31
@eladiw eladiw marked this pull request as ready for review June 8, 2022 20:28
@eladiw
Copy link
Contributor Author

eladiw commented Jun 9, 2022

/test

@github-actions
Copy link

github-actions bot commented Jun 9, 2022

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/2466146573 (with refid 3de19bb9)

(in response to this comment from @eladiw)

damoodamoo
damoodamoo reviewed Jun 9, 2022
View reviewed changes
Copy link
Member

@damoodamoo damoodamoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing lots of airlock context, but done my best to review.

My main concern would be using any blob triggers to determine stage transitions - as this will pretty much pin us to one file and require a lot of refactoring if we wanted to support multiple in future.

airlock_processor/BlobCreatedTrigger/__init__.py Show resolved Hide resolved
airlock_processor/BlobCreatedTrigger/function.json Show resolved Hide resolved
airlock_processor/StatusChangedQueueTrigger/__init__.py Outdated Show resolved Hide resolved
airlock_processor/shared_code/blob_operations.py Outdated Show resolved Hide resolved
airlock_processor/shared_code/blob_operations.py Outdated Show resolved Hide resolved
airlock_processor/shared_code/blob_operations.py Outdated Show resolved Hide resolved
@eladiw eladiw force-pushed the feature/airlock_1836 branch 4 times, most recently from 7361316 to d4175cf Compare June 12, 2022 18:16
@eladiw eladiw changed the title Airlock processor handles request Submission, approval, rejection and block Airlock processor handles request Submission Jun 13, 2022
@eladiw
Copy link
Contributor Author

eladiw commented Jun 13, 2022

@damoodamoo @tamirkamara fixed all comments

tamirkamara
tamirkamara approved these changes Jun 14, 2022
View reviewed changes
Copy link
Collaborator

@tamirkamara tamirkamara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a couple of small comments. Up to you...

@eladiw
Handling submitted/approved/rejected statusChangedEvent
10e80f3 
terraform support for the az function
@eladiw
Consolidate blob created queues to single SB topic + two subscriptions
ba871e1 
Added support for blob created event handling in the airlock processor
airlock processor uses managed identity to get keys (+ added role assignment to SAs)
@eladiw
minor refactor
e4c8766
@eladiw
fix according to comments
b848823
@eladiw
linting
30c2a3e
@eladiw
refactor names
91bb8ea
@eladiw
fixing according to comments
4d97fed
@eladiw
fix according to comments
8257cd2
@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test-extended

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2492978719 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/2493332545 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/destroy-test-env

@github-actions
Copy link

🤖 pr-bot 🤖

/destroy-test-env is not recognised as a valid command.

You can use the following commands:
    /test - build, deploy and run smoke tests on a PR
    /test-extended - build, deploy and run smoke & extended tests on a PR
    /test-shared-services - test the deployment of shared services on a PR build
    /test-force-approve - force approval of the PR tests (i.e. skip the deployment checks)
    /test-destroy-env - delete the validation environment for a PR (e.g. to enable testing a deployment from a clean start after previous tests)
    /help - show this help

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test-destroy-env

@github-actions
Copy link

Destroying PR test environment (RG: rg-tre3de19bb9)... (run: https://github.com/microsoft/AzureTRE/actions/runs/2493701571)

@github-actions
Copy link

PR test environment destroy complete (RG: rg-tre3de19bb9)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test-extended

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2493857357 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test-extended

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2494018111 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/2494112493 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw
Copy link
Contributor Author

eladiw commented Jun 14, 2022

/test-extended

@github-actions
Copy link

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2494217775 (with refid 3de19bb9)

(in response to this comment from @eladiw)

@eladiw eladiw merged commit 11a3962 into microsoft:main Jun 14, 2022
@eladiw eladiw deleted the feature/airlock_1836 branch June 14, 2022 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@damoodamoo damoodamoo damoodamoo left review comments

@tamirkamara tamirkamara tamirkamara approved these changes

@anatbal anatbal Awaiting requested review from anatbal

@LizaShak LizaShak Awaiting requested review from LizaShak

Assignees
No one assigned
Labels
airlock
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Airlock processor handles Request submission
3 participants
@eladiw @tamirkamara @damoodamoo