Pipeline Property Substitution #2052

damoodamoo · 2022-06-16T15:21:32Z

Closes #1679

What is being addressed

This PR adds support for a template designer to update a secondary resource from a primary, using properties from the primary in the upgrade.

To do this, a designer can use the syntax {{ resource.properties.prop_name }} , which will be translated at runtime to the real value. Using this method, Nexus + Gitea (in this PR) now update the firewall as a secondary resource rather than directly via their own terraform. Since we have queueing in place it should mean that Gitea + Nexus can be deployed in parallel more safely.

To support this the way outputs are returned from bundles into the cosmos object has been updated. Rather than converting every output to a string in the resource, we now allow a property bag to contain outputs of complex types.

In the UI, the notifications panel will show the pipeline steps such:

github-actions · 2022-06-16T15:22:18Z

Unit Test Results

288 tests +287 286 ✔️ +285 9s ⏱️ - 31m 24s
    1 suites ±    0     2 💤 +    2
    1 files ±    0     0 ❌ ±    0

Results for commit 0c60bf7. ± Comparison against base commit 16f4c60.

This pull request removes 1 and adds 288 tests. Note that renamed tests count towards both.

test_workspace_services ‑ test_create_guacamole_service_into_aad_workspace

tests_ma.test_api.test_errors.test_422_error ‑ test_frw_validation_error_format
tests_ma.test_api.test_errors.test_error ‑ test_frw_validation_error_format
tests_ma.test_api.test_routes.test_api_access.TestTemplateRoutesThatRequireAdminRights ‑ test_get_workspace_templates_requires_admin_rights
tests_ma.test_api.test_routes.test_api_access.TestTemplateRoutesThatRequireAdminRights ‑ test_post_user_resource_templates_requires_admin_rights
tests_ma.test_api.test_routes.test_api_access.TestTemplateRoutesThatRequireAdminRights ‑ test_post_workspace_service_templates_requires_admin_rights
tests_ma.test_api.test_routes.test_api_access.TestTemplateRoutesThatRequireAdminRights ‑ test_post_workspace_templates_requires_admin_rights
tests_ma.test_api.test_routes.test_api_access.TestUserResourcesOwnerOrResearcherRoutesAccess ‑ test_delete_user_resource_raises_403_if_user_is_not_workspace_owner_or_researcher
tests_ma.test_api.test_routes.test_api_access.TestUserResourcesOwnerOrResearcherRoutesAccess ‑ test_get_user_resource_raises_403_if_user_is_not_workspace_owner_or_researcher
tests_ma.test_api.test_routes.test_api_access.TestUserResourcesOwnerOrResearcherRoutesAccess ‑ test_get_user_resources_raises_403_if_user_is_not_researcher_or_owner_of_workspace
tests_ma.test_api.test_routes.test_api_access.TestUserResourcesOwnerOrResearcherRoutesAccess ‑ test_patch_user_resource_raises_403_if_user_is_not_workspace_owner_or_researcher
…

♻️ This comment has been updated with latest results.

damoodamoo · 2022-06-16T15:36:26Z

/test-extended

github-actions · 2022-06-16T15:36:46Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2510104310 (with refid 4e00469a)

(in response to this comment from @damoodamoo)

damoodamoo · 2022-06-16T16:14:30Z

/test-destroy-env

github-actions · 2022-06-16T16:15:14Z

Destroying branch test environment (RG: rg-trec7302e0e)... (run: https://github.com/microsoft/AzureTRE/actions/runs/2510319711)

github-actions · 2022-06-16T16:15:15Z

Destroying PR test environment (RG: rg-tre4e00469a)... (run: https://github.com/microsoft/AzureTRE/actions/runs/2510319711)

github-actions · 2022-06-16T16:15:16Z

Branch test environment destroy complete (RG: rg-trec7302e0e)

github-actions · 2022-06-16T16:31:26Z

PR test environment destroy complete (RG: rg-tre4e00469a)

damoodamoo · 2022-06-16T16:34:27Z

/test-extended

github-actions · 2022-06-16T16:34:42Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2510424861 (with refid 4e00469a)

(in response to this comment from @damoodamoo)

marrobi · 2022-06-16T21:32:27Z

Love it. Might be worth someone looking over the code - not sure I can give it the service it deserves.

Would be good to know what is/isn't supported - for example could I upgrade a workspace with a new property?

templates/shared_services/sonatype-nexus/template_schema.json

docs/tre-templates/pipeline-templates/pipeline-schema.md

api_app/tests_ma/conftest.py

api_app/tests_ma/test_service_bus/test_resource_request_sender.py

resource_processor/vmss_porter/runner.py

ross-p-smith · 2022-06-17T00:09:26Z

Would be good to know what is/isn't supported - for example could I upgrade a workspace with a new property?

The docs say only "shared-services" are supported at the moment.

Co-authored-by: Ross Smith <ross-p-smith@users.noreply.github.com>

tamirkamara

I didn't review the logic and would only comment:

If gitea/nexues don't need azcli anymore, can you please delete the dockerfile.tmpl as well (from the porter.yaml too)
Would merging still make it possible to use the "old" firewall script (as we do so elsewhere), or does it need to be upgrades in the other places immediately?

damoodamoo · 2022-06-20T07:39:02Z

@tamirkamara

If gitea/nexues don't need azcli anymore, can you please delete the dockerfile.tmpl as well (from the porter.yaml too)

I have only removed the firewall terraform resources in these bundles, nothing else has changed so no change to tooling needed.

Would merging still make it possible to use the "old" firewall script (as we do so elsewhere), or does it need to be upgrades in the other places immediately?

By 'old' firewall script do you mean the existing one in main? It's still possible for other resources to reference the firewall as before, and existing things should continue to work. It's a good point though, and what i'll do is rename the rules that gitea + nexus now create to avoid clashes with any existing implementations.

…ft/AzureTRE into damoo/1679-pipeline-substitution

ross-p-smith

Fantastico!! What a PR ❤️

damoodamoo · 2022-06-20T08:09:53Z

/test-destroy-env

github-actions · 2022-06-20T08:10:37Z

Destroying PR test environment (RG: rg-tre4e00469a)... (run: https://github.com/microsoft/AzureTRE/actions/runs/2527522640)

github-actions · 2022-06-20T08:10:38Z

Destroying branch test environment (RG: rg-trec7302e0e)... (run: https://github.com/microsoft/AzureTRE/actions/runs/2527522640)

github-actions · 2022-06-20T08:10:40Z

Branch test environment destroy complete (RG: rg-trec7302e0e)

github-actions · 2022-06-20T08:28:53Z

PR test environment destroy complete (RG: rg-tre4e00469a)

damoodamoo · 2022-06-20T08:30:08Z

/test-extended

github-actions · 2022-06-20T08:30:28Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/2527629989 (with refid 4e00469a)

(in response to this comment from @damoodamoo)

damoodamoo · 2022-06-20T10:32:57Z

/test-shared-services

github-actions · 2022-06-20T10:33:16Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/2528324609 (with refid 4e00469a)

(in response to this comment from @damoodamoo)

damoodamoo · 2022-06-20T11:03:20Z

/test-shared_services

github-actions · 2022-06-20T11:03:40Z

🤖 pr-bot 🤖

/test-shared_services is not recognised as a valid command.

You can use the following commands:
    /test - build, deploy and run smoke tests on a PR
    /test-extended - build, deploy and run smoke & extended tests on a PR
    /test-shared-services - test the deployment of shared services on a PR build
    /test-force-approve - force approval of the PR tests (i.e. skip the deployment checks)
    /test-destroy-env - delete the validation environment for a PR (e.g. to enable testing a deployment from a clean start after previous tests)
    /help - show this help

(in response to this comment from @damoodamoo)

damoodamoo added 8 commits June 14, 2022 21:24

initial substitution logic, without array manipulations

de08f3e

gitea templates updated for testing

1aea240

outputs as complex types rather than just strings

e280e2e

remove / replace wip

6f16531

array logic with tests

95c1994

refactored for clarity, moved substitutions into retry block

825575e

moved gitea to using a pipeline

2d1d493

updated docs

87b01b4

damoodamoo changed the title ~~Pipeline Substitution~~ Pipeline Property Substitution Jun 16, 2022

damoodamoo requested a review from marrobi June 16, 2022 15:22

damoodamoo added 2 commits June 16, 2022 16:23

api version bump

8b8b541

main merge

b3cc7d6

damoodamoo added 2 commits June 16, 2022 16:37

rp bump

3b779b1

gitea version

348028d

Merge branch 'main' into damoo/1679-pipeline-substitution

3e500c8

Merge branch 'main' into damoo/1679-pipeline-substitution

09b62d5