Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[metricsadvisor] live tests setup #14396

Merged
merged 4 commits into from
Oct 12, 2020
Merged

[metricsadvisor] live tests setup #14396

merged 4 commits into from
Oct 12, 2020

Conversation

kristapratico
Copy link
Member

@kristapratico kristapratico commented Oct 9, 2020 •
edited
Loading

Once checked into master, will run pipeline generator and make any adjustments necessary in another PR

@kristapratico kristapratico self-assigned this Oct 9, 2020
@kristapratico kristapratico marked this pull request as ready for review October 12, 2020 19:40
xiangyan99
xiangyan99 reviewed Oct 12, 2020
View reviewed changes
sdk/metricsadvisor/tests.yml
AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret)
AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
AZURE_METRICS_ADVISOR_ENDPOINT: $(metricsadvisor-test-service-endpoint)
AZURE_METRICS_ADVISOR_SUBSCRIPTION_KEY: $(metricsadvisor-test-subscription-key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we share these variables with other language?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all environment variables are shared across languages except API key

xiangyan99
xiangyan99 reviewed Oct 12, 2020
View reviewed changes
sdk/metricsadvisor/azure-ai-metricsadvisor/tests/async_tests/base_testcase_async.py Outdated
self.alert_id = self.get_settings_value("ALERT_ID")
self.anomaly_detection_configuration_id = "c0f2539f-b804-4ab9-a70f-0da0c89c76d8"
self.anomaly_alert_configuration_id = "ff3014a0-bbbb-41ec-a637-677e77b81299"
self.metric_id = "3d48ed3e-6e6e-4391-b78f-b00dfee1e6f5"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we hard code the id here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we don't have to add them to the test secrets keyvault.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if we modify the secrets that we saved in keyvault, we have to update them as well, right?

I mean these ids are coupled with other secrets.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's a good point. We were trying to minimize the number of things we were adding to the keyvault, but I think it probably makes more sense to allow ease of update here. I'll add them back as env vars.

@kristapratico kristapratico merged commit 5eced3f into master Oct 12, 2020
@kristapratico kristapratico deleted the ma-live-tests branch October 12, 2020 21:44
xiangyan99 added a commit that referenced this pull request Oct 14, 2020
@xiangyan99 @kristapratico
@azure-sdk @turalf @KieranBrantnerMagee @hectorhdzg @xiafu-msft @chlowell @chidozieononiwu @bradleydamato
FI from master (#14501)
548f0be 
* [metricsadvisor] live tests setup (#14396)

* add live test yml for metrics advisor

* some fixes

* hardcode test ids and rerecord a few tests

* add ids back as env variables

* [formrecognizer] initial business cards (#14026)

* Sync eng/common directory with azure-sdk-tools for PR 1081 (#14445)

Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#1081
 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/master/eng/common/README.md#workflow)

* Communication Administration: Add on-demand resource creation for live-tests  (#14366)

* Add communication service preparer

* Refactor dynamic resource creation testing code

* Remove setup method of base test class

* Change fake conn str to valid format

* Remove unused import

* Remove main test file

* Reduce RG expiry durtion

* Async test code change with resource preparer

* Add chaching for resource-preparer

* Move helper into shared

* Move preparer into shared

* Uncomment test code

* Remove CommunicationResourceGroupPreparer use common instead

* Add base testcase for sync pnm

* move phone_number_helper to phone_number folder

* Add base pnm async testcase

* Add __init__ for test folders

* Fix identity tests to refer to shared test folder

* Refactor base async phonenumber test class

* Make decorator consisitent across all async tests utm

* Replace fake resource value

* Fix type in test file names

* Remove commented cache setting code

* Add livetest recording files

* Refresh recording files

* Update recording files

* Reorganize the test folder structure

* Add mgmt pacakge to the dev reqs

Co-authored-by: tural farhadov <tufarhad@microsoft.com>

* [ServiceBus] Small stress test and sample touchups (#14304)

* Emplace sample generated for customer issue on the side for future reference. (generate sas token and authenticate)
* Adjust stress test that was using push receive instead of pull.
* Add async version of non-session to session send failure test, make both cached to improve perf.

* Codegen for Azure Monitor (#14057)

* Codegen fro Azure Monitor

* codeowner

* commit 2

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md

* doc setings

* Update eng/.docsettings.yml

* Update eng/.docsettings.yml

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md

* Update eng/.docsettings.yml

* version

* oops

* comments

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/LICENSE.txt

Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>

Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>

* [Storage][FileShare]Regenerate for delete leased snapshot (#14469)

* add repr to models (#14475)

* remove validations (#14476)

* Ma remove readonly assignments (#14478)

* Ensure azure-identity readme has exactly one H1 (#14480)

The docs publishing system disallows multiple H1 headings in a document.

* Sync eng/common directory with azure-sdk-tools for PR 1088 (#14482)

* Add pipeline configuration for cleaning up upstream branches

* Fix powershell naming

* Switch to use github API

* Use github API to delete branches

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>

* enable live tests (#14467)

* Removed asynctestcase.py (#14444)

* testcase.py now uses AzureTestCase (#14443)

Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: turalf <tural.ferhadov@gmail.com>
Co-authored-by: tural farhadov <tufarhad@microsoft.com>
Co-authored-by: KieranBrantnerMagee <kibrantn@microsoft.com>
Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>
Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>
Co-authored-by: Xiaoxi Fu <49707495+xiafu-msft@users.noreply.github.com>
Co-authored-by: Charles Lowell <chlowe@microsoft.com>
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Bradley D'Amato <53579156+bradleydamato@users.noreply.github.com>
xiangyan99 added a commit that referenced this pull request Nov 7, 2020
@xiangyan99 @kristapratico
@azure-sdk @turalf @KieranBrantnerMagee @hectorhdzg @xiafu-msft @chlowell @chidozieononiwu @bradleydamato
Config the default blocksize to 32k for Python 3.7+ (#14442)
c17c635 
* customize httpadapter to honor kwargs in send method

* update

* update

* update

* update

* update

* update

* update

* FI from master (#14501)

* [metricsadvisor] live tests setup (#14396)

* add live test yml for metrics advisor

* some fixes

* hardcode test ids and rerecord a few tests

* add ids back as env variables

* [formrecognizer] initial business cards (#14026)

* Sync eng/common directory with azure-sdk-tools for PR 1081 (#14445)

Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#1081
 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/master/eng/common/README.md#workflow)

* Communication Administration: Add on-demand resource creation for live-tests  (#14366)

* Add communication service preparer

* Refactor dynamic resource creation testing code

* Remove setup method of base test class

* Change fake conn str to valid format

* Remove unused import

* Remove main test file

* Reduce RG expiry durtion

* Async test code change with resource preparer

* Add chaching for resource-preparer

* Move helper into shared

* Move preparer into shared

* Uncomment test code

* Remove CommunicationResourceGroupPreparer use common instead

* Add base testcase for sync pnm

* move phone_number_helper to phone_number folder

* Add base pnm async testcase

* Add __init__ for test folders

* Fix identity tests to refer to shared test folder

* Refactor base async phonenumber test class

* Make decorator consisitent across all async tests utm

* Replace fake resource value

* Fix type in test file names

* Remove commented cache setting code

* Add livetest recording files

* Refresh recording files

* Update recording files

* Reorganize the test folder structure

* Add mgmt pacakge to the dev reqs

Co-authored-by: tural farhadov <tufarhad@microsoft.com>

* [ServiceBus] Small stress test and sample touchups (#14304)

* Emplace sample generated for customer issue on the side for future reference. (generate sas token and authenticate)
* Adjust stress test that was using push receive instead of pull.
* Add async version of non-session to session send failure test, make both cached to improve perf.

* Codegen for Azure Monitor (#14057)

* Codegen fro Azure Monitor

* codeowner

* commit 2

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md

* doc setings

* Update eng/.docsettings.yml

* Update eng/.docsettings.yml

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md

* Update eng/.docsettings.yml

* version

* oops

* comments

* Update sdk/monitor/opentelemetry-exporter-azuremonitor/LICENSE.txt

Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>

Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>

* [Storage][FileShare]Regenerate for delete leased snapshot (#14469)

* add repr to models (#14475)

* remove validations (#14476)

* Ma remove readonly assignments (#14478)

* Ensure azure-identity readme has exactly one H1 (#14480)

The docs publishing system disallows multiple H1 headings in a document.

* Sync eng/common directory with azure-sdk-tools for PR 1088 (#14482)

* Add pipeline configuration for cleaning up upstream branches

* Fix powershell naming

* Switch to use github API

* Use github API to delete branches

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>

* enable live tests (#14467)

* Removed asynctestcase.py (#14444)

* testcase.py now uses AzureTestCase (#14443)

Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: turalf <tural.ferhadov@gmail.com>
Co-authored-by: tural farhadov <tufarhad@microsoft.com>
Co-authored-by: KieranBrantnerMagee <kibrantn@microsoft.com>
Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>
Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>
Co-authored-by: Xiaoxi Fu <49707495+xiafu-msft@users.noreply.github.com>
Co-authored-by: Charles Lowell <chlowe@microsoft.com>
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Bradley D'Amato <53579156+bradleydamato@users.noreply.github.com>

* update

* update

* update

* update

* update

* update

* update

* update

Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: turalf <tural.ferhadov@gmail.com>
Co-authored-by: tural farhadov <tufarhad@microsoft.com>
Co-authored-by: KieranBrantnerMagee <kibrantn@microsoft.com>
Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>
Co-authored-by: Hector Hernandez <39923391+hectorhdzg@users.noreply.github.com>
Co-authored-by: Xiaoxi Fu <49707495+xiafu-msft@users.noreply.github.com>
Co-authored-by: Charles Lowell <chlowe@microsoft.com>
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Bradley D'Amato <53579156+bradleydamato@users.noreply.github.com>
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-python that referenced this pull request May 14, 2021
CodeGen from PR 14396 in Azure/azure-rest-api-specs
ab39f93 
add error response flag (Azure#14396)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiangyan99 xiangyan99 xiangyan99 approved these changes

@danieljurek danieljurek Awaiting requested review from danieljurek

Assignees

@kristapratico kristapratico

Labels
Cognitive - Metrics Advisor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kristapratico @xiangyan99