[Key Vault] Test keys library against a shared vault

[mccoyp]

Resolves #15435.

This drops KV's existing preparers, and instead uses a PowerShellPreparer to fetch an existing vault URL from an environment variable.
test-resources.json and test-resources-post.ps1 are added to make resource group, vault, and managed HSM creation as easy as running a script.
test-resources-cleanup cleans up test resources from a vault, which are identified by a "livekvtest" prefix -- this is automatically added by get_resource_name().

[check-enforcer]

This pull request is protected by Check Enforcer.

What is Check Enforcer?

Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass.

Why am I getting this message?

You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged.

What should I do now?

If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows:
/check-enforcer evaluate
Typically evaulation only takes a few seconds. If you know that your pull request is not covered by a pipeline and this is expected you can override Check Enforcer using the following command:
/check-enforcer override
Note that using the override command triggers alerts so that follow-up investigations can occur (PRs still need to be approved as normal).

What if I am onboarding a new service?

Often, new services do not have validation pipelines associated with them, in order to bootstrap pipelines for a new service, you can issue the following command as a pull request comment:
/azp run prepare-pipelines
This will run a pipeline that analyzes the source tree and creates the pipelines necessary to build and validate your pull request. Once the pipeline has been created you can trigger the pipeline using the following comment:
/azp run python - [service] - ci

[mccoyp]

/azp run python - keyvault - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mccoyp]

/check-enforcer evaluate

[mccoyp]

/check-enforcer override

[mccoyp]

/azp run python - keyvault - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[chlowell]

Were the recordings made with the code as it is in the latest revision of this PR (it looks like they were recorded by test cases sharing a single client instance)?

[mccoyp]

These recordings were all made after other changes were made, so I'm not sure why that would be the case

[chlowell]

It's because the ChallengeAuthenticationPolicy uses a process-wide cache of challenge information. In a live run, the first test case caches challenge information and subsequent test cases--all using the same vault, whose URL is the cache key--use that cached information rather than send a request to get it from Key Vault. Therefore only one recording contains the initial empty request, which recording being determined by order of execution. This implies that when you play the tests back in a different order than they were recorded, you can expect failure because the challenge cache will be empty until the test case with the recorded challenge runs. Each test that runs before the challenge is played back will send a request with no recorded response: vcr should raise.

But in fact this doesn't happen; the tests pass because:

1. they don't configure vcr to match request bodies (because the bodies contain random data)
2. the initial, empty request the client sends to elicit a challenge from Key Vault therefore matches the subsequent, authorized request so far as vcr is concerned
3. when it gets the empty request, vcr replays the response to the authorized request
4. that response doesn't have a WWW-Authenticate header, so the auth policy fails to parse a challenge from it
5. the policy's failure mode in this case is to return the response (because it doesn't know what else to do with it)
6. the returned response isn't an error, so the pipeline doesn't complain
7. actually the response is exactly what the test asked for when it called client.do_whatever()
8. success!

So these recordings create an error condition that probably should, but coincidentally does not, cause test failures 😆

The easiest solution is to clear the cache between tests:

azure-sdk-for-python/sdk/keyvault/azure-keyvault-keys/tests/test_challenge_auth.py

Line 33 in 0763539

HttpChallengeCache.clear()

...something the preparer could handle?

[mccoyp]

I can ask Sean about adding it to the PowerShellPreparer if this is something that can be generalized to other libraries (if it's not, we could subclass the preparer and do this from there), but I've verified locally that this could also be done in a tearDown method of each test class. The former would be cleaner though 🤔

[chlowell]

It's a Key Vault only thing. I thought tearDown was called after all the test cases have run?

[mccoyp]

One would think that would be the case, but it's called after each test method finishes up -- kind of convenient, kind of confusing