Merge branch 'master' into blob-exists-feature

.github/CODEOWNERS

@@ -25,11 +25,11 @@
 # PRLabel: %Storage
 /sdk/storage/                                                        @amishra-dev @zezha-msft @annatisch @rakshith91 @xiafu-msft @tasherif-msft @kasobol-msft
 
-/sdk/applicationinsights/                                            @alexeldeib      
+/sdk/applicationinsights/                                            @alexeldeib
 
 # PRLabel: %Batch
 /sdk/batch/                                                          @bgklein @xingwu1
-/sdk/cognitiveservices/azure-cognitiveservices-vision-customvision/  @areddish          
+/sdk/cognitiveservices/azure-cognitiveservices-vision-customvision/  @areddish
 
 # PRLabel: %KeyVault
 /sdk/keyvault/                                                       @schaabs @chlowell @iscai-msft
@@ -52,7 +52,7 @@
 
 # PRLabel: %Data Factory
 /sdk/datafactory/                                                    @hvermis
-/sdk/datalake/                                                       @ro-joowan                                                     
+/sdk/datalake/                                                       @ro-joowan
 /sdk/datadatamigration/                                              @vchske
 
 # PRLabel: %Event Grid
@@ -88,6 +88,9 @@
 # PRLabel: %Cognitive - Form Recognizer
 /sdk/formrecognizer/                                                 @kristapratico @iscai-msft @rakshith91
 
+# PRLabel: %Tables
+/sdk/tables/                                                         @seankane-msft
+
 # Smoke Tests
 /common/smoketest/                                                   @lmazuel @chlowell @annatisch @rakshith91 @shurd @southpolesteve
 

CONTRIBUTING.md

@@ -111,6 +111,10 @@ Mypy install and run.
 **Example: Invoke tox, breaking into the debugger on failure**
 `tox -e whl -c ../../../eng/tox/tox.ini -- --pdb`
 
+### More Reading
+
+We maintain an [additional document](doc/eng_sys_checks.md) that has a ton of detail as to what is actually _happening_ in these executions.
+
 ### Dev Feed
 Daily dev build version of Azure sdk packages for python are available and are uploaded to Azure devops feed daily. We have also created a tox environment to test a package against dev built version of dependent packages. Below is the link to Azure devops feed.
 [`https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python`](https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python)

README.md

@@ -2,7 +2,7 @@
 
 [![Packages](https://img.shields.io/badge/packages-latest-blue.svg)](https://azure.github.io/azure-sdk/releases/latest/python.html) [![Dependencies](https://img.shields.io/badge/dependency-report-blue.svg)](https://azuresdkartifacts.blob.core.windows.net/azure-sdk-for-python/dependencies/dependencies.html) [![DepGraph](https://img.shields.io/badge/dependency-graph-blue.svg)](https://azuresdkartifacts.blob.core.windows.net/azure-sdk-for-python/dependencies/InterdependencyGraph.html) [![Python](https://img.shields.io/pypi/pyversions/azure-core.svg?maxAge=2592000)](https://pypi.python.org/pypi/azure/) [![Build Status](https://dev.azure.com/azure-sdk/public/_apis/build/status/python/python%20-%20core%20-%20ci?branchName=master)](https://dev.azure.com/azure-sdk/public/_build/latest?definitionId=458&branchName=master)
 
-This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our [public developer docs](https://docs.microsoft.com/en-us/python/azure/) or our versioned [developer docs](https://azure.github.io/azure-sdk-for-python).
+This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our [public developer docs](https://docs.microsoft.com/python/azure/) or our versioned [developer docs](https://azure.github.io/azure-sdk-for-python).
 
 ## Getting started
 

SECURITY.md

@@ -4,15 +4,15 @@
 
 Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
 
-If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.
 
 ## Reporting Security Issues
 
 **Please do not report security vulnerabilities through public GitHub issues.**
 
 Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).
 
-If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).
+If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/msrc/pgp-key-msrc).
 
 You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 
 
@@ -36,6 +36,6 @@ We prefer all communications to be in English.
 
 ## Policy
 
-Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).
+Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/msrc/cvd).
 
 <!-- END MICROSOFT SECURITY.MD BLOCK -->

common/smoketest/key_vault_base.py

@@ -1,17 +1,22 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
 import os
-from azure.identity import DefaultAzureCredential, KnownAuthorities
+from azure.identity import AzureAuthorityHosts, DefaultAzureCredential
+
 
 class KeyVaultBase:
     credential_type = DefaultAzureCredential
     host_alias_map = {
-        'AzureChinaCloud': KnownAuthorities.AZURE_CHINA,
-        'AzureGermanCloud': KnownAuthorities.AZURE_GERMANY,
-        'AzureUSGovernment': KnownAuthorities.AZURE_GOVERNMENT,
-        'AzureCloud': KnownAuthorities.AZURE_PUBLIC_CLOUD,
+        "AzureChinaCloud": (AzureAuthorityHosts.AZURE_CHINA, "2016-10-01"),
+        "AzureGermanCloud": (AzureAuthorityHosts.AZURE_GERMANY, "2016-10-01"),
+        "AzureUSGovernment": (AzureAuthorityHosts.AZURE_GOVERNMENT, "2016-10-01"),
+        "AzureCloud": (AzureAuthorityHosts.AZURE_PUBLIC_CLOUD, "7.1"),
     }
 
-    # Instantiate a default credential based on the credential_type
-    def get_default_credential(self, authority_host_alias=None):
-        alias = authority_host_alias or os.environ.get("AZURE_CLOUD")
-        authority_host = self.host_alias_map.get(alias, KnownAuthorities.AZURE_PUBLIC_CLOUD)
-        return self.credential_type(authority=authority_host)
+    def get_client_args(self, authority_host_alias=None):
+        alias = authority_host_alias or os.environ.get("AZURE_CLOUD", "AzureCloud")
+        authority_host, api_version = self.host_alias_map[alias]
+        credential = self.credential_type(authority=authority_host)
+        return {"api_version": api_version, "credential": credential, "vault_url": os.environ["AZURE_PROJECT_URL"]}

common/smoketest/key_vault_base_async.py

@@ -1,5 +1,10 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
 from key_vault_base import KeyVaultBase
 from azure.identity.aio import DefaultAzureCredential
 
+
 class KeyVaultBaseAsync(KeyVaultBase):
     credential_type = DefaultAzureCredential

common/smoketest/key_vault_certificates.py

@@ -2,19 +2,15 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.certificates import CertificateClient, CertificatePolicy
 from key_vault_base import KeyVaultBase
 
+
 class KeyVaultCertificates(KeyVaultBase):
     def __init__(self):
-
-        credential = self.get_default_credential()
-        self.certificate_client = CertificateClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
-
+        args = self.get_client_args()
+        self.certificate_client = CertificateClient(**args)
         self.certificate_name = "cert-name-" + uuid.uuid1().hex
 
     def create_certificate(self):

common/smoketest/key_vault_certificates_async.py

@@ -2,19 +2,16 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.certificates import CertificatePolicy
 from azure.keyvault.certificates.aio import CertificateClient
 from key_vault_base_async import KeyVaultBaseAsync
 
+
 class KeyVaultCertificates(KeyVaultBaseAsync):
     def __init__(self):
-        credential = self.get_default_credential()
-        self.certificate_client = CertificateClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
-
+        args = self.get_client_args()
+        self.certificate_client = CertificateClient(**args)
         self.certificate_name = "cert-name-" + uuid.uuid1().hex
 
     async def create_certificate(self):

common/smoketest/key_vault_keys.py

@@ -2,19 +2,15 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.keys import KeyClient
 from key_vault_base import KeyVaultBase
 
 
 class KeyVaultKeys(KeyVaultBase):
     def __init__(self):
-        credential = self.get_default_credential()
-        self.key_client = KeyClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
-
+        args = self.get_client_args()
+        self.key_client = KeyClient(**args)
         self.key_name = "key-name-" + uuid.uuid1().hex
 
     def create_rsa_key(self):

common/smoketest/key_vault_keys_async.py

@@ -2,20 +2,15 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.keys.aio import KeyClient
 from key_vault_base_async import KeyVaultBaseAsync
 
 
 class KeyVaultKeys(KeyVaultBaseAsync):
     def __init__(self):
-
-        credential = self.get_default_credential()
-        self.key_client = KeyClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
-
+        args = self.get_client_args()
+        self.key_client = KeyClient(**args)
         self.key_name = "key-name-" + uuid.uuid1().hex
 
     async def create_rsa_key(self):

common/smoketest/key_vault_secrets.py

@@ -2,18 +2,15 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.secrets import SecretClient
 from key_vault_base import KeyVaultBase
 
+
 class KeyVaultSecrets(KeyVaultBase):
     def __init__(self):
-        credential = self.get_default_credential()
-        self.secret_client = SecretClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
-
+        args = self.get_client_args()
+        self.secret_client = SecretClient(**args)
         self.secret_name = "secret-name-" + uuid.uuid1().hex
         self.secret_Value = "secret-value"
 

common/smoketest/key_vault_secrets_async.py

@@ -2,17 +2,15 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
-import os
 import uuid
 from azure.keyvault.secrets.aio import SecretClient
 from key_vault_base_async import KeyVaultBaseAsync
 
+
 class KeyVaultSecrets(KeyVaultBaseAsync):
     def __init__(self):
-        credential = self.get_default_credential()
-        self.secret_client = SecretClient(
-            vault_url=os.environ["AZURE_PROJECT_URL"], credential=credential
-        )
+        args = self.get_client_args()
+        self.secret_client = SecretClient(**args)
         self.secret_name = "secret-name-" + uuid.uuid1().hex
         self.secret_value = "secret-value"
 

doc/README.md

@@ -2,4 +2,6 @@ This folder contains some documentations for this repository:
 
 The folder structure is the following
 - [sphinx](./sphinx) : contains the documentation source code for https://azure.github.io/azure-sdk-for-python/
-- [dev](./dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK)
+- [dev](./dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK)
+
+The file [eng_sys_checks](eng_sys_checks.md) is a read up as to what a standard `ci.yml` will actually execute.

doc/dev/mgmt/generating-integration-test.md

@@ -123,7 +123,7 @@ now you can run live integration test:
 
     pytest -s sdk/attestation/azure-mgmt-attestation
 
->NOTE: To create service principal, follow instructions here: https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal
+>NOTE: To create service principal, follow instructions here: https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal
 
 ## Fixing Test
 

doc/dev/mgmt/tests.md

@@ -66,10 +66,10 @@ you must use an OAuth authentication method which gives you a token:
 
 Certificate authentication does not allow you to record HTTP queries for testing.
 
-### Get a token with Azure Active Directory user/password. This is considered deprecated and should not be used anymore (https://docs.microsoft.com/en-us/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-legacy).
+### Get a token with Azure Active Directory user/password. This is considered deprecated and should not be used anymore (https://docs.microsoft.com/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-legacy).
 
 1.  Connect to the [Azure Classic Portal](https://manage.windowsazure.com/) with your admin account.
-2.  Create a user in your default AAD https://azure.microsoft.com/en-us/documentation/articles/active-directory-create-users/
+2.  Create a user in your default AAD https://azure.microsoft.com/documentation/articles/active-directory-create-users/
    **You must NOT activate Multi-Factor Authentication!**
 3.  Go to Settings - Administrators.
 4.  Click on *Add* and enter the email of the new user.
@@ -89,13 +89,13 @@ credentials = UserPassCredentials(
 ### Get a token with Active Directory application and service principal
 
 Follow this detailed tutorial to set up an Active Directory application and service principal:
-https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-service-principal-portal/
+https://azure.microsoft.com/documentation/articles/resource-group-create-service-principal-portal/
 
 To use the credentials from Python,
 you need the application ID (a.k.a. client ID),
 authentication key (a.k.a. client secret),
 tenant ID and subscription ID from the Azure portal for use in the next step.
-[This section of the above tutorial](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
+[This section of the above tutorial](https://docs.microsoft.com/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key)
 describes where to find them
 (besides the subscription ID,
 which is in the "Overview" section of the "Subscriptions" blade.)

doc/dev/release.md

@@ -16,11 +16,11 @@ Python packages are uploaded to [PyPI](https://pypi.org/). Once you've uploaded
 
 ### Production - Deploy with Azure Dev Ops
 
-To avoid "accidental" pushes to our target repositories, [approval](https://docs.microsoft.com/en-us/azure/devops/pipelines/release/approvals/approvals?view=azure-devops) will be requested directly prior to the final PyPI publish. Reference this [wiki page](https://aka.ms/python-approval-groups) and click on `Release to PyPI Approvers` to add yourself to the group for PyPI publishing. 
+To avoid "accidental" pushes to our target repositories, [approval](https://docs.microsoft.com/azure/devops/pipelines/release/approvals/approvals?view=azure-devops) will be requested directly prior to the final PyPI publish. Reference this [wiki page](https://aka.ms/python-approval-groups) and click on `Release to PyPI Approvers` to add yourself to the group for PyPI publishing. 
 
 Instead of a single central pipeline, the python SDK has moved to `service directory` associated build pipelines. These are driven by yml templates at the root of each service folder. [Example for storage service folder.](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/storage/ci.yml#L44)
 
-As an aside, please note that the preview feature `multi-stage pipelines` must be enabled to properly interact with unified pipelines. If you aren't aware, find out how to enable by visiting [this link.](https://docs.microsoft.com/en-us/azure/devops/project/navigation/preview-features?view=azure-devops)
+As an aside, please note that the preview feature `multi-stage pipelines` must be enabled to properly interact with unified pipelines. If you aren't aware, find out how to enable by visiting [this link.](https://docs.microsoft.com/azure/devops/project/navigation/preview-features?view=azure-devops)
 
 #### Releasing Through Unified Pipelines