[Schema Registry + Avro Serializer] 1.0.0b1 (#13124)

* init commit

* avro serializer structure

* adding avro serializer

* tweak api version and fix a typo

* test template

* avro serializer sync draft

* major azure sr client work done

* add sample docstring for sr

* avro serializer async impl

* close the writer

* update avro se/de impl

* update avro serializer impl

* fix apireview reported error in sr

* srav namespace, setup update

* doc update

* update doc and api

* impl, doc update

* partial update according to laruent's feedback

* be consistent with eh extension structure

* more update code according to feedback

* update credential config

* rename package name to azure-schemaregistry-avroserializer

* fix pylint

* try ci fix

* fix test for py27 as avro only accept unicode

* first round of review feedback

* remove temp ci experiment

* init add conftest.py to pass py2.7 test

* laurent feedback update

* remove dictmixin for b1, update comment in sample

* update api in avroserializer and update test and readme

* update test, docs and links

* add share requirement

* update avro dependency

* pr feedback and livetest update

eng/ignore-links.txt

@@ -1 +1,10 @@
-https://docs.microsoft.com/python/api/overview/azure/{{package_doc_id}}
+https://docs.microsoft.com/python/api/overview/azure/{{package_doc_id}}
+https://pypi.org/project/azure-schemaregistry
+https://azuresdkdocs.blob.core.windows.net/$web/python/azure-schemaregistry/latest/index.html
+https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry/samples/sync_samples/schema_registry.py
+https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry/samples/async_samples/schema_registry_async.py
+https://pypi.org/project/azure-schemaregistry-avroserializer
+https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer/CHANGELOG.md
+https://azuresdkdocs.blob.core.windows.net/$web/python/azure-schemaregistry-avroserializer/latest/index.html
+https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer/samples
+https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer/samples/avro_serializer.py

sdk/schemaregistry/azure-schemaregistry-avroserializer/CHANGELOG.md

@@ -0,0 +1,11 @@
+# Release History
+
+## 1.0.0b1 (2020-09-08)
+
+Version 1.0.0b1 is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Schema Registry Avro Serializer.
+
+**New features**
+
+- `SchemaRegistryAvroSerializer` is the top-level client class that provides the functionality to encode and decode avro data utilizing the avro library. It will automatically register schema and retrieve schema from Azure Schema Registry Service. It provides two methods:
+  - `serialize`: Serialize dict data into bytes according to the given schema and register schema if needed.
+  - `deserialize`: Deserialize bytes data into dict data by automatically retrieving schema from the service.

sdk/schemaregistry/azure-schemaregistry-avroserializer/MANIFEST.in

@@ -0,0 +1,6 @@
+include *.md
+include azure/__init__.py
+include azure/schemaregistry/__init__.py
+include azure/schemaregistry/serializer/__init__.py
+recursive-include tests *.py
+recursive-include samples *.py

sdk/schemaregistry/azure-schemaregistry-avroserializer/README.md

@@ -0,0 +1,174 @@
+# Azure Schema Registry Avro Serializer client library for Python
+
+Azure Schema Registry Avro Serializer provides the ability to serialize and deserialize data according
+to the given avro schema. It is integrated with Azure Schema Registry SDK and will automatically register and get schema.
+
+[Source code][source_code] | [Package (PyPi)][pypi] | [API reference documentation][api_docs] | [Samples][sr_avro_samples] | [Changelog][change_log]
+
+## Getting started
+
+### Install the package
+
+Install the Azure Schema Registry Avro Serializer client library and Azure Identity client library for Python with [pip][pip]:
+
+```Bash
+pip install azure-schemaregistry-avroserializer azure-identity
+```
+
+### Prerequisites: 
+To use this package, you must have:
+* Azure subscription - [Create a free account][azure_sub]
+* Azure Schema Registry
+* Python 2.7, 3.5 or later - [Install Python][python]
+
+### Authenticate the client
+Interaction with Schema Registry Avro Serializer starts with an instance of SchemaRegistryAvroSerializer class. You need the endpoint, AAD credential and schema group name to instantiate the client object. 
+
+**Create client using the azure-identity library:**
+
+```python
+from azure.schemaregistry import SchemaRegistryClient
+from azure.schemaregistry.serializer.avroserializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+credential = DefaultAzureCredential()
+endpoint = '<< ENDPOINT OF THE SCHEMA REGISTRY >>'
+schema_group = '<< GROUP NAME OF THE SCHEMA >>'
+schema_registry_client = SchemaRegistryClient(endpoint, credential)
+serializer = SchemaRegistryAvroSerializer(schema_registry_client, schema_group)
+```
+
+## Key concepts
+
+- Avro: Apache Avro™ is a data serialization system.
+
+## Examples
+
+The following sections provide several code snippets covering some of the most common Schema Registry tasks, including:
+
+- [Serialization](#serialization)
+- [Deserialization](#deserialization)
+
+### Serialization
+
+```python
+import os
+from azure.schemaregistry import SchemaRegistryClient
+from azure.schemaregistry.serializer.avroserializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+token_credential = DefaultAzureCredential()
+endpoint = os.environ['SCHEMA_REGISTRY_ENDPOINT']
+schema_group = "<your-group-name>"
+
+schema_registry_client = SchemaRegistryClient(endpoint, token_credential)
+serializer = SchemaRegistryAvroSerializer(schema_registry_client, schema_group)
+
+schema_string = """
+{"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+     {"name": "name", "type": "string"},
+     {"name": "favorite_number",  "type": ["int", "null"]},
+     {"name": "favorite_color", "type": ["string", "null"]}
+ ]
+}"""
+
+with serializer:
+    dict_data = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
+    encoded_bytes = serializer.serialize(dict_data, schema_string)
+```
+
+### Deserialization
+
+```python
+import os
+from azure.schemaregistry import SchemaRegistryClient
+from azure.schemaregistry.serializer.avroserializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+token_credential = DefaultAzureCredential()
+endpoint = os.environ['SCHEMA_REGISTRY_ENDPOINT']
+schema_group = "<your-group-name>"
+
+schema_registry_client = SchemaRegistryClient(endpoint, token_credential)
+serializer = SchemaRegistryAvroSerializer(schema_registry_client, schema_group)
+
+with serializer:
+    encoded_bytes = b'<data_encoded_by_azure_schema_registry_avro_serializer>'
+    decoded_data = serializer.deserialize(encoded_bytes)
+```
+
+## Troubleshooting
+
+### General
+
+Azure Schema Registry Avro Serializer raise exceptions defined in [Azure Core][azure_core].
+
+### Logging
+This library uses the standard
+[logging][python_logging] library for logging.
+Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
+level.
+
+Detailed DEBUG level logging, including request/response bodies and unredacted
+headers, can be enabled on a client with the `logging_enable` argument:
+```python
+import sys
+import logging
+from azure.schemaregistry import SchemaRegistryClient
+from azure.schemaregistry.serializer.avroserializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+# Create a logger for the SDK
+logger = logging.getLogger('azure.schemaregistry')
+logger.setLevel(logging.DEBUG)
+
+# Configure a console output
+handler = logging.StreamHandler(stream=sys.stdout)
+logger.addHandler(handler)
+
+credential = DefaultAzureCredential()
+schema_registry_client = SchemaRegistryClient("<your-end-point>", credential)
+# This client will log detailed information about its HTTP sessions, at DEBUG level
+serializer = SchemaRegistryAvroSerializer(schema_registry_client, "<your-group-name>", logging_enable=True)
+```
+
+Similarly, `logging_enable` can enable detailed logging for a single operation,
+even when it isn't enabled for the client:
+```py
+serializer.serialie(dict_data, schema_content, logging_enable=True)
+```
+
+## Next steps
+
+### More sample code
+
+Please find further examples in the [samples][sr_avro_samples] directory demonstrating common Azure Schema Registry Avro Serializer scenarios.
+
+## Contributing
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+<!-- LINKS -->
+[pip]: https://pypi.org/project/pip/
+[pypi]: https://pypi.org/project/azure-schemaregistry-avroserializer
+[python]: https://www.python.org/downloads/
+[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/README.md
+[azure_sub]: https://azure.microsoft.com/free/
+[python_logging]: https://docs.python.org/3/library/logging.html
+[sr_avro_samples]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer/samples
+[api_reference]: https://azuresdkdocs.blob.core.windows.net/$web/python/azure-schemaregistry-avroserializer/latest/index.html
+[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer
+[change_log]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/schemaregistry/azure-schemaregistry-avroserializer/CHANGELOG.md

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/__init__.py

@@ -0,0 +1,26 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/__init__.py

@@ -0,0 +1,25 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/serializer/__init__.py

@@ -0,0 +1,25 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/serializer/avroserializer/__init__.py

@@ -0,0 +1,34 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+from ._version import VERSION
+
+__version__ = VERSION
+
+from ._schema_registry_avro_serializer import SchemaRegistryAvroSerializer
+
+__all__ = [
+    "SchemaRegistryAvroSerializer"
+]