Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New SDK - SchemaRegistry #13657

Merged
merged 51 commits into from
Sep 5, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
63045ea
Generated the SchemaRegistry SDK. Set up project and custom client. F…
MiYanni Jul 22, 2020
ce5b15c
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Jul 30, 2020
6b44ba3
Update to latest swagger with description fixes.
MiYanni Jul 31, 2020
e0b8961
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Aug 5, 2020
4c7a288
Fixed changlog. Regen surface.
MiYanni Aug 5, 2020
202090a
Regen with correct swagger.
MiYanni Aug 5, 2020
96e42d3
Started adding client methods.
MiYanni Aug 6, 2020
66a4426
Update SchemaId to be struct. Regen api.
MiYanni Aug 7, 2020
aa5b5cb
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Aug 11, 2020
86932ac
Initial attempt of serializer.
MiYanni Aug 12, 2020
7ff033c
Update swagger version. Regenerate rest client. Update public surface…
MiYanni Aug 13, 2020
ba0addd
Added happy path tests for each client method. Added recordings.
MiYanni Aug 13, 2020
3c8e1b1
Move serialization stuff into the Azure.Data.SchemaRegistry.Avro proj…
MiYanni Aug 21, 2020
3853114
Modify SchemaProperties to have smaller set of properties. Rerun and …
MiYanni Aug 21, 2020
1b5999c
Still attempting to get Avro library usable functionality. Some refle…
MiYanni Aug 27, 2020
eaec4d9
Initial serializer implementation. Only does specific records.
MiYanni Aug 29, 2020
8ca6a68
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Aug 31, 2020
af23793
Fixed issues with specific record schema de/serialization.
MiYanni Sep 1, 2020
8d04053
Got GenericRecord working. Needs cleanup.
MiYanni Sep 1, 2020
306fa4f
Code cleanup begins!
MiYanni Sep 1, 2020
d65d746
:-) Cleaned up code for serializers. Wrote proper async implementation.
MiYanni Sep 2, 2020
6daf008
Renamed classes to be more accurate. Removal of commented code.
MiYanni Sep 2, 2020
aff9ef7
Moved serializer over to main project. Removed other project and solu…
MiYanni Sep 2, 2020
52220c9
Added 1 test for the serializer.
MiYanni Sep 3, 2020
9940ec4
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Sep 3, 2020
817f11d
Update api.
MiYanni Sep 3, 2020
b61cd0e
Added back the separate solution. Named it appropriately.
MiYanni Sep 3, 2020
96887b6
Addressing some changes from API view comments.
MiYanni Sep 4, 2020
7b25995
Added some negative tests. Updated public surface descriptions.
MiYanni Sep 4, 2020
88339c2
In progress writing readmes.
MiYanni Sep 4, 2020
5194247
Fixed position reset in deserialization stream.
MiYanni Sep 4, 2020
0e1122b
Made constants for wire format lengths/positions.
MiYanni Sep 4, 2020
a46462b
Updated property description. Made static field for UTF8 encoding.
MiYanni Sep 4, 2020
2d19b51
Removed unused Directory.Build.props.
MiYanni Sep 4, 2020
54f3d48
Removed unused description.
MiYanni Sep 4, 2020
88d46a2
Continuing readme. Started adding snippets.
MiYanni Sep 4, 2020
608ada2
Merge branch 'master' into schemaregistry-new-sdk
MiYanni Sep 4, 2020
7008405
First readme completed.
MiYanni Sep 4, 2020
6317936
Finally! Created readmes and snippets.
MiYanni Sep 4, 2020
713b987
Put credential scope string into a constant.
MiYanni Sep 4, 2020
09adad4
Addressing a couple documentation PR comments and removed a code comm…
MiYanni Sep 4, 2020
f331a51
Added global suppressions for namespace name.
MiYanni Sep 5, 2020
747e336
Forgot using for memory streams in tests and snippets.
MiYanni Sep 5, 2020
3f1cd01
Fixed bugged diagnostic scope names. Put scope names into constants.
MiYanni Sep 5, 2020
c87bb3b
Updated ci configuration. Removed resources template since resources …
MiYanni Sep 5, 2020
e152c57
Added the other package to CI.
MiYanni Sep 5, 2020
ff50a0c
Updated sample file names to match convention.
MiYanni Sep 5, 2020
87348be
Added missing readme links.
MiYanni Sep 5, 2020
aa78861
Fixed the readme snippets so they don't format oddly.
MiYanni Sep 5, 2020
7d9a67f
Updated schema components descriptions.
MiYanni Sep 5, 2020
03d9a62
Fixed command to acquire packages.
MiYanni Sep 5, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.29709.97
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Azure.Data.SchemaRegistry", "src\Azure.Data.SchemaRegistry.csproj", "{E33D09D9-D809-472C-82E6-6A26BDB86FC2}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Azure.Data.SchemaRegistry.Tests", "tests\Azure.Data.SchemaRegistry.Tests.csproj", "{4F476D56-DDE7-43D3-8CB4-BA1E77F5A300}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Azure.Core.TestFramework", "..\..\core\Azure.Core.TestFramework\src\Azure.Core.TestFramework.csproj", "{8052009B-2126-44A3-88CD-4F3B17894C64}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{E33D09D9-D809-472C-82E6-6A26BDB86FC2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E33D09D9-D809-472C-82E6-6A26BDB86FC2}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E33D09D9-D809-472C-82E6-6A26BDB86FC2}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E33D09D9-D809-472C-82E6-6A26BDB86FC2}.Release|Any CPU.Build.0 = Release|Any CPU
{4F476D56-DDE7-43D3-8CB4-BA1E77F5A300}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4F476D56-DDE7-43D3-8CB4-BA1E77F5A300}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4F476D56-DDE7-43D3-8CB4-BA1E77F5A300}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4F476D56-DDE7-43D3-8CB4-BA1E77F5A300}.Release|Any CPU.Build.0 = Release|Any CPU
{8052009B-2126-44A3-88CD-4F3B17894C64}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{8052009B-2126-44A3-88CD-4F3B17894C64}.Debug|Any CPU.Build.0 = Debug|Any CPU
{8052009B-2126-44A3-88CD-4F3B17894C64}.Release|Any CPU.ActiveCfg = Release|Any CPU
{8052009B-2126-44A3-88CD-4F3B17894C64}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {A97F4B90-2591-4689-B1F8-5F21FE6D6CAE}
EndGlobalSection
EndGlobal
7 changes: 7 additions & 0 deletions sdk/schemaregistry/Azure.Data.SchemaRegistry/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Release History

## 1.0.0-beta.1 (2020-09-08)
- Added SchemaRegistryClient with 3 operations:
- RegisterSchema
- GetSchemaId
- GetSchema
158 changes: 158 additions & 0 deletions sdk/schemaregistry/Azure.Data.SchemaRegistry/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# Azure Schema Registry client library for .NET

The Azure Schema Registry service allows developers to provide and retrieve data schemas from a centralized repository for use in messaging systems.

## Getting started

### Install the package

Install the Azure Schema Registry client library for .NET with [NuGet][nuget]:

```bash
dotnet add package Azure.Data.SchemaRegistry --version 1.0.0-beta.1
```

### Prerequisites

* An [Azure subscription][azure_sub]
* An [Event Hubs namespace][event_hubs_namespace]

If you need to [create an Event Hubs namespace][create_event_hubs_namespace], you can use the Azure Portal or [Azure PowerShell][azure_powershell].

You can use Azure PowerShell to create the Event Hubs namespace with the following command:

```PowerShell
New-AzEventHubNamespace -ResourceGroupName myResourceGroup -NamespaceName namespace_name -Location eastus
```

### Authenticate the client

In order to interact with the Azure Schema Registry service, you'll need to create an instance of the [Schema Registry Client][schema_registry_client] class. To create this client, you'll need Azure resource credentials and the Event Hubs namespace hostname.

#### Get credentials

To acquire authenicated credentials and start interacting with Azure resources, please see the [quickstart guide here][quickstart_guide].

#### Get Event Hubs namespace hostname

The simpliest way is to use the [Azure portal][azure_portal] and navigate to your Event Hubs namespace. From the Overview tab, you'll see `Host name`. Copy the value from this field.

#### Create SchemaRegistryClient

Once you have the Azure resource credentials and the Event Hubs namespace hostname, you can create the [SchemaRegistryClient][schema_registry_client]. You'll also need the [Azure.Identity][azure_identity] package to create the credential.

```C# Snippet:CreateSchemaRegistryClient
string endpoint = "<event_hubs_namespace_hostname>";
var credentials = new ClientSecretCredential(
"<tenant_id>",
"<client_id>",
"<client_secret>"
);
var client = new SchemaRegistryClient(endpoint, credentials);
```

## Key concepts

### Schemas

A schema has 6 components:
- Group Name: The name of the group of schemas in the Schema Registry instance.
- Schema Name: The name of the schema.
- Schema ID: The ID assigned by the Schema Registry instance for the schema.
- Serialization Type: The format used for serialization of the schema. For example, Avro.
- Schema Content: The string representation of the schema.
- Schema Version: The version assigned to the schema in the Schema Registry instance.

These components play different roles. Some are used as input into the operations and some are outputs. Currently, [SchemaProperties][schema_properties] only exposes those properties that are potential outputs that are used in SchemaRegistry operations. Those exposed properties are `Content` and `Id`.

## Examples

The following shows examples of what is available through the SchemaRegistryClient. There are both sync and async methods available for these client operations.

* [Register a schema](#register-a-schema)
* [Retrieve a schema ID](#retrieve-a-schema-id)
* [Retrieve a schema](#retrieve-a-schema)

### Register a schema

Register a schema to be stored in the Azure Schema Registry.

```C# Snippet:RegisterSchema
MiYanni marked this conversation as resolved.
Show resolved Hide resolved
string schemaName = "<schema_name>";
string groupName = "<schema_group_name>";
SerializationType schemaType = SerializationType.Avro;
// Example schema's content
string schemaContent = @"
{
""type"" : ""record"",
""namespace"" : ""TestSchema"",
""name"" : ""Employee"",
""fields"" : [
{ ""name"" : ""Name"" , ""type"" : ""string"" },
{ ""name"" : ""Age"", ""type"" : ""int"" }
]
}";

Response<SchemaProperties> schemaProperties = client.RegisterSchema(groupName, schemaName, schemaType, schemaContent);
```

### Retrieve a schema ID

Retrieve a previously registered schema ID from the Azure Schema Registry.

```C# Snippet:RetrieveSchemaId
string schemaName = "<schema_name>";
string groupName = "<schema_group_name>";
SerializationType schemaType = SerializationType.Avro;
// Example schema's content
string schemaContent = @"
{
""type"" : ""record"",
""namespace"" : ""TestSchema"",
""name"" : ""Employee"",
""fields"" : [
{ ""name"" : ""Name"" , ""type"" : ""string"" },
{ ""name"" : ""Age"", ""type"" : ""int"" }
]
}";

Response<SchemaProperties> schemaProperties = client.GetSchemaId(groupName, schemaName, schemaType, schemaContent);
string schemaId = schemaProperties.Value.Id;
```

### Retrieve a schema

Retrieve a previously registered schema's content from the Azure Schema Registry.

```C# Snippet:RetrieveSchema
string schemaId = "<schema_id>";

Response<SchemaProperties> schemaProperties = client.GetSchema(schemaId);
string schemaContent = schemaProperties.Value.Content;
```

MiYanni marked this conversation as resolved.
Show resolved Hide resolved
## Contributing
MiYanni marked this conversation as resolved.
Show resolved Hide resolved

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][code_of_conduct_faq] or contact [opencode@microsoft.com][email_opencode] with any additional questions or comments.

![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-net%2Fsdk%2Ftemplate%2FAzure.Template%2FREADME.png)

<!-- LINKS -->
[nuget]: https://www.nuget.org/
[event_hubs_namespace]: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about
[azure_powershell]: https://docs.microsoft.com/en-us/powershell/azure/
[create_event_hubs_namespace]: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-powershell#create-an-event-hubs-namespace
[quickstart_guide]: https://github.com/Azure/azure-sdk-for-net/blob/master/doc/mgmt_preview_quickstart.md
[schema_registry_client]: src/SchemaRegistryClient.cs
[azure_portal]: https://ms.portal.azure.com/
[schema_properties]: src/SchemaProperties.cs
[azure_identity]: https://www.nuget.org/packages/Azure.Identity
[cla]: https://cla.microsoft.com
[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
[code_of_conduct_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[email_opencode]: mailto:opencode@microsoft.com
[azure_sub]: https://azure.microsoft.com/free/
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
namespace Azure.Data.SchemaRegistry
{
public partial class SchemaProperties
{
internal SchemaProperties() { }
public string Content { get { throw null; } }
public string Id { get { throw null; } }
}
public partial class SchemaRegistryClient
{
protected SchemaRegistryClient() { }
public SchemaRegistryClient(string endpoint, Azure.Core.TokenCredential credential) { }
public SchemaRegistryClient(string endpoint, Azure.Core.TokenCredential credential, Azure.Data.SchemaRegistry.SchemaRegistryClientOptions options) { }
public virtual Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties> GetSchema(string schemaId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public virtual System.Threading.Tasks.Task<Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties>> GetSchemaAsync(string schemaId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public virtual Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties> GetSchemaId(string groupName, string schemaName, Azure.Data.SchemaRegistry.SerializationType serializationType, string schemaContent, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public virtual System.Threading.Tasks.Task<Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties>> GetSchemaIdAsync(string groupName, string schemaName, Azure.Data.SchemaRegistry.SerializationType serializationType, string schemaContent, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public virtual Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties> RegisterSchema(string groupName, string schemaName, Azure.Data.SchemaRegistry.SerializationType serializationType, string schemaContent, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public virtual System.Threading.Tasks.Task<Azure.Response<Azure.Data.SchemaRegistry.SchemaProperties>> RegisterSchemaAsync(string groupName, string schemaName, Azure.Data.SchemaRegistry.SerializationType serializationType, string schemaContent, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
}
public partial class SchemaRegistryClientOptions : Azure.Core.ClientOptions
{
public SchemaRegistryClientOptions(Azure.Data.SchemaRegistry.SchemaRegistryClientOptions.ServiceVersion version = Azure.Data.SchemaRegistry.SchemaRegistryClientOptions.ServiceVersion.V2017_04) { }
public enum ServiceVersion
{
V2017_04 = 1,
}
}
[System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)]
public readonly partial struct SerializationType : System.IEquatable<Azure.Data.SchemaRegistry.SerializationType>
{
private readonly object _dummy;
private readonly int _dummyPrimitive;
public SerializationType(string value) { throw null; }
public static Azure.Data.SchemaRegistry.SerializationType Avro { get { throw null; } }
public bool Equals(Azure.Data.SchemaRegistry.SerializationType other) { throw null; }
[System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
public override bool Equals(object obj) { throw null; }
[System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
public override int GetHashCode() { throw null; }
public static bool operator ==(Azure.Data.SchemaRegistry.SerializationType left, Azure.Data.SchemaRegistry.SerializationType right) { throw null; }
public static implicit operator Azure.Data.SchemaRegistry.SerializationType (string value) { throw null; }
public static bool operator !=(Azure.Data.SchemaRegistry.SerializationType left, Azure.Data.SchemaRegistry.SerializationType right) { throw null; }
public override string ToString() { throw null; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<Description>Azure Schema Registry SDK</Description>
<AssemblyTitle>Azure Schema Registry SDK</AssemblyTitle>
<Version>1.0.0-beta.1</Version>
<PackageTags>Azure;Schema Registry;SchemaRegistry;.NET;Data;$(PackageCommonTags)</PackageTags>
<TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Apache.Avro" VersionOverride="1.10.0" />
</ItemGroup>

<!-- Shared source from Azure.Core -->
<ItemGroup>
<Compile Include="$(AzureCoreSharedSources)ArrayBufferWriter.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)ClientDiagnostics.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)ContentTypeUtilities.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)DiagnosticScope.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)DiagnosticScopeFactory.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)HttpMessageSanitizer.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)TaskExtensions.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)OperationHelpers.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
<Compile Include="$(AzureCoreSharedSources)Argument.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" />
</ItemGroup>

<Import Project="$(MSBuildThisFileDirectory)..\..\..\core\Azure.Core\src\Azure.Core.props" />

</Project>

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading