Skip to content

Commit

Permalink
Update blobs migration guide to include GenerateSas; Fix storage migr…
Browse files Browse the repository at this point in the history 
…ation test bugs (#22380)
  • Loading branch information
@amnguye
amnguye authored Jul 2, 2021
1 parent fec743d commit 38441a4
Show file tree
Hide file tree
Showing 2 changed files with 182 additions and 68 deletions.
51 changes: 40 additions & 11 deletions sdk/storage/Azure.Storage.Blobs/AzureStorageNetMigrationV12.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ v12
The new library only supports constructing a client with a fully constructed SAS URI. Note that since client URIs are immutable once created, a new client instance with a new SAS must be created in order to rotate a SAS.

```C# Snippet:SampleSnippetsBlobMigration_SasUri
BlobClient blob = new BlobClient(new Uri(blobLocationWithSas));
BlobClient blob = new BlobClient(sasUri);
```

#### Connection string
Expand Down Expand Up @@ -446,7 +446,7 @@ v12

v12 has explicit methods for listing by hierarchy.
```C# Snippet:SampleSnippetsBlobMigration_ListHierarchy
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix);
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix, delimiter: delimiter);
await foreach (BlobHierarchyItem item in results)
{
MyConsumeBlobItemFunc(item);
Expand Down Expand Up @@ -546,19 +546,50 @@ sasBlobToken = blob.GetSharedAccessSignature(null, policyName);

v12

The modern SDK uses a builder pattern for constructing a SAS token. Clients are not involved in the process.
The modern SDK uses a builder pattern for constructing a SAS token. Similar to the pattern to create a SAS in v11, you can generate a SAS URI from the client. This is the preferred method in order to prevent passing the key to more than one place. It also is more convenient to generate from the client in the case of authenticating with a connection string, as you would not have to parse the connection string to grab the storage account name and key.

To create a simple SAS with any optional parameters, use the convenience overload of GenerateSas which only requires taking in permissions and the expiry time.

```C# Snippet:SampleSnippetsBlobMigration_GenerateSas
// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);

Uri sasUri;
// Ensure our client has the credentials required to generate a SAS
if (blobClient.CanGenerateSasUri)
{
// Create full, self-authenticating URI to the resource from the BlobClient
sasUri = blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1));

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
```

To create a more complex SAS pass the SAS builder to the GenerateSas method.

```C# Snippet:SampleSnippetsBlobMigration_GenerateSas_Builder
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// Since we are generating from the client, the client will have the container and blob name
// Specify any optional paremeters here
StartsOn = DateTimeOffset.UtcNow.AddHours(-1)
};

// Create full, self-authenticating URI to the resource from the BlobClient
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);
```

You can also generate a SAS without use of the client.

```C# Snippet:SampleSnippetsBlobMigration_SasBuilder
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// with no url in a client to read from, container and blob name must be provided if applicable
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
BlobName = blobName
};
// permissions applied separately, using the appropriate enum to the scope of your SAS
sasBuilder.SetPermissions(BlobSasPermissions.Read);

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
Expand All @@ -574,10 +605,8 @@ If using a stored access policy, construct your `BlobSasBuilder` from the exampl

```C# Snippet:SampleSnippetsBlobMigration_SasBuilderIdentifier
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
Identifier = "mysignedidentifier"
};
```
Expand Down
199 changes: 142 additions & 57 deletions sdk/storage/Azure.Storage.Blobs/samples/Sample03_Migrations.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,27 +55,14 @@ public void AuthWithSasCredential()
try
{
container.Create();
container.GetBlobClient(blobName).Upload(BinaryData.FromString("hello world"));
BlobClient blobClient = container.GetBlobClient(blobName);
blobClient.Upload(BinaryData.FromString("hello world"));

// build SAS URI for sample
BlobSasBuilder sas = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
};
sas.SetPermissions(BlobSasPermissions.All);

StorageSharedKeyCredential credential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

UriBuilder sasUri = new UriBuilder(this.StorageAccountBlobUri);
sasUri.Path = $"{containerName}/{blobName}";
sasUri.Query = sas.ToSasQueryParameters(credential).ToString();

string blobLocationWithSas = sasUri.Uri.ToString();
Uri sasUri = blobClient.GenerateSasUri(BlobSasPermissions.All, DateTimeOffset.UtcNow.AddHours(1));

#region Snippet:SampleSnippetsBlobMigration_SasUri
BlobClient blob = new BlobClient(new Uri(blobLocationWithSas));
BlobClient blob = new BlobClient(sasUri);
#endregion

var stream = new MemoryStream();
Expand Down Expand Up @@ -584,17 +571,18 @@ void MyConsumeBlobItemFunc(BlobHierarchyItem item)

// show in snippet where the prefix goes, but our test doesn't want a prefix for its data set
string blobPrefix = null;
string delimiter = "/";

#region Snippet:SampleSnippetsBlobMigration_ListHierarchy
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix);
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix, delimiter: delimiter);
await foreach (BlobHierarchyItem item in results)
{
MyConsumeBlobItemFunc(item);
}
#endregion

Assert.IsTrue(expectedBlobNamesResult.SetEquals(downloadedBlobNames));
Assert.IsTrue(new HashSet<string> { virtualDirName }.SetEquals(downloadedPrefixNames));
Assert.IsTrue(new HashSet<string> { virtualDirName + '/' }.SetEquals(downloadedPrefixNames));
}
finally
{
Expand Down Expand Up @@ -700,15 +688,12 @@ public async Task SasBuilder()

#region Snippet:SampleSnippetsBlobMigration_SasBuilder
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// with no url in a client to read from, container and blob name must be provided if applicable
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
BlobName = blobName
};
// permissions applied separately, using the appropriate enum to the scope of your SAS
sasBuilder.SetPermissions(BlobSasPermissions.Read);

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
Expand All @@ -729,13 +714,103 @@ public async Task SasBuilder()
}
}

[Test]
public async Task GenerateSas()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
var container = new BlobContainerClient(ConnectionString, containerName);
BlobUriBuilder uriBuilder = new BlobUriBuilder(container.Uri) { BlobName = blobName };
Uri blobUri = uriBuilder.ToUri();

try
{
await container.CreateAsync();
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

#region Snippet:SampleSnippetsBlobMigration_GenerateSas
// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);

Uri sasUri;
// Ensure our client has the credentials required to generate a SAS
if (blobClient.CanGenerateSasUri)
{
// Create full, self-authenticating URI to the resource from the BlobClient
sasUri = blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1));

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
#endregion
else
{
Assert.Fail("Unable to create SAS URI");
}
}
finally
{
await container.DeleteIfExistsAsync();
}
}

[Test]
public async Task GenerateSas_Builder()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
var container = new BlobContainerClient(ConnectionString, containerName);
BlobUriBuilder uriBuilder = new BlobUriBuilder(container.Uri) { BlobName = blobName };
Uri blobUri = uriBuilder.ToUri();

try
{
await container.CreateAsync();
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);
// Create BlobSasBuilder and specify parameters
#region Snippet:SampleSnippetsBlobMigration_GenerateSas_Builder
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// Since we are generating from the client, the client will have the container and blob name
// Specify any optional paremeters here
StartsOn = DateTimeOffset.UtcNow.AddHours(-1)
};

// Create full, self-authenticating URI to the resource from the BlobClient
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);
#endregion

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
finally
{
await container.DeleteIfExistsAsync();
}
}

[Test]
public async Task SasBuilderIdentifier()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
DateTimeOffset expiresOn = DateTimeOffset.UtcNow.AddDays(1);
DateTimeOffset startsOn = DateTimeOffset.UtcNow.AddHours(-1);
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
Expand All @@ -744,6 +819,7 @@ public async Task SasBuilderIdentifier()
try
{
await container.CreateAsync();
BlobClient blobClient = container.GetBlobClient(blobName);
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

// Create one or more stored access policies.
Expand All @@ -754,8 +830,8 @@ public async Task SasBuilderIdentifier()
Id = "mysignedidentifier",
AccessPolicy = new BlobAccessPolicy
{
StartsOn = DateTimeOffset.UtcNow.AddHours(-1),
ExpiresOn = DateTimeOffset.UtcNow.AddDays(1),
StartsOn = startsOn,
ExpiresOn = expiresOn,
Permissions = "rw"
}
}
Expand All @@ -765,22 +841,14 @@ public async Task SasBuilderIdentifier()

#region Snippet:SampleSnippetsBlobMigration_SasBuilderIdentifier
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
Identifier = "mysignedidentifier"
};
#endregion

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
{
BlobContainerName = containerName,
BlobName = blobName,
Sas = sasBuilder.ToSasQueryParameters(sharedKeyCredential)
};
Uri sasUri = uriBuilder.ToUri();
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);

// successful download indicates pass
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
Expand Down Expand Up @@ -926,18 +994,26 @@ public async Task RetryPolicy()
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-file");
var containerClient = new BlobContainerClient(ConnectionString, containerName);
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_RetryPolicy
BlobClientOptions blobClientOptions = new BlobClientOptions();
blobClientOptions.Retry.Mode = RetryMode.Exponential;
blobClientOptions.Retry.Delay = TimeSpan.FromSeconds(10);
blobClientOptions.Retry.MaxRetries = 6;
BlobServiceClient service = new BlobServiceClient(connectionString, blobClientOptions);
BlobClient blobClient = service.GetBlobContainerClient(containerName).GetBlobClient(blobName);
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream);
#endregion
try
{
await containerClient.CreateIfNotExistsAsync();
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_RetryPolicy
BlobClientOptions blobClientOptions = new BlobClientOptions();
blobClientOptions.Retry.Mode = RetryMode.Exponential;
blobClientOptions.Retry.Delay = TimeSpan.FromSeconds(10);
blobClientOptions.Retry.MaxRetries = 6;
BlobServiceClient service = new BlobServiceClient(connectionString, blobClientOptions);
BlobClient blobClient = service.GetBlobContainerClient(containerName).GetBlobClient(blobName);
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream);
#endregion
}
finally
{
await containerClient.DeleteIfExistsAsync();
}

Assert.Pass();
}
Expand All @@ -953,15 +1029,24 @@ public async Task MaximumExecutionTime()
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-file");
var containerClient = new BlobContainerClient(ConnectionString, containerName);
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_MaximumExecutionTime
BlobClient blobClient = containerClient.GetBlobClient(blobName);
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30));
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream, cancellationTokenSource.Token);
#endregion

try
{
await containerClient.CreateIfNotExistsAsync();
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_MaximumExecutionTime
BlobClient blobClient = containerClient.GetBlobClient(blobName);
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30));
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream, cancellationTokenSource.Token);
#endregion
}
finally
{
await containerClient.DeleteIfExistsAsync();
}

Assert.Pass();
}
Expand Down

0 comments on commit 38441a4

Please sign in to comment.