Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update blobs migration guide to include GenerateSas; Fix migration test bugs #22380

Merged
merged 5 commits into from
Jul 2, 2021
Merged

Update blobs migration guide to include GenerateSas; Fix migration test bugs #22380

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2e1d4c2
Update migration guide to include GenerateSas; Fix migrationt test bugs
amnguye Jul 1, 2021
847eec6
One more fix
amnguye Jul 1, 2021
51800cd
PR Comments - make generateSas the more preferred pattern to create a…
amnguye Jul 1, 2021
4601e2e
Add snippet for using basic GenerateSas
amnguye Jul 2, 2021
2a9733b
Merge branch 'main' into feature/storage/migration-sas
amnguye Jul 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 40 additions & 11 deletions sdk/storage/Azure.Storage.Blobs/AzureStorageNetMigrationV12.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ v12
The new library only supports constructing a client with a fully constructed SAS URI. Note that since client URIs are immutable once created, a new client instance with a new SAS must be created in order to rotate a SAS.

```C# Snippet:SampleSnippetsBlobMigration_SasUri
BlobClient blob = new BlobClient(new Uri(blobLocationWithSas));
BlobClient blob = new BlobClient(sasUri);
```

#### Connection string
Expand Down Expand Up @@ -446,7 +446,7 @@ v12

v12 has explicit methods for listing by hierarchy.
```C# Snippet:SampleSnippetsBlobMigration_ListHierarchy
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix);
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix, delimiter: delimiter);
await foreach (BlobHierarchyItem item in results)
{
MyConsumeBlobItemFunc(item);
Expand Down Expand Up @@ -546,19 +546,50 @@ sasBlobToken = blob.GetSharedAccessSignature(null, policyName);

v12

The modern SDK uses a builder pattern for constructing a SAS token. Clients are not involved in the process.
The modern SDK uses a builder pattern for constructing a SAS token. Similar to the pattern to create a SAS in v11, you can generate a SAS URI from the client. This is the preferred method in order to prevent passing the key to more than one place. It also is more convenient to generate from the client in the case of authenticating with a connection string, as you would not have to parse the connection string to grab the storage account name and key.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should first promote client.GenerateSas that doesn't even take builder but just permissions and date.


To create a simple SAS with any optional parameters, use the convenience overload of GenerateSas which only requires taking in permissions and the expiry time.

```C# Snippet:SampleSnippetsBlobMigration_GenerateSas
// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);

Uri sasUri;
// Ensure our client has the credentials required to generate a SAS
if (blobClient.CanGenerateSasUri)
{
// Create full, self-authenticating URI to the resource from the BlobClient
sasUri = blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1));

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
```

To create a more complex SAS pass the SAS builder to the GenerateSas method.

```C# Snippet:SampleSnippetsBlobMigration_GenerateSas_Builder
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// Since we are generating from the client, the client will have the container and blob name
// Specify any optional paremeters here
StartsOn = DateTimeOffset.UtcNow.AddHours(-1)
};

// Create full, self-authenticating URI to the resource from the BlobClient
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);
```

You can also generate a SAS without use of the client.

```C# Snippet:SampleSnippetsBlobMigration_SasBuilder
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// with no url in a client to read from, container and blob name must be provided if applicable
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
BlobName = blobName
};
// permissions applied separately, using the appropriate enum to the scope of your SAS
sasBuilder.SetPermissions(BlobSasPermissions.Read);

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
Expand All @@ -574,10 +605,8 @@ If using a stored access policy, construct your `BlobSasBuilder` from the exampl

```C# Snippet:SampleSnippetsBlobMigration_SasBuilderIdentifier
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
Identifier = "mysignedidentifier"
};
```
Expand Down
199 changes: 142 additions & 57 deletions sdk/storage/Azure.Storage.Blobs/samples/Sample03_Migrations.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,27 +55,14 @@ public void AuthWithSasCredential()
try
{
container.Create();
container.GetBlobClient(blobName).Upload(BinaryData.FromString("hello world"));
BlobClient blobClient = container.GetBlobClient(blobName);
blobClient.Upload(BinaryData.FromString("hello world"));

// build SAS URI for sample
BlobSasBuilder sas = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
};
sas.SetPermissions(BlobSasPermissions.All);

StorageSharedKeyCredential credential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

UriBuilder sasUri = new UriBuilder(this.StorageAccountBlobUri);
sasUri.Path = $"{containerName}/{blobName}";
sasUri.Query = sas.ToSasQueryParameters(credential).ToString();

string blobLocationWithSas = sasUri.Uri.ToString();
Uri sasUri = blobClient.GenerateSasUri(BlobSasPermissions.All, DateTimeOffset.UtcNow.AddHours(1));

#region Snippet:SampleSnippetsBlobMigration_SasUri
BlobClient blob = new BlobClient(new Uri(blobLocationWithSas));
BlobClient blob = new BlobClient(sasUri);
#endregion

var stream = new MemoryStream();
Expand Down Expand Up @@ -584,17 +571,18 @@ void MyConsumeBlobItemFunc(BlobHierarchyItem item)

// show in snippet where the prefix goes, but our test doesn't want a prefix for its data set
string blobPrefix = null;
string delimiter = "/";

#region Snippet:SampleSnippetsBlobMigration_ListHierarchy
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix);
IAsyncEnumerable<BlobHierarchyItem> results = containerClient.GetBlobsByHierarchyAsync(prefix: blobPrefix, delimiter: delimiter);
await foreach (BlobHierarchyItem item in results)
{
MyConsumeBlobItemFunc(item);
}
#endregion

Assert.IsTrue(expectedBlobNamesResult.SetEquals(downloadedBlobNames));
Assert.IsTrue(new HashSet<string> { virtualDirName }.SetEquals(downloadedPrefixNames));
Assert.IsTrue(new HashSet<string> { virtualDirName + '/' }.SetEquals(downloadedPrefixNames));
}
finally
{
Expand Down Expand Up @@ -700,15 +688,12 @@ public async Task SasBuilder()

#region Snippet:SampleSnippetsBlobMigration_SasBuilder
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// with no url in a client to read from, container and blob name must be provided if applicable
BlobContainerName = containerName,
BlobName = blobName,
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
BlobName = blobName
};
// permissions applied separately, using the appropriate enum to the scope of your SAS
sasBuilder.SetPermissions(BlobSasPermissions.Read);

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
Expand All @@ -729,13 +714,103 @@ public async Task SasBuilder()
}
}

[Test]
public async Task GenerateSas()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
var container = new BlobContainerClient(ConnectionString, containerName);
BlobUriBuilder uriBuilder = new BlobUriBuilder(container.Uri) { BlobName = blobName };
Uri blobUri = uriBuilder.ToUri();

try
{
await container.CreateAsync();
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

#region Snippet:SampleSnippetsBlobMigration_GenerateSas
// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);

Uri sasUri;
// Ensure our client has the credentials required to generate a SAS
if (blobClient.CanGenerateSasUri)
{
// Create full, self-authenticating URI to the resource from the BlobClient
sasUri = blobClient.GenerateSasUri(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1));

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
#endregion
else
{
Assert.Fail("Unable to create SAS URI");
}
}
finally
{
await container.DeleteIfExistsAsync();
}
}

[Test]
public async Task GenerateSas_Builder()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
var container = new BlobContainerClient(ConnectionString, containerName);
BlobUriBuilder uriBuilder = new BlobUriBuilder(container.Uri) { BlobName = blobName };
Uri blobUri = uriBuilder.ToUri();

try
{
await container.CreateAsync();
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

// Create a BlobClient with a shared key credential
BlobClient blobClient = new BlobClient(blobUri, sharedKeyCredential);
// Create BlobSasBuilder and specify parameters
#region Snippet:SampleSnippetsBlobMigration_GenerateSas_Builder
BlobSasBuilder sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddHours(1))
{
// Since we are generating from the client, the client will have the container and blob name
// Specify any optional paremeters here
StartsOn = DateTimeOffset.UtcNow.AddHours(-1)
};

// Create full, self-authenticating URI to the resource from the BlobClient
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);
#endregion

// Use newly made as SAS URI to download the blob
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
}
finally
{
await container.DeleteIfExistsAsync();
}
}

[Test]
public async Task SasBuilderIdentifier()
{
string accountName = StorageAccountName;
string accountKey = StorageAccountKey;
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-blob");
DateTimeOffset expiresOn = DateTimeOffset.UtcNow.AddDays(1);
DateTimeOffset startsOn = DateTimeOffset.UtcNow.AddHours(-1);
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(StorageAccountName, StorageAccountKey);

// setup blob
Expand All @@ -744,6 +819,7 @@ public async Task SasBuilderIdentifier()
try
{
await container.CreateAsync();
BlobClient blobClient = container.GetBlobClient(blobName);
await container.GetBlobClient(blobName).UploadAsync(BinaryData.FromString("hello world"));

// Create one or more stored access policies.
Expand All @@ -754,8 +830,8 @@ public async Task SasBuilderIdentifier()
Id = "mysignedidentifier",
AccessPolicy = new BlobAccessPolicy
{
StartsOn = DateTimeOffset.UtcNow.AddHours(-1),
ExpiresOn = DateTimeOffset.UtcNow.AddDays(1),
StartsOn = startsOn,
ExpiresOn = expiresOn,
Permissions = "rw"
}
}
Expand All @@ -765,22 +841,14 @@ public async Task SasBuilderIdentifier()

#region Snippet:SampleSnippetsBlobMigration_SasBuilderIdentifier
// Create BlobSasBuilder and specify parameters
BlobSasBuilder sasBuilder = new BlobSasBuilder()
BlobSasBuilder sasBuilder = new BlobSasBuilder
{
BlobContainerName = containerName,
BlobName = blobName,
Identifier = "mysignedidentifier"
};
#endregion

// Create full, self-authenticating URI to the resource
BlobUriBuilder uriBuilder = new BlobUriBuilder(StorageAccountBlobUri)
{
BlobContainerName = containerName,
BlobName = blobName,
Sas = sasBuilder.ToSasQueryParameters(sharedKeyCredential)
};
Uri sasUri = uriBuilder.ToUri();
Uri sasUri = blobClient.GenerateSasUri(sasBuilder);

// successful download indicates pass
await new BlobClient(sasUri).DownloadToAsync(new MemoryStream());
Expand Down Expand Up @@ -926,18 +994,26 @@ public async Task RetryPolicy()
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-file");
var containerClient = new BlobContainerClient(ConnectionString, containerName);
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_RetryPolicy
BlobClientOptions blobClientOptions = new BlobClientOptions();
blobClientOptions.Retry.Mode = RetryMode.Exponential;
blobClientOptions.Retry.Delay = TimeSpan.FromSeconds(10);
blobClientOptions.Retry.MaxRetries = 6;
BlobServiceClient service = new BlobServiceClient(connectionString, blobClientOptions);
BlobClient blobClient = service.GetBlobContainerClient(containerName).GetBlobClient(blobName);
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream);
#endregion
try
{
await containerClient.CreateIfNotExistsAsync();
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_RetryPolicy
BlobClientOptions blobClientOptions = new BlobClientOptions();
blobClientOptions.Retry.Mode = RetryMode.Exponential;
blobClientOptions.Retry.Delay = TimeSpan.FromSeconds(10);
blobClientOptions.Retry.MaxRetries = 6;
BlobServiceClient service = new BlobServiceClient(connectionString, blobClientOptions);
BlobClient blobClient = service.GetBlobContainerClient(containerName).GetBlobClient(blobName);
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream);
#endregion
}
finally
{
await containerClient.DeleteIfExistsAsync();
}

Assert.Pass();
}
Expand All @@ -953,15 +1029,24 @@ public async Task MaximumExecutionTime()
string containerName = Randomize("sample-container");
string blobName = Randomize("sample-file");
var containerClient = new BlobContainerClient(ConnectionString, containerName);
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_MaximumExecutionTime
BlobClient blobClient = containerClient.GetBlobClient(blobName);
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30));
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream, cancellationTokenSource.Token);
#endregion

try
{
await containerClient.CreateIfNotExistsAsync();
await containerClient.GetBlobClient(blobName).UploadAsync(BinaryData.FromString(data));

#region Snippet:SampleSnippetsBlobMigration_MaximumExecutionTime
BlobClient blobClient = containerClient.GetBlobClient(blobName);
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30));
Stream targetStream = new MemoryStream();
await blobClient.DownloadToAsync(targetStream, cancellationTokenSource.Token);
#endregion
}
finally
{
await containerClient.DeleteIfExistsAsync();
}

Assert.Pass();
}
Expand Down