Merge pull request Azure#399 from rickle-msft/dev_breaking

Dev breaking

BreakingChanges.txt

@@ -8,6 +8,7 @@
 * Removed RetryReader type as it's functionality was moved to be built into the DownloadResponse. RetryReaderOptions are now named ReliableDownloadOptions.
 * Restructured the access conditions to be more logically adhere to their respective functions.
 * Added support for context parameter on each api to allow communication with the pipeline from the application level
+* Added an option to configure that maximum size data that will be uploaded in a single shot via the TransferManager.
 
 2018.08.22 Version 10.0.4-rc
 * Changed BlobURL.startCopy sourceAccessConditions parameter to be HTTPAccessConditions as lease is not actually supported.

CONTRIBUTING.md

@@ -20,7 +20,11 @@ The Azure Storage development team uses Intellij. However, any preferred IDE or
 ## Tests
 
 ### Configuration
-The only step to configure testing is to set the appropriate environment variable. Create environment variables named "ACCOUNT_NAME", "ACCOUNT_KEY", "SECONDARY_ACCOUNT_NAME", and "SECONDARY_ACCOUNT_KEY". The first two will be used for most requests. The second two will only be used for tests requiring two accounts.
+The only step to configure testing is to set the appropriate environment variables. Create environment variables named "ACCOUNT_NAME" and "ACCOUNT_KEY", holding your Azure storage account name and key respectively. This will satisfy most tests. 
+To run any tests requiring two accounts (generally those testing copy-related apis), set environment variables "SECONDARY_ACCOUNT_NAME", and "SECONDARY_ACCOUNT_KEY".
+To run any tests related to setting blob tiers on block blobs, set environment variables "BLOB_STORAGE_ACCOUNT_NAME" and "BLOB_STORAGE_ACCOUNT_KEY". Note that a GPV2 account is also sufficient here.
+To run any tests related to setting blob tiers on page blobs, set environment variables "PREMIUM_ACCOUNT_NAME" and "PREMIUM_ACCOUNT_KEY".
+It is valid to use a single account for multiple scenarios; a GPV2 account would work for both the primary account and the blob storage account, for instance. The only restriction is that the primary and secondary accounts must be distinct.
 
 ### Running
 To actually run tests, right click on the test class in the Package Explorer or the individual test in the Outline and select Run As->GroovyTest. Alternatively, run mvn test from the command line.

ChangeLog.txt

@@ -1,10 +1,18 @@
 XXXX.XX.XX Version XX.X.X
-* Added overloads which only accept the required parameters.
-* TransferManager options now throw if an IProgressReceiver is passed as it is not currently supported. Support will be added in a later release.
-* Added CopyFromURL, which will do a synchronous server-side copy, meaning the service will not return an HTTP response until it has completed the copy.
 * HTTPGetterInfo was made an internal type as it is an internal implementation detail.
 * DownloadResponse constructor was made internal as there is no need for customers to construct their own responses; all HTTP responses should be generated internally.
 * Removed DEFAULT and NONE static variables. Empty constructors should be used instead. DEFAULT static values were error prone and unsafe to use because although the field was final, the objects were mutable, so it was possible the value could be changed accidentally and alter the behavior of the program.
+* Optimized the TransferManager download to file method to skip the initial HEAD request.
+* Added an option to configure that maximum size data that will be uploaded in a single shot via the TransferManager.
+* Added request Http Method, URL, and headers to logging messages.
+
+2018.10.29 Version 10.2.0
+* Added overloads which only accept the required parameters.
+* Added CopyFromURL, which will do a synchronous server-side copy, meaning the service will not return an HTTP response until it has completed the copy.
+* Added support for IProgressReceiver in TransferManager operations. This parameter was previously ignored but is now supported.
+* Removed internal dependency on javafx to be compatible with openjdk.
+* Fixed a bug that would cause downloading large files with the TransferManager to fail.
+* Fixed a bug in BlobURL.download() logic for setting up reliable download. This had the potential to download the wrong range when a download stream was retried.
 
 2018.09.11 Version 10.1.0
 * Interfaces for helper types updated to be more consistent throughout the library. All types, with the exception of the options for pipeline factories, use a fluent pattern.

ISSUE_TEMPLATE.md

@@ -0,0 +1,10 @@
+### Which service(blob, file, queue, table) does this issue concern?
+
+
+### Which version of the SDK was used?
+
+
+### What problem was encountered?
+
+
+### Have you found a mitigation/solution?

README.md

@@ -31,7 +31,7 @@ To get the binaries of this library as distributed by Microsoft, ready for use w
 <dependency>
 	<groupId>com.microsoft.azure</groupId>
 	<artifactId>azure-storage-blob</artifactId>
-	<version>10.1.0</version>
+	<version>10.2.0</version>
 </dependency>
 ```
 

pom.xml

@@ -14,7 +14,7 @@
 
     <groupId>com.microsoft.azure</groupId>
     <artifactId>azure-storage-blob</artifactId>
-    <version>10.1.0</version>
+    <version>10.2.0</version>
 
     <name>Azure Storage Blob</name>
     <description>The Azure Storage Java Blob library.</description>
@@ -61,19 +61,6 @@
         </pluginRepository>
     </pluginRepositories>
 
-    <repositories>
-        <repository>
-            <id>ossrh</id>
-            <name>Sonatype Snapshots</name>
-            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
-            <layout>default</layout>
-            <snapshots>
-                <enabled>true</enabled>
-                <updatePolicy>always</updatePolicy>
-            </snapshots>
-        </repository>
-    </repositories>
-
     <dependencies>
         <dependency>
             <groupId>com.microsoft.rest.v2</groupId>

src/main/java/com/microsoft/azure/storage/blob/BlobURL.java

@@ -322,26 +322,6 @@ public Single<DownloadResponse> download() {
         return this.download(null, null, false, null);
     }
 
-    /**
-     * Reads a range of bytes from a blob. The response also includes the blob's properties and metadata. For more
-     * information, see the <a href="https://docs.microsoft.com/rest/api/storageservices/get-blob">Azure Docs</a>.
-     * <p>
-     * Note that the response body has reliable download functionality built in, meaning that a failed download stream
-     * will be automatically retried. This behavior may be configured with {@link ReliableDownloadOptions}.
-     *
-     * @param range
-     *         {@link BlobRange}
-     *
-     * @return Emits the successful response.
-     *
-     * @apiNote ## Sample Code \n
-     * [!code-java[Sample_Code](../azure-storage-java/src/test/java/com/microsoft/azure/storage/Samples.java?name=upload_download "Sample code for BlobURL.download")] \n
-     * For more samples, please see the [Samples file](%https://github.com/Azure/azure-storage-java/blob/master/src/test/java/com/microsoft/azure/storage/Samples.java)
-     */
-    public Single<DownloadResponse> download(BlobRange range) {
-        return this.download(range, null, false, null);
-    }
-
     /**
      * Reads a range of bytes from a blob. The response also includes the blob's properties and metadata. For more
      * information, see the <a href="https://docs.microsoft.com/rest/api/storageservices/get-blob">Azure Docs</a>.
@@ -374,7 +354,7 @@ public Single<DownloadResponse> download(BlobRange range, BlobAccessConditions a
         range = range == null ? new BlobRange() : range;
         accessConditions = accessConditions == null ? new BlobAccessConditions() : accessConditions;
         HTTPGetterInfo info = new HTTPGetterInfo()
-                .withCount(range.offset())
+                .withOffset(range.offset())
                 .withCount(range.count())
                 .withETag(accessConditions.modifiedAccessConditions().ifMatch());
 

src/main/java/com/microsoft/azure/storage/blob/Constants.java

@@ -55,6 +55,10 @@ final class Constants {
      * The query parameter for snapshots.
      */
     static final String SNAPSHOT_QUERY_PARAMETER = "snapshot";
+    /**
+     * The word redacted.
+     */
+    static final String REDACTED = "REDACTED";
     /**
      * The default amount of parallelism for TransferManager operations.
      */
@@ -152,6 +156,16 @@ static final class HeaderConstants {
          */
         static final String RANGE_HEADER_FORMAT = "bytes=%d-%d";
 
+        /**
+         * The copy source header.
+         */
+        static final String COPY_SOURCE = "x-ms-copy-source";
+
+        /**
+         * The version header.
+         */
+        static final String VERSION = "x-ms-version";
+
         /**
          * The current storage version header value.
          */
@@ -170,10 +184,97 @@ static final class HeaderConstants {
         /**
          * Specifies the value to use for UserAgent header.
          */
-        static final String USER_AGENT_VERSION = "10.1.0";
+        static final String USER_AGENT_VERSION = "10.2.0";
 
         private HeaderConstants() {
             // Private to prevent construction.
         }
     }
+
+    static final class UrlConstants {
+
+        /**
+         * The SAS service version parameter.
+         */
+        static final String SAS_SERVICE_VERSION = "sv";
+
+        /**
+         * The SAS services parameter.
+         */
+        static final String SAS_SERVICES = "ss";
+
+        /**
+         * The SAS resource types parameter.
+         */
+        static final String SAS_RESOURCES_TYPES = "srt";
+
+        /**
+         * The SAS protocol parameter.
+         */
+        static final String SAS_PROTOCOL = "spr";
+
+        /**
+         * The SAS start time parameter.
+         */
+        static final String SAS_START_TIME = "st";
+
+        /**
+         * The SAS expiration time parameter.
+         */
+        static final String SAS_EXPIRY_TIME = "se";
+
+        /**
+         * The SAS IP range parameter.
+         */
+        static final String SAS_IP_RANGE = "sip";
+
+        /**
+         * The SAS signed identifier parameter.
+         */
+        static final String SAS_SIGNED_IDENTIFIER = "si";
+
+        /**
+         * The SAS signed resource parameter.
+         */
+        static final String SAS_SIGNED_RESOURCE = "sr";
+
+        /**
+         * The SAS signed permissions parameter.
+         */
+        static final String SAS_SIGNED_PERMISSIONS = "sp";
+
+        /**
+         * The SAS signature parameter.
+         */
+        static final String SAS_SIGNATURE = "sig";
+
+        /**
+         * The SAS cache control parameter.
+         */
+        static final String SAS_CACHE_CONTROL = "rscc";
+
+        /**
+         * The SAS content disposition parameter.
+         */
+        static final String SAS_CONTENT_DISPOSITION = "rscd";
+
+        /**
+         * The SAS content encoding parameter.
+         */
+        static final String SAS_CONTENT_ENCODING = "rsce";
+
+        /**
+         * The SAS content language parameter.
+         */
+        static final String SAS_CONTENT_LANGUAGE = "rscl";
+
+        /**
+         * The SAS content type parameter.
+         */
+        static final String SAS_CONTENT_TYPE = "rsct";
+
+        private UrlConstants() {
+            // Private to prevent construction.
+        }
+    }
 }

src/main/java/com/microsoft/azure/storage/blob/DownloadResponse.java

@@ -75,55 +75,60 @@ public Flowable<ByteBuffer> body(ReliableDownloadOptions options) {
             return this.rawResponse.body();
         }
 
-        return this.rawResponse.body()
-                /*
-                Update how much data we have received in case we need to retry and propagate to the user the data we
-                have received.
-                 */
-                .doOnNext(buffer -> {
-                    this.info.withOffset(this.info.offset() + buffer.remaining());
-                    if (info.count() != null) {
-                        this.info.withCount(this.info.count() - buffer.remaining());
-                    }
-                })
-                .onErrorResumeNext(throwable -> {
-                    // So far we have tried once but retried zero times.
-                    return tryContinueFlowable(throwable, 0, optionsReal);
-                });
+        /*
+        We pass -1 for currentRetryCount because we want tryContinueFlowable to receive a value of 0 for number of
+        retries as we have not actually retried yet, only made the initial try. Because applyReliableDownload() will
+        add 1 before calling into tryContinueFlowable, we set the initial value to -1.
+         */
+        return this.applyReliableDownload(this.rawResponse.body(), -1, optionsReal);
     }
 
     private Flowable<ByteBuffer> tryContinueFlowable(Throwable t, int retryCount, ReliableDownloadOptions options) {
         // If all the errors are exhausted, return this error to the user.
         if (retryCount > options.maxRetryRequests() || !(t instanceof IOException)) {
             return Flowable.error(t);
         } else {
+            /*
+            We wrap this in a try catch because we don't know the behavior of the getter. Most errors would probably
+            come from an unsuccessful request, which would be propagated through the onError methods. However, it is
+            possible the method call that returns a Single is what throws (like how our apis throw some exceptions at
+            call time rather than at subscription time.
+             */
             try {
                 // Get a new response and try reading from it.
                 return getter.apply(this.info)
-                        .flatMapPublisher(response -> {
-                            // Do not compound the number of retries; just get the raw body.
-                            ReliableDownloadOptions newOptions = new ReliableDownloadOptions()
-                                    .withMaxRetryRequests(0);
-
-                            return response.body(newOptions)
-                                    .doOnNext(buffer -> {
-                                        this.info.withOffset(this.info.offset() + buffer.remaining());
-                                        if (info.count() != null) {
-                                            this.info.withCount(this.info.count() - buffer.remaining());
-                                        }
-                                    })
-                                    .onErrorResumeNext(t2 -> {
-                                        // Increment the retry count and try again with the new exception.
-                                        return tryContinueFlowable(t2, retryCount + 1, options);
-                                    });
-                        });
+                        .flatMapPublisher(response ->
+                            /*
+                            Do not compound the number of retries by passing in another set of downloadOptions; just get
+                            the raw body.
+                             */
+                            this.applyReliableDownload(this.rawResponse.body(), retryCount, options));
             } catch (Exception e) {
                 // If the getter fails, return the getter failure to the user.
                 return Flowable.error(e);
             }
         }
     }
 
+    private Flowable<ByteBuffer> applyReliableDownload(Flowable<ByteBuffer> data,
+            int currentRetryCount, ReliableDownloadOptions options) {
+        return data
+                .doOnNext(buffer -> {
+                    /*
+                    Update how much data we have received in case we need to retry and propagate to the user the data we
+                    have received.
+                     */
+                    this.info.withOffset(this.info.offset() + buffer.remaining());
+                    if (this.info.count() != null) {
+                        this.info.withCount(this.info.count() - buffer.remaining());
+                    }
+                })
+                .onErrorResumeNext(t2 -> {
+                    // Increment the retry count and try again with the new exception.
+                    return tryContinueFlowable(t2, currentRetryCount + 1, options);
+                });
+    }
+
     public int statusCode() {
         return this.rawResponse.statusCode();
     }

src/main/java/com/microsoft/azure/storage/blob/HTTPGetterInfo.java

@@ -55,7 +55,7 @@ public Long count() {
      */
     public HTTPGetterInfo withCount(Long count) {
         if (count != null) {
-            Utility.assertInBounds("count", count, 0, Integer.MAX_VALUE);
+            Utility.assertInBounds("count", count, 0, Long.MAX_VALUE);
         }
         this.count = count;
         return this;

src/main/java/com/microsoft/azure/storage/blob/IProgressReceiver.java

@@ -14,10 +14,14 @@
  */
 package com.microsoft.azure.storage.blob;
 
+import io.reactivex.Flowable;
+
 /**
  * An {@code IProgressReceiver} is an object that can be used to report progress on network transfers. When specified on
  * transfer operations, the {@code reportProgress} method will be called periodically with the total number of bytes
- * transferred. The user may configure this method to report progress in whatever format desired.
+ * transferred. The user may configure this method to report progress in whatever format desired. It is recommended
+ * that this type be used in conjunction with
+ * {@link ProgressReporter#addProgressReporting(Flowable, IProgressReceiver)}.
  */
 public interface IProgressReceiver {