feat: Add "transfer_manager" module for concurrent uploads and downlo…

…ads, as a preview feature (#943) * checkpoint before design doc impl * checkpoint * more tests * code and tests for transfer manager complete * proactively close temp files when finished reading * respond to comments; destroy tmp files as they are consumed * Add system tests, docstrings, address feedback * Respond to review comments * verify md5 hash of downloaded file in test * lint * default empty strings for root arguments * fix bug with blob constructor * add warning about files not being deleted if their downloads fail * docs: Add samples to multithread branch (#918) * add samples, tests pending * add snippet tests * snippet and snippets_test.py linting * snippets; recursive directory creation; rename some params * Add directory upload snippet * fix: remove chunked downloads; change max_workers to threads * update snippets to add thread info * fix snippets test issue due to change in dependency * snippet nomenclature * fix samples for real this time
googleapis · Dec 6, 2022 · 9998a5e · 9998a5e
1 parent 1ca0233
commit 9998a5e
Show file tree

Hide file tree

Showing 7 changed files with 1,210 additions and 19 deletions.
diff --git a/google/cloud/storage/constants.py b/google/cloud/storage/constants.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 """Constants used across google.cloud.storage modules."""
 
 # Storage classes

diff --git a/google/cloud/storage/fileio.py b/google/cloud/storage/fileio.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Support for file-like I/O."""
+"""Module for file-like access of blobs, usually invoked via Blob.open()."""
 
 import io
 import warnings
@@ -101,10 +101,12 @@ class BlobReader(io.BufferedIOBase):
         - ``if_metageneration_match``
         - ``if_metageneration_not_match``
         - ``timeout``
+
+        Note that download_kwargs are also applied to blob.reload(), if a reload
+        is needed during seek().
     """
 
     def __init__(self, blob, chunk_size=None, retry=DEFAULT_RETRY, **download_kwargs):
-        """docstring note that download_kwargs also used for reload()"""
         for kwarg in download_kwargs:
             if kwarg not in VALID_DOWNLOAD_KWARGS:
                 raise ValueError(