Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OPT+RF of zarr downloads: do not wait for full files listing + compute %done from total zarr size #1443

Merged
merged 4 commits into from
May 30, 2024

Commits on May 14, 2024

  1. RF: do not require fetching all files for a zarr asset to start initi…

    …ating downloads
    
    Delay assignment of file_qty until it is known, but otherwise proceed to
    initiating downloads
    yarikoptic committed May 14, 2024
    Configuration menu
    Copy the full SHA
    c0df9af View commit details
    Browse the repository at this point in the history
  2. BF: use total zarr_size to compute done% for zarr

    Since maxsize is dynamically computed as we go through the files.
    The idea, I guess, was that it would grow rapidly before actual
    downloads commense but it is not the case, so we endup with done%
    being always close to 100% since we get those reports on final
    downloads completed close to when individual files are downloaded.
    
    So this should close #1407 .
    
    But for total zarr file to be used, we needed to account also for
    skipped files.  I added reporting of sizes for skipped files as well.
    It seems there is no negative side effect on regular files download.
    So now for the %done of zarr we might be getting to 100% of original
    size having downloaded nothing.  But IMHO it is ok since user does
    not care as much of how many "subparts" are downloaded, but rather
    to have adequate progress report back.
    
    There also could be side effects if  -e skip  and we skip
    download of some updated files which would be smaller than the local
    ones so altogether we would get over 100% total at the end.
    yarikoptic committed May 14, 2024
    Configuration menu
    Copy the full SHA
    743d034 View commit details
    Browse the repository at this point in the history
  3. Fix up test for the changes in prior commit + robustify few places in…

    … the modified code logic
    yarikoptic committed May 14, 2024
    Configuration menu
    Copy the full SHA
    b17aa63 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Code review: minor typing etc recommendations

    Co-authored-by: John T. Wodder II <jwodder@users.noreply.github.com>
    yarikoptic and jwodder authored May 20, 2024
    Configuration menu
    Copy the full SHA
    08a4050 View commit details
    Browse the repository at this point in the history