Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FileExistsError after restarting Zarr download #1473

Closed
kabilar opened this issue Jul 30, 2024 · 2 comments
Closed

FileExistsError after restarting Zarr download #1473

kabilar opened this issue Jul 30, 2024 · 2 comments
Labels
zarr

Comments

@kabilar
Copy link
Member

kabilar commented Jul 30, 2024

After restarting the download of a Zarr archive I receive a FileExistsError.

Steps to replicate:

  1. Start initial download (This may a while due to Make zarr download begin as soon as initial page of zarr files is obtained #1447)

    dandi download https://api.dandiarchive.org/api/assets/9c225928-114b-4a8b-ba33-df1e9c9b8946/download/

  2. Interrupt download (control+C a few times).

  3. Restart download by running command in step 1.

    PATH                                   SIZE      DONE                DONE% CHECKSUM STATUS        MESSAGE          
...8_sample-01_chunk-03_hipCT.ome.zarr 548.7 GB  714.5 MB               0%          error         FileExistsError  
Summary:                               548.7 GB  714.5 MB                           1 error       1 FileExistsError
                                                 0.13%                                                             
                                                 ETA: 2 days
Tail of error log 
...
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/50 has correct md5 4bf6d41410bf4b3165b25a22531c6683
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6185463808 Starting new download in new download directory
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/56&download=true
2024-07-30T13:19:24-0500 [INFO    ] dandi 87089:6168637440 File 0/0/21/55 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/55 has correct md5 3d1babd7e4f77a523682e2601fc6f272
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6168637440 Starting new download in new download directory
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/57&download=true
2024-07-30T13:19:24-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F10%2F13 HTTP/1.1" 200 119248
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:24-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/56&download=true HTTP/1.1" 302 0
2024-07-30T13:19:24-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F35%2F7
2024-07-30T13:19:24-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/57&download=true HTTP/1.1" 302 0
2024-07-30T13:19:25-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181924Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e942c1c5e48203ad8e0baa15e4a02f078e607260b2028533a53b3ec7f816568d HTTP/1.1" 200 2673285
2024-07-30T13:19:25-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/57?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181924Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=5969a678de7856ab4d02fe531e33ef9d4eb6791332849d2f6d35f65ba0c20d2c HTTP/1.1" 200 2676335
2024-07-30T13:19:25-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F35%2F7 HTTP/1.1" 200 119128
2024-07-30T13:19:25-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/52 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/52 has correct md5 d5da93a8b524fadecbf969c0d469953a
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6151811072 Starting new download in new download directory
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/58&download=true
2024-07-30T13:19:25-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/58&download=true HTTP/1.1" 302 0
2024-07-30T13:19:25-0500 [INFO    ] dandi 87089:6185463808 File 0/0/21/56 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/56 has correct md5 b92bdd5ad8fec1f42c02fd39087200d7
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6185463808 Starting new download in new download directory
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/59&download=true
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:25-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F5%2F28%2F31
2024-07-30T13:19:25-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/59&download=true HTTP/1.1" 302 0
2024-07-30T13:19:26-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181925Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=ac3eb5c395d537378a60238804259a3ff4b87e1b17805fdfef2cb85127cd894d HTTP/1.1" 200 2686299
2024-07-30T13:19:26-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/59?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181925Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=9a73d9b40629fca7852d28321afa4c7157f6cd4164656c542a3ad3eac80b8a9c HTTP/1.1" 200 2674468
2024-07-30T13:19:26-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F5%2F28%2F31 HTTP/1.1" 200 118730
2024-07-30T13:19:26-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:26-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F6%2F20%2F24
2024-07-30T13:19:26-0500 [INFO    ] dandi 87089:6185463808 File 0/0/21/59 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:26-0500 [DEBUG   ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/59 has correct md5 904f4f16663f28332e3d8e2a8163535d
2024-07-30T13:19:26-0500 [DEBUG   ] dandi 87089:6185463808 Starting new download in new download directory
2024-07-30T13:19:26-0500 [DEBUG   ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/6&download=true
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/6&download=true HTTP/1.1" 302 0
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181926Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=9941ba5054930e984204ecdfec08515e7d0e0d36113e11422c6b70e073e5c363 HTTP/1.1" 200 2613746
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F6%2F20%2F24 HTTP/1.1" 200 118773
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F13%2F17
2024-07-30T13:19:27-0500 [INFO    ] dandi 87089:6185463808 File 0/0/21/6 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/6 has correct md5 b94968e81ae75e8f143da5b5740be73e
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6185463808 Starting new download in new download directory
2024-07-30T13:19:27-0500 [INFO    ] dandi 87089:6134984704 File 0/0/21/54 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/60&download=true
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/54 has correct md5 d84164ebd71daa9a0ad5735a50e3e0d8
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6134984704 Starting new download in new download directory
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/61&download=true
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F13%2F17 HTTP/1.1" 200 119042
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/60&download=true HTTP/1.1" 302 0
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:27-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/61&download=true HTTP/1.1" 302 0
2024-07-30T13:19:27-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F7%2F1
2024-07-30T13:19:28-0500 [DEBUG   ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181927Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=50de1e9f0200ecdb414aa1e5f4b979fa46b6cca1d966341b473eb06a88e7efb9 HTTP/1.1" 200 2681247
2024-07-30T13:19:28-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/61?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181927Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=4bd042babf66bc9ad2f961414c74e6449813d688215637fa5370f7458314a7d5 HTTP/1.1" 200 2656592
2024-07-30T13:19:28-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F7%2F1 HTTP/1.1" 200 119199
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F8%2F30%2F35
2024-07-30T13:19:28-0500 [INFO    ] dandi 87089:6168637440 File 0/0/21/57 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:28-0500 [INFO    ] dandi 87089:6134984704 File 0/0/21/61 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/61 has correct md5 a95b9e4c5915bbdf37867fa2a8e8752a
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/57 has correct md5 169080f9518ba092676cdbd0279ce57f
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6168637440 Starting new download in new download directory
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6134984704 Starting new download in new download directory
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/63&download=true
2024-07-30T13:19:28-0500 [DEBUG   ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/62&download=true
2024-07-30T13:19:29-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/63&download=true HTTP/1.1" 302 0
2024-07-30T13:19:29-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/62&download=true HTTP/1.1" 302 0
2024-07-30T13:19:29-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/62?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181928Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=58acbd691b3fd92d46d6d131e52fed9a69f80c29a5d06b55ca97e5339d4d225e HTTP/1.1" 200 2657730
2024-07-30T13:19:29-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/63?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181928Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=dce88a88655339997bf1ce16a1fc02233dacdfc5e86e6fdde21cf647d4b8ea4d HTTP/1.1" 200 2678755
2024-07-30T13:19:29-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/58 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/58 has correct md5 f2d7adb84b3f2abae4c9ec1e21d5a5df
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6151811072 Starting new download in new download directory
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/64&download=true
2024-07-30T13:19:29-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/64&download=true HTTP/1.1" 302 0
2024-07-30T13:19:29-0500 [INFO    ] dandi 87089:6168637440 File 0/0/21/63 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/63 has correct md5 b62031d4affb8ad34be5aae93dae7ce1
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6168637440 Starting new download in new download directory
2024-07-30T13:19:29-0500 [DEBUG   ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/65&download=true
2024-07-30T13:19:30-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F8%2F30%2F35 HTTP/1.1" 200 118872
2024-07-30T13:19:30-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/65&download=true HTTP/1.1" 302 0
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F9%2F23%2F28
2024-07-30T13:19:30-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/64?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181929Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=364511498060605ef00b5ef99af92730cfe29b6b33668acf1dad857047067a20 HTTP/1.1" 200 2725934
2024-07-30T13:19:30-0500 [DEBUG   ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/65?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181930Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e216335107232e0f960675278ad2e56be8e9d9adb3352c9424f9f12cfebabafe HTTP/1.1" 200 2668276
2024-07-30T13:19:30-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F9%2F23%2F28 HTTP/1.1" 200 118601
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F0%2F8%2F4
2024-07-30T13:19:30-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/64 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/64 has correct md5 8ad14ae58529eca307d31f2a8a129768
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6151811072 Starting new download in new download directory
2024-07-30T13:19:30-0500 [DEBUG   ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/66&download=true
2024-07-30T13:19:31-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/66&download=true HTTP/1.1" 302 0
2024-07-30T13:19:31-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F0%2F8%2F4 HTTP/1.1" 200 118901
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F2%2F0%2F12
2024-07-30T13:19:31-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/66?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181930Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e15aeac9910655184d3c83ecf471a9fb468a0f51f08d88dc81f785fc568eb9a8 HTTP/1.1" 200 2660776
2024-07-30T13:19:31-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F2%2F0%2F12 HTTP/1.1" 200 118489
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F5%2F1%2F6
2024-07-30T13:19:31-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/66 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/66 has correct md5 2b15a1dbfa01eff8b9c15aac257781cb
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6151811072 Starting new download in new download directory
2024-07-30T13:19:31-0500 [DEBUG   ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/67&download=true
2024-07-30T13:19:32-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/67&download=true HTTP/1.1" 302 0
2024-07-30T13:19:32-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F5%2F1%2F6 HTTP/1.1" 200 118561
2024-07-30T13:19:32-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/67?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181931Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=0ac17568c826a86953ae1c251484c57dd50d0b376f26dfc5bc3f87c0cda0c011 HTTP/1.1" 200 2655398
2024-07-30T13:19:32-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:32-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F8%2F11%2F14
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F8%2F11%2F14 HTTP/1.1" 200 118273
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:33-0500 [INFO    ] dandi 87089:6134984704 File 0/0/21/62 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:33-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/67 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=3%2F5%2F0%2F6
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/62 has correct md5 c368c5d8eab19ba140443a3992f36359
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/67 has correct md5 ed8176f25a2f2748b8228fcbe5e1edb5
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6134984704 Starting new download in new download directory
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6151811072 Starting new download in new download directory
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/68&download=true
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/69&download=true
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/68&download=true HTTP/1.1" 302 0
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/69&download=true HTTP/1.1" 302 0
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=3%2F5%2F0%2F6 HTTP/1.1" 200 17657
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6112997376 Response: 200
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/69?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181933Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=5a8dbcbdec5895f7a48170d17e99ecb5f2078a55f3804bd355ce9521c1f17529 HTTP/1.1" 200 46969
2024-07-30T13:19:33-0500 [DEBUG   ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/68?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181933Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=f13e1927a0ff0d3296a9473e857eefa17392a52a446991391a1ae8ad46be92ce HTTP/1.1" 200 500088
2024-07-30T13:19:33-0500 [INFO    ] dandi 87089:6151811072 File 0/0/21/69 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:33-0500 [INFO    ] dandi 87089:6168637440 File 0/0/21/65 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/69 has correct md5 add4f234b32b690e4b59eaa2e3af3f3e
2024-07-30T13:19:33-0500 [DEBUG   ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/65 has correct md5 9a7ae1fca026f6fd9c6ffedbb8a6e05a
2024-07-30T13:19:34-0500 [INFO    ] dandi 87089:6134984704 File 0/0/21/68 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:34-0500 [DEBUG   ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/68 has correct md5 120b2982407d875633da290ea7119418
2024-07-30T13:19:35-0500 [INFO    ] dandi 87089:6185463808 File 0/0/21/60 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded
2024-07-30T13:19:35-0500 [DEBUG   ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/60 has correct md5 63e9b19ddb0ccd436ae221dd5c2465bb
2024-07-30T13:19:44-0500 [ERROR   ] dandi 87089:6112997376 Caught while downloading sub-I58_sample-01_chunk-03_hipCT.ome.zarr:
Traceback (most recent call last):
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 360, in _download_generator_guard
    yield from generator
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 918, in _download_zarr
    for path, status in it:
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 346, in __next__
    return self.get()
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 380, in get
    self._end()
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 400, in _end
    e.get()
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 110, in get
    raise e.with_traceback(tb) from None
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 274, in _process
    x = next(it)
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 981, in pairing
    for d in gen:
  File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 574, in _download_file
    raise FileExistsError(f"File {path!r} already exists")
FileExistsError: File PosixPath('sub-I58_sample-01_chunk-03_hipCT.ome.zarr/.zattrs') already exists
2024-07-30T13:19:44-0500 [INFO    ] dandi 87089:8265534464 Logs saved in /Users/kabilar/Library/Logs/dandi-cli/2024.07.30-18.14.56Z-87089.log

Thank you.

cc @satra @aaronkanzer
@kabilar kabilar added the zarr label Jul 30, 2024
@yarikoptic
Copy link
Member

behaves as expected:

❯ dandi download --help | grep existing
     dandi download https://identifiers.org/DANDI:000027 --existing refresh
  -e, --existing [error|skip|overwrite|overwrite-different|refresh]
                                  What to do if a file found existing locally.

@kabilar
Copy link
Member Author

kabilar commented Jul 30, 2024

Thanks Yarik. Apologize for missing this option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
zarr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yarikoptic @kabilar