Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UPLOADED state to Zarr models #1698

Merged
merged 2 commits into from
Nov 7, 2023
Merged

Add UPLOADED state to Zarr models #1698

merged 2 commits into from
Nov 7, 2023

Conversation

jjnesbitt
Copy link
Member

@jjnesbitt jjnesbitt commented Oct 4, 2023

This is a breaking API change.

This adds an UPLOADED state to the Zarr models, to delineate between when a user calls "finalize", and when the zarr begins ingestion.

The use of the Zarr status is now as follows:

  1. User creates a Zarr, it defaults to PENDING
  2. User uploads data to a zarr
  3. User is done uploading data and calls the finalize endpoint, placing the Zarr in an UPLOADED state and kicking off the ingestion task
  4. The ingestion task is begun on that zarr, placing it in an INGESTING state
  5. Ingestion is finished, placing it in a COMPLETE state
  6. If the User wants to upload more data, they request files from the Zarr file creation endpoint, placing the Zarr into a PENDING state
  7. Go-to bullet 2

@jwodder The changes regarding the CLI are as follows:

  1. When calling finalize on a zarr, the returned status of the zarr will now be UPLOADED, instead of PENDING
  2. The zarr file create and delete endpoints will now return the ZarrArchive.INGEST_ERROR_MSG if the zarr in question is in either the UPLOADED or INGESTING state
  3. The ZarrArchive.INGEST_ERROR_MSG has been changed from "Zarr archive already ingested" to "Zarr archive is currently ingesting or has already ingested"

Copy link
Member

@waxlamp waxlamp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you (in the PR description) sketch out the intended progression through the different statuses after this change?

@yarikoptic
Copy link
Member

We returned dandi-cli into green state again. Do you think you @AlmightyYakob or @danlamanna can workout a needed patch for it for this API change? would be appreciated. If not -- that is ok, just let me know.

@jjnesbitt
Copy link
Member Author

We returned dandi-cli into green state again. Do you think you @AlmightyYakob or @danlamanna can workout a needed patch for it for this API change? would be appreciated. If not -- that is ok, just let me know.

It seems no change is needed to the CLI, since it awaits the COMPLETE state to ensure there is a valid checksum.

@jjnesbitt jjnesbitt added patch Increment the patch version when merged release Create a release when this pr is merged labels Nov 6, 2023
@jjnesbitt jjnesbitt merged commit c875d22 into master Nov 7, 2023
10 checks passed
@jjnesbitt jjnesbitt deleted the zarr-uploaded-status branch November 7, 2023 17:54
@dandibot
Copy link
Member

dandibot commented Nov 7, 2023

🚀 PR was released in v0.3.60 🚀

@dandibot dandibot added the released This issue/pull request has been released. label Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch Increment the patch version when merged release Create a release when this pr is merged released This issue/pull request has been released.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants