-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resolve all asset types being evaluated during upload validation #2062
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need a bit more explanation of the actual bug, and why this fixes it. We need to be careful with Zarr statuses because those have a complex relationship with Assets that results in (perhaps overly) complex semantics.
@@ -90,7 +92,9 @@ def version_aggregate_assets_summary(version: Version) -> None: | |||
|
|||
assets_summary = aggregate_assets_summary( | |||
asset.full_metadata | |||
for asset in version.assets.filter(status=Asset.Status.VALID) | |||
for asset in version.assets.filter( | |||
Q(status=Asset.Status.VALID) | Q(zarr__status=ZarrArchiveStatus.UPLOADED) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What was the problem that led to this particular change? We need @jjnesbitt to confirm, but I don't think we should be counting Zarr archives in the UPLOADED
state (those have not yet been "finalized" in the terminology of the Zarr API; once they are, I believe they move into the COMPLETE
state).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch -- updated to COMPLETE
here and in the unit test.
The expansion of the filter
query here is to match similar logic in the validate_pending_asset_metadata
Celery task -- specific code reference here
In the case where there is no
only evaluates the |
Moving PR #1818 (Cc @kabilar) to here so there is access to env vars in CI
Also relates to /linc-archive/pull/34 and #1814
During validation of
zarr
based datasets, a validation check occurs on theassetSummary
field in thedandiset.yaml
Prior to this PR, the only "assets" evaluated were "blob". In the query that is updated in this PR, the checks for
size
are expanded to not justblob
but other "asset" types within the Asset model -- see link below:dandi-archive/dandiapi/api/models/asset.py
Lines 137 to 145 in 1be249a
Prior to the fix in this PR, the following validation would fail on assets in
dandischema
such aszarr
: https://github.com/dandi/dandi-schema/blob/bdb7dd681435f1092814041eb01eb4fabe87bf57/dandischema/models.py#L1497