How to integrate a zarr component into stac-browser

[rabernat]

We are working on referencing Zarr stores as collection-level assets from STAC catalogs (see radiantearth/stac-spec#781 for discussion). We will basically point to a .zmetadata json file which represents the zarr "consolidated metadata" for the store. Here is an example of what this file looks like: https://storage.googleapis.com/cmip6/tracmip/A3hr/aqua4xCO2/AM21/hus/v20190116/.zmetadata

Now we would like to figure out how to render some of this metadata in stac browser. I hacked together some vue components for doing this from a single zarr store here: https://github.com/rabernat/zarr-metadata-vue-browser; but I am very far from a competent vue developer.

Can you provide some guidance on the overall architecture for integrating a custom collection-level asset viewer into a stac browser?

cc @charlesbluca @jhamman

[lossyrob]

Hi @rabernat - looks like a lot of great work is happening to get zarr and STAC working together! It looks like what you need is #38 to be implemented, and then a way to render the zarr metadata explicitly.

I ran the zarr-metadata-vue-browser agains the test .zmetadata you mentioned in this issue, and I think is a good starting point for adding metadata to the stac-browser if there's a collection-level asset that is a zarr file.

My first guess at the way to implement this is to have a computer property on the Catalog (e.g. like for providerscomponent that would read and return the zarr metadata asset information if it was available, and then similar to how something like providers are rendered render the zarr components for each zarr metadata that exists. It might be tricky to contain that all in the sidebar information, so perhaps this info would deserve it's own tab.

What is the way zarr metadata is being specified as a collection asset? Will it have a specific media_type? That's how we determine what assets can show up on a map as a COG - by checking the media type of assets.

[rabernat]

What is the way zarr metadata is being specified as a collection asset? Will it have a specific media_type?

This is what @charlesbluca and I are working on in pangeo-data/pangeo-datastore-stac#4. I think the .zmetadata file will be an asset with media_type='application/json'. We are open to advice on the proper role. I guess metadata? Can we also have a special role for zarr-consolidated-metadata/

[lossyrob]

@rabernat makes sense - for the stac-browser case, it would be easiest to pick out the zarr metadata if the role was specific to zarr. However the logic could also easily look for assets with the metadata role that had a .zmetadata extension, so I think whatever is decided that's best for the spec will work as long as the metadata can differentiated somehow.

[rabernat]

In pangeo-data/pangeo-datastore-stac#6, @charlesbluca added the assets. They look like this

  "assets": {
    "zmetadata": {
      "href": "https://storage.googleapis.com/pangeo-ecco-sose/.zmetadata",
      "description": "Consolidated metadata file for Zarr store",
      "type": "application/json",
      "roles": [
        "metadata",
        "zarr-consolidated-metadata"
      ]
    }
  }

https://github.com/pangeo-data/pangeo-datastore-stac/blob/c8044d7b30590353a4a2b8903a7ae190df9a27a5/master/ocean/SOSE/collection.json#L70-L80

My first guess at the way to implement this is to have a computer property on the Catalog

So would a PR to implement this in stac-browser be our next step? This sounds like a concrete action we could work on.

It might be tricky to contain that all in the sidebar information, so perhaps this info would deserve it's own tab.

👍 There is a ton of info in the zarr metadata!

[lossyrob]

Thanks for the description of the roles - that makes sense and should be easy to identify in stac-browser.

So would a PR to implement this in stac-browser be our next step?

A PR would be awesome! Please let me know how I can be helpful navigating the codebase or thinking through any challenges you face while working through it.

[charlesbluca]

Thanks for your help in moving this forward @lossyrob! A couple questions:

• Should the GET request for the Zarr metadata file be done within the computed property (in common.js), or within the Vue component it's rendered in using the mounted hook?
• Looking into Add collection assets and summaries #47, would the collection summaries tab be a good place to hold the rendered Zarr metadata, or would it be better to give it its own separate tab?

[lossyrob]

@charlesbluca I'm actually going to move the summary metadata to the right bar per Matthias's suggestion, as I agree that's a more proper place for summary properties since they more closely match the metadata being displayed in the sidebar.

However the Zarr metadata is specific enough that I think having its own tab makes sense, so I'd suggest repeating the tab approach I took in #47 but for a Zarr-specific tab.

For the GET request question - IMO common.js is overloaded at this point, and as much as we can keep things out of it and in it's own components the better. Fetching in the component as you mentioned would be the better approach to me.

[charlesbluca]

Thanks for the suggestions! Your tab approach in the PR should be a great help in getting this implemented - this shouldn't be too far off from @rabernat's initial example.

[charlesbluca]

Now that #57 is merged, we have a ZarrMetadataTab component to render a zarr-consolidated-metadata asset if it exists within a Collection. Some other places we would like to expand upon this:

• Versioned Zarr metadata assets, along with a check in the component's parsing function to account for version
• Adding styling to make the component's design more cohesive with that of STAC Browser (either independently or in a future sprint)
• Resolving CORS/requester pays conflicts with Zarr datasets in our STAC catalogs, so that the contained datasets can be rendered in STAC Index

I'm happy to discuss more ideas about where to go from here, although I'd imagine that might be better fit for a new issue.

[m-mohr]

Agreed. Feel free to create new issues. I'm closing here to keep the issues "clean". Thanks for your contribution to STAC.

[christophenoel]

Just for my understanding: using a collection-level store holding all related products make sense if the number of products is quite limited...

But in the scope of a satellite mission (Sentinel-2, Prisma, ...) aggregating millions of products in a collection Zarr store, would lead to a gigantic consolidated metadata file.

How would you handle such mission ? Would you define maybe a per region/tile store (or per month store) ?

Thanks.

[m-mohr]

Hi @christophenoel,

I'm not sure this is the right place to ask this question as this is just about visualization in STAC Browser and no maintainer has actual experience with ZARR and the changes here are not included in STAC Browser v3 anymore. This probably needs to be asked somewhere in a ZARR forum or so.