Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate stac-populator optional component #482

Closed
wants to merge 1 commit into from

Conversation

mishaschwartz
Copy link
Collaborator

Overview

This component has not been functional since it was deprecated in crim-ca/stac-populator#12

Changes

Non-breaking changes

  • deprecates non-functional component

Breaking changes
None

Related Issue / Discussion

Additional Information

Links to other issues or sources.

CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

@github-actions github-actions bot added component/STAC Features or components related to STAC documentation Improvements or additions to documentation labels Nov 27, 2024
Copy link
Collaborator

@fmigneault fmigneault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you mean by it not working.
It runs each time I reboot the STAC components.

pavics-compose logs -f --tail 100 stac-populator 

[...]
stac-populator     | [INFO] STAC item [65e1ae2daea7231392c89b320c854562] already exists on [http://stac:8000/stac/collections/c604ffb6d610adbb9a6b4787db7b8fd7] (409), updating..
stac-populator     | {
stac-populator     |     "id": "614d9747c2a022a5200565e6904a2ff6",
stac-populator     |     "body": {
stac-populator     |         "type": "asset",
stac-populator     |         "properties": {
stac-populator     |             "uri": "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/disk2/cccs_portal/indices/Final/BCCAQv2_CMIP6/txgt_29/MS/ssp585/simulations/txgt_29_mon_BCCAQ2v2+ANUSPLIN300_FGOALS-g3_historical+ssp585_1950-2100.nc",
stac-populator     |             "filename": "txgt_29_mon_BCCAQ2v2+ANUSPLIN300_FGOALS-g3_historical+ssp585_1950-2100.nc",
stac-populator     |             "extension": ".nc",
stac-populator     |             "dataset_name": "BCCAQv2_CMIP6",
stac-populator     |             "variable_id": "txgt_29",
stac-populator     |             "freq": "MS",
stac-populator     |             "scenario": "ssp585",
stac-populator     |             "collection_id": "CMIP6"
stac-populator     |         },
stac-populator     |         "categories": [
stac-populator     |             "data"
stac-populator     |         ],
stac-populator     |         "item_id": "eda9db16602332a26d6bf6df77387a46"
stac-populator     |     }
stac-populator     | }
stac-populator     | [INFO] STAC item [eda9db16602332a26d6bf6df77387a46] already exists on [http://stac:8000/stac/collections/c604ffb6d610adbb9a6b4787db7b8fd7] (409), updating..
stac-populator     | 2024-11-27 03:35:02,924 stac_generator.plugins.extraction_methods.regex INFO Found matches for regex extract
stac-populator     | 2024-11-27 03:35:02,925 stac_generator.plugins.extraction_methods.header.header WARNING Header extract backend not found for https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/disk2/cccs_portal/indices/Final/BCCAQv2/txgt_32/YS/rcp45/simulations_spatialAvgCensus/BCCAQv2+ANUSPLIN300_MPI-ESM-MR_historical+rcp45_r1i1p1_1950-2100_txgt_32_YS_spatialAvgCensus.nc
stac-populator     | 2024-11-27 03:35:02,948 stac_generator.plugins.extraction_methods.object_store_stats INFO Extracting metadata for: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/disk2/cccs_portal/indices/Final/BCCAQv2_CMIP6/txgt_29/MS/ssp585/simulations/txgt_29_mon_BCCAQ2v2+ANUSPLIN300_EC-Earth3-Veg_historical+ssp585_1950-2100.nc with checksum: None
stac-populator     | 2024-11-27 03:35:02,980 stac_generator.plugins.extraction_methods.object_store_stats INFO Extracting metadata for: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/disk2/cccs_portal/indices/Final/BCCAQv2/txgt_32/YS/rcp45/simulations_spatialAvgCensus/BCCAQv2+ANUSPLIN300_MPI-ESM-LR_historical+rcp45_r1i1p1_1950-2100_txgt_32_YS_spatialAvgCensus.nc with checksum: None
stac-populator     | [INFO] STAC collection [c604ffb6d610adbb9a6b4787db7b8fd7] already exists on [http://stac:8000/stac] (409), updating..
stac-populator     | [INFO] STAC collection [0798aa197d54eb4332767a5a4077fb0f] already exists on [http://stac:8000/stac] (409), updating..
stac-populator     | STAC asset generator ran for 200 seconds. Exiting.
stac-populator exited with code 0

Collections are properly validated/updated against existing/added files:
https://hirondelle.crim.ca/stac/collections/c604ffb6d610adbb9a6b4787db7b8fd7

I agree it needs some updates to make it work with latest changes in the repo (CLI refactor and all), but the component is not broken in its current state with the way it's defined.

@mishaschwartz
Copy link
Collaborator Author

You're using an old version of the image that is cached locally on your machine.

The image is tagged with :master which means that it is not properly versioned and it is updated as the codebase changes.

docker image rm ghcr.io/crim-ca/stac-populator:master
docker run ghcr.io/crim-ca/stac-populator:master ./populate.sh  # this exits with the message "./populate.sh: no such file or directory"

@fmigneault
Copy link
Collaborator

Tag :master is not an auto-updated tag. It is just a poorly named 1-year old tag.
It is not cached locally. It is the same one that is regularly retrieved from:
https://github.com/crim-ca/stac-populator/pkgs/container/stac-populator/148542132?tag=master

@mishaschwartz
Copy link
Collaborator Author

What happens when you try this?

docker image rm ghcr.io/crim-ca/stac-populator:master
docker run ghcr.io/crim-ca/stac-populator:master ./populate.sh # this exits with the message "./populate.sh: no such file or directory"

@fmigneault
Copy link
Collaborator

Oh wow, it is indeed a different version.
The git commit inside the docker is: crim-ca/stac-populator@7341bf7
Therefore, the one just before @dchandan started working on the repository.
I suggest we pin it with 0.0.0 and I push it to the registry to keep track of it.

@fmigneault
Copy link
Collaborator

fmigneault commented Nov 27, 2024

Here it is: https://github.com/crim-ca/stac-populator/pkgs/container/stac-populator/98905655?tag=0.0.0
It matched an existing sha256 that was already in the container registry.

mishaschwartz added a commit that referenced this pull request Dec 3, 2024
## Overview

This sets the docker image for the `stac-populator` component to a
version that actually contains the code that is executed when
`stac-populator` is called. The previous image no longer contained the
relevant code.

## Changes

**Non-breaking changes**
- stac-populator image changed to version 0.0.0

**Breaking changes**
None

## Related Issue / Discussion

- Closes #482

## Additional Information

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/STAC Features or components related to STAC documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants