Gen3 Monthly Release 2023.06 AnVIL internalstaging internalstaging.theanvil.io - 2023.06 1688661830

[PlanXCyborg]

Applying version 2023.06 to internalstaging.theanvil.io

[k-burt-uch]

This should probably be 5.0.1, there's an issue with time stamps in the drs response in 2023.06 that breaks the drs_pull in the gen3sdk.

[github-actions]

internalstaging.theanvil.io/manifest.json

Deployment changes

• fence
  • Allow setting INDEXD_PASSWORD via environment variable. (allow setting INDEXD_PASSWORD via env var fence#1092)
  • Systems that use visas parsing must update their Fence configuration to set ENABLE_VISA_UPDATE_CRON to True and set GLOBAL_PARSE_VISAS_ON_LOGIN to either True or None (GPE-891 Add ENABLE_VISA_UPDATE_CRON setting fence#1087)
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
• indexd
  • New columns added to index_record table. Alembic migration needed. (feat(DCF-450): Add DRS Fields indexd#357)
• metadata
  • Envs using AggMDS before must run the AggMDS sync job after updating to this version to migrate AggMDS reocrds syntax (HP-1044 Feat/aggmds vlmd metadata-service#90)
• peregrine
  • The pidgin service can be removed if data-portal is on version 5.4.2/2023.05 or newer. Its functionalities are available at peregrine's /coremetadata endpoint (PXP-10708 Add coremetadata endpoint peregrine#198)
  • ASK AN ADMINISTRATOR TO DELETE THE PIDGIN DEPLOYMENT if you remove pidgin. This is NOT automatic (PXP-10708 Add coremetadata endpoint peregrine#198)
  • Peregrine does not rely on configuration files in the cloud-automation repository anymore. The Peregrine configuration files are in the Peregrine repository. (PXP-10163 Python 3.9 peregrine#191)
• pidgin
  • If you are seeing this message, you are deploying the Pidgin service after it has been deprecated. If you are using Peregrine version 3.2.0/2023.04 or newer AND data-portal version 5.4.2/2023.05 or newer, the Pidgin service can and should be removed. (PXP-10708 Deployment notes for release automation pidgin#48)
• portal
  • If Peregrine is on version 3.2.0/2023.04 or newer AND data-portal is on version 5.4.2/2023.05 or newer, the Pidgin service ⚠️ can and should be removed ⚠️ (PXP-10708 Use new coremetadata URL data-portal#1252)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)
• sower.container.image.pelican-export
  • The Pelican export job should now be deployed alongside the metadata-delete-expired-objects cronjob: add "metadata-delete-expired-objects": "quay.io/cdis/metadata-delete-expired-objects:<version>" to the versions block of the manifest, run gen3 kube-setup-metadata-delete-expired-objects-cronjob and grant the metadata-delete-expired-objects-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) and (resource=/programs, method=delete, service=fence) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)

Breaking changes

• fence
  • Removed most Prometheus support (feat(google_update): updated Flask and deps, also don't fail early with Google errors PXP-10900 fence#1086)
  • The default value for the GLOBAL_PARSE_VISAS_ON_LOGIN setting is now False instead of None. By default, parsing visas on login is now disabled and the parse_visas parameter cannot be used (GPE-891 Add ENABLE_VISA_UPDATE_CRON setting fence#1087)
• indexd
  • DRS responses for update_time and created_time reflect the content's updated and created times, not the time they were indexed or their index record was updated. (feat(DCF-450): Add DRS Fields indexd#357)
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
• metadata
  • Data format of response returned from /aggregate/metadata/guid/{guid} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
  • Data format of response returned from /aggregate/metadata/{name} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
• sower.container.image.download-indexd-manifest
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.get-dbgap-metadata
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.manifest-indexing
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.metadata-manifest-ingestion
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.pelican-export
  • The Pelican export job now requires access to submit data in MDS. In cloud-automation setups, run kubectl delete secrets pelicanservice-g3auto, rm Gen3Secrets/g3auto/pelicanservice/config.json and gen3 kube-setup-pelicanjob. Then, grant the pelican-export-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)