Gen3 Monthly Release 2022.05 gen3-neuro.datacommons.io 1651313335

[PlanXCyborg]

Applying version 2022.05 to gen3-neuro.datacommons.io

[PlanXCyborg]

gen3-neuro.datacommons.io/manifest.json

Deployment changes

• arborist
  • Needs Fence later than 4.22.2 or 2020.08 (which introduces the new JWT scopes claim). ((PXP-6617): remove scopes from jwt aud claim arborist#130)
• fence
  • Requires a Fence DB migration (chore(migration): add authz column to project fence#1003)
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • This commit includes patches to allow old style refresh tokens and API keys to pass new style validation, while issuing new style refresh tokens and API keys. Since refresh tokens and API keys have a default TTL of 30 days, stay on this tag for 30 days if possible. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Old access tokens will NOT work once this is deployed so active sessions may be temporarily interrupted. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
• hatchery
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add gen3-volume-location to the Hatchery configuration file for each container to overwrite the directory that API key will be saved to (PR fix(account): portal to master #33) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option ecs for enabling ECS decoupled workspace (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option subnet for setup transit gateway subnet addresses (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
• indexd
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
  • When running gen3authz, backoff package needed separate poetry run pip install (PXP-6824: return 401 errors from Arborist indexd#304)
• metadata
  • A job has been created to make aggregate MDS population more robust feat(agg-mds-job): add new agg mds job cloud-automation#1722 (feat(populate): improved populate script, docs metadata-service#33)
  • Must set the appropriate AGG_MDS_NAMESPACE value in metadata.env (Feat/mds adapter metadata-service#29)
  • Per chore(metadata): migrate agg MDS configs #3332 a metadata/aggregate_config.json file should be created as part of the manifest config for a commons and the appropriate environment vars for metadata should be defined in the manifest JSON (chore(configs): remove configs from source control metadata-service#30)
  • The new command to populate an aggregate MDS is kubectl exec $(gen3 pod metadata) -- python /src/src/mds/populate.py --config /aggregate_config.json --hostname esproxy-service --port 9200 (chore(configs): remove configs from source control metadata-service#30)
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• portal
  • The pay model support depends on this PR: PXP-8779 HP-347 Feat/ecs hatchery#38 (PXP-8520 Feat/paymodels data-portal#900)
  • set "showExternalLoginsOnProfile": true for showing external logins on the profile page (Feat/profile fed auth data-portal#934)
  • Discovery: introduced a new config section discoveryConfig.features.search.tagSearchDropdown, see portal-config.md for more info (HP-555 Feat/disco tag dropdown viewer data-portal#927)
  • To enable this check, set features.exportToWorkspace.verifyExternalLogins flag to true in gitops (Check all WTS OIDCs are signed in before download/export to WS data-portal#921)
  • Misc: add option for bundle name: ecosystem, which is the same as heal bundle under the hood (Chore/discovery for more bundles data-portal#918)
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)
• sower
  • PS: This is follow-up on this one: feat(dcf): dcf prod release 2.2.7 #24 ([PXP-8941] [PXP-8859] Feat/update go 1.17 sower#25)
• ssjdispatcher
  • PS: This is follow-up on this one: Kids First ETL Mapping #39 ([PXP-8943] [PXP-8859] Feat/update go 1.17 ssjdispatcher#40)
• wts
  • This version of WTS requires a DB migration (in cloud-automation deployments, the DB should be migrated automatically) (PXP-9499 Encrypt refresh tokens workspace-token-service#40)

Breaking changes

• arborist
  • In step with Fence changes to issued JWTs, in which scopes are moved into a dedicated scope claim (Fence 4.22.2/2020.08) and out of the aud claim (Fence 5.0.0/2021.07), this commit changes Arborist JWT validation logic so that it does the same validation it was doing on the aud field, but does it on the scope field instead. This Arborist version must therefore be paired with a Fence later than 4.22.2/2020.08. Conversely, Fence 5.0.0/2021.07 will require Arborist later than this commit. ((PXP-6617): remove scopes from jwt aud claim arborist#130)
• fence
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • Remove scopes from "aud" claim in Fence-issued tokens. This means that in general, old tokens (minted by previous Fence releases) will not pass validation by this Fence. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Access tokens will still include scopes for compatibility with other microservices, but scopes are no longer guaranteed and should no longer be expected in the aud claim from this version on. ID tokens, refresh tokens, and API keys will not have scopes in the aud claim. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add custom "scope" claim in tokens and put scopes there instead. Custom validation for scopes claim added in Authutils 6.0.1 (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add issuer to "aud" claim (alongside other aud values--so when there is a Fence client involved, this means the aud claim will include both Fence's BASE_URL and the client_id) (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Remove custom "aud" claim validation and use normal validation instead (by bumping to Authutils 6.0.1). This means checking that the aud claim contains a value that identifies the consuming service, i.e. Fence's BASE_URL. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
• metadata
  • Do not publish GitHub Packages (feat: do not publish GitHub Packages metadata-service#42)
  • ES indices will not be namespaced per environment (Feat/mds adapter metadata-service#29)
• portal
  • Rewrite some logic to get rid of deprecated React component lifecycle hooks (PXP-7942 Feat/newer react data-portal#848)
  • Removed react-slick and slick-carousel, use antd Carousel component instead (PXP-7942 Feat/newer react data-portal#848)
• sheepdog
  • Sheepdog now validates formats. If your dictionary uses formats and you have existing data or submission pipelines that do not respect these formats, this might be a breaking change (PXP-9192 Validate format sheepdog#363)