BDCat staging release 1675275499

[PlanXCyborg]

copying files from preprod.gen3.biodatacatalyst.nhlbi.nih.gov to staging.gen3.biodatacatalyst.nhlbi.nih.gov

[PlanXCyborg]

staging.gen3.biodatacatalyst.nhlbi.nih.gov/manifest.json

Deployment changes

• awshelper
  • Update fluentd to v1.15.3-debian-cloudwatch-1.0 (fix(fluentd-karpenter): Updated fluentd config to work with container… cloud-automation#2142)
  • Add .global.karpenter to manifest if you'd like to use karpenter. (feat(karpenter): Added configuration to use karpenter instead of clus… cloud-automation#2128)
  • To set up the metadata-delete-expired-objects job, run the gen3 kube-setup-metadata-delete-expired-objects-cronjob command. Note that the metadata-delete-expired-objects-job client must be granted access to (resource=/mds_gateway, method=access, service=mds_gateway) and (resource=/programs, method=delete, service=fence) ((PXP-9516): Add metadata-delete-expired-objects job cloud-automation#1941)
  • Currently requires to manually update prisma api key in json format under this path in the adminvm (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • ~/Gen3Secrets/prisma/apikey.json (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • format has to be the following: (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • { (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • "AccessKeyID": "XXXX", (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • "SecretKey": "XXXX" (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • } (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • This PR extends kube-setup-hatchery to create a kubernetes secret with for prisma cloud credentials, and extending the hatchery deployment to look for this secret and use it, if it exists. (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
• dashboard
  • This change will add some CSP and CORS related header to the response that returned from the dashboard services. They should not be interruptive, but each team is encouraged to double check their webpages hosted by dashboard to ensure they still works (fix: sec patch gen3-statics#27)
  • Because of these CSP directives being added, starting from this version, if anyone what to load remote scripts in their dashboard-hosted webpages, they will need to update the CSP directives in this dashboard service if the current directives doesn't fit, or to ship the script files with the page (fix: sec patch gen3-statics#27)
• fence
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
  • Configure the ALLOWED_DATA_UPLOAD_BUCKETS setting to allow users to upload to buckets other than DATA_UPLOAD_BUCKET (PXP-10361 Allow specifying the upload bucket fence#1051)
  • B/c the default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP, you MUST ensure that an instance that should comply with FedRAMP does NOT override this fence configuration (e.g. it should be 15 minutes) SESSION_TIMEOUT: 900 (feat(session): change default to 15 minutes [PPS-122] fence#1040)
• metadata
  • configuration files will need to be updated (PXP-8825 & BRH 206  metadata-service#65)
  • Database migration for MDS (PXP-10258 feat/alias metadata-service#63)
• portal
  • New portal config options to exclude File nodes from the charts added: use "graphql.chartNodesExcludeFiles" to exclude File nodes from graphql chart, or use "components.index.homepageChartNodesExcludeFiles" to exclude File nodes from homepage chart. Note both of them are default to false so you have to explicitly set them to true to enable them (PXP-10565/PPS-177 fix: exclude file nodes from chart with options data-portal#1208)
  • this change assumes WTS is already a core/central service. The dependency is hereby moved to the homepage (instead of the ./workspace page) (Centralize WTS refresh token initialization data-portal#1095)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)

Breaking changes

• arborist
  • The GET /auth/mapping endpoint does not accept username as a query parameter anymore. It only supports parsing the username from the provided JWT. To specify a username, use the POST /auth/mapping endpoint. (PXP-10229 Client auth mapping arborist#153)
• fence
  • Default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP (feat(session): change default to 15 minutes [PPS-122] fence#1040)
• indexd
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (fix(migrate): improve acl->authz migration, remove deprecated endpoints indexd#336)
• metadata
  • The configuration file is not compatible with the previous release. This is due to the new support for defining a schema, new configuration options, and changes to the adapters. (PXP-8825 & BRH 206  metadata-service#65)

[PlanXCyborg]

staging.gen3.biodatacatalyst.nhlbi.nih.gov/manifest.json

Deployment changes

• awshelper
  • Update fluentd to v1.15.3-debian-cloudwatch-1.0 (fix(fluentd-karpenter): Updated fluentd config to work with container… cloud-automation#2142)
  • Add .global.karpenter to manifest if you'd like to use karpenter. (feat(karpenter): Added configuration to use karpenter instead of clus… cloud-automation#2128)
  • To set up the metadata-delete-expired-objects job, run the gen3 kube-setup-metadata-delete-expired-objects-cronjob command. Note that the metadata-delete-expired-objects-job client must be granted access to (resource=/mds_gateway, method=access, service=mds_gateway) and (resource=/programs, method=delete, service=fence) ((PXP-9516): Add metadata-delete-expired-objects job cloud-automation#1941)
  • Currently requires to manually update prisma api key in json format under this path in the adminvm (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • ~/Gen3Secrets/prisma/apikey.json (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • format has to be the following: (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • { (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • "AccessKeyID": "XXXX", (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • "SecretKey": "XXXX" (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • } (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
  • This PR extends kube-setup-hatchery to create a kubernetes secret with for prisma cloud credentials, and extending the hatchery deployment to look for this secret and use it, if it exists. (BRH-301: Extend hatchery to launch prismacloud containers cloud-automation#2061)
• dashboard
  • This change will add some CSP and CORS related header to the response that returned from the dashboard services. They should not be interruptive, but each team is encouraged to double check their webpages hosted by dashboard to ensure they still works (fix: sec patch gen3-statics#27)
  • Because of these CSP directives being added, starting from this version, if anyone what to load remote scripts in their dashboard-hosted webpages, they will need to update the CSP directives in this dashboard service if the current directives doesn't fit, or to ship the script files with the page (fix: sec patch gen3-statics#27)
• fence
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
  • Configure the ALLOWED_DATA_UPLOAD_BUCKETS setting to allow users to upload to buckets other than DATA_UPLOAD_BUCKET (PXP-10361 Allow specifying the upload bucket fence#1051)
  • B/c the default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP, you MUST ensure that an instance that should comply with FedRAMP does NOT override this fence configuration (e.g. it should be 15 minutes) SESSION_TIMEOUT: 900 (feat(session): change default to 15 minutes [PPS-122] fence#1040)
• metadata
  • configuration files will need to be updated (PXP-8825 & BRH 206  metadata-service#65)
  • Database migration for MDS (PXP-10258 feat/alias metadata-service#63)
• portal
  • New portal config options to exclude File nodes from the charts added: use "graphql.chartNodesExcludeFiles" to exclude File nodes from graphql chart, or use "components.index.homepageChartNodesExcludeFiles" to exclude File nodes from homepage chart. Note both of them are default to false so you have to explicitly set them to true to enable them (PXP-10565/PPS-177 fix: exclude file nodes from chart with options data-portal#1208)
  • this change assumes WTS is already a core/central service. The dependency is hereby moved to the homepage (instead of the ./workspace page) (Centralize WTS refresh token initialization data-portal#1095)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)

Breaking changes

• arborist
  • The GET /auth/mapping endpoint does not accept username as a query parameter anymore. It only supports parsing the username from the provided JWT. To specify a username, use the POST /auth/mapping endpoint. (PXP-10229 Client auth mapping arborist#153)
• fence
  • Default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP (feat(session): change default to 15 minutes [PPS-122] fence#1040)
• indexd
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (fix(migrate): improve acl->authz migration, remove deprecated endpoints indexd#336)
• metadata
  • The configuration file is not compatible with the previous release. This is due to the new support for defining a schema, new configuration options, and changes to the adapters. (PXP-8825 & BRH 206  metadata-service#65)