sync gen3staging with gen3qa

[diw2]

Deployment changes

[PlanXCyborg]

gen3staging.kidsfirstdrc.org/manifest.json

Deployment changes

• arborist
  • Assuming any commons running Arborist < 2.3 was using user sync only as AuthZ provider (fix(dcp): peregrine:feat_bagit2 #111)
  • One-off: Update authz_provider to the correct values (chore(edit gitops json for cvb using brain) #109)
  • Database deployment (updating manifest for dcp #42)
  • See migrations folder for database schema and setup utilities (updating manifest for dcp #42)
• fence
  • Fence config ENABLED_IDENTITY_PROVIDERS section deprecated and replaced by LOGIN_OPTIONS section to allow enabling several login options for the same provider (backwards compatible) (Initial ETL mapping #703)
  • Fence does NOT handle c999 the same as before. If using dbgap sync w/ consent code parsing and indexd records have acls with just phsid to represent objects someone w/ c999 should be able to see, you will need to ensure that the records also have the phsid.consent_code as well or users will lose access (Update manifest.json #702)
  • Fence has a new configuration to support more dbGaP syncing options: if you require "Common Exchange Area" data access, review the new configuration and supply necessary values (Update manifest.json #702)
  • Requires manual work in Arborist DB to fill in the empty authz_provider fields (fix(niaiddata-sidecar): fixed the namespace for the sidecar in niaiddata #696)
  • new cloud automation job for database migration feat(jobs): fence db create job cloud-automation#977 (Update gitops.json #694)
  • configuration added for requester pays Google Buckets. if this is needed, you'll need to update configuration. default config does not support requester pays bucket access (UI Changes NDE AIDS #668)
• indexd
  • Add script to run for filling out records' authz fields based on the acl fields. (fix(stage-etl): etl busted in stage #218)
  • cc @david4096 (Fix - demo for NIAID #188)
  • The database table index_record_ace's primary key is (did, acl), so we shouldn't allow duplicates in the ACL list when making an IndexRecord. Using a set removes the duplicates. (feat/niaid qc #198)
• peregrine
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (feat(ibdgc):canary test fence #142)
  • Optional configuration variable ARBORIST (arborist base URL) (fix(portal): bumped to v2.1.1 #138)
  • Requires Arborist to be deployed (fix(portal): bumped to v2.1.1 #138)
  • look for INDEX_CLIENT instead of SIGNPOST in app.config (feat(peregrine):v1.0.4 and peregrine sidecar #135)
  • get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (feat(peregrine):v1.0.4 and peregrine sidecar #135)
• pidgin
  • added required configuration variable "API_HEALTH_URL" for status endpoint (Feat/new manifests #31)
  • use the latest version of the python 3 base image (updating the manifest #29)
  • The Dockerfile now uses an Alpine image (feat(dcf): dcf prod release 2.2.7 #24)
• portal
  • InCommon login needs fence 4.8.0 (feat(buttons): file tab buttons #595)
  • Need Guppy 0.3.7+ to handle large file size (feat(niaiddata-revproxy): fix fence endpoint for niaiddata in revproxy #603)
  • if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (chore/pfb-stage #592)
  • New optional portal config field called adminPreAppliedFilters. Goes in data/config/*.json. (Update manifest.json #576)
  • For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (feat(export-to-workspace): Add export to workspace for nci demo #571)
  • Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (Update manifest.json #570)
  • Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (Update manifest.json #570)
  • (by default). (Update manifest.json #570)
  • Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (Update manifest.json #570)
  • New portal configuration and images for AnVIL (chore(public_datasets): #569)
  • Requires a url to be specified in dataExplorerConfig called terraExportURL (Feat/ncicrcd demo hatchery #568)
  • This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (Feat/ncicrcd demo hatchery #568)
  • Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (chore/patch file filter #565)
  • fixed DAT requires Guppy >= 0.3.1 (Additional changes to Exploration faceted search #563)
  • DAT requires Guppy >= 0.3.0 (chore(bump arbostist to fix resource) #561)
  • This change requires sower version >= 0.3.0. (chore(dcf-arborist): update to 2.0.6 #562)
  • This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (chore(fix usermal path) #558)
  • require fileExplorerConfig block with buttons (feat(update-image): Update the jupyter niaid image which changes work… #554)
  • New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)
  • Feat/logout user opt out cloud-automation#818 (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)
  • enable source map tool so that developers could have better debugging under dev mode (Feat/added tiered access #514)
  • Fix warning from fsevents packages during docker image building (chore(hatchery and ambassador):staging data stage #488)
  • Add flag to npm ci in dockerfile to support npm prepared files from ui-component library (chore(hatchery and ambassador):staging data stage #488)
  • Workspace button in DCF Sandbox (chore(netpolicy):turn on in stagingdatastage #466)
• sheepdog
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (Revert file counts #304)
  • Optional configuration variable ARBORIST (arborist base URL) (Add freesurfer Brain #298)
  • Requires Arborist to be deployed (Add freesurfer Brain #298)
  • look for INDEX_CLIENT instead of SIGNPOST in app.config (feat(stage): etl fix #291)
  • get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (feat(stage): etl fix #291)

[atharvar28]

LGTM 👍

[paulineribeyre]

Created ticket PXP-5191 for the bug related to the failing integration test. we merged because KF doesn't use sheepdog