Automated manifest pr for gen3.theanvil.io at 07/10/2019 15:45

[emalinowski]

arborist # Release Notes

For: uc-cdis/arborist

Notes since tag: 1.2.1

Notes to tag/commit: 2.0.1

Generated: 2019-07-10

Improvements

• Add some more cute badges to the readme. (chore(dcfdemo_arranger_1):update manifest #83)
• Return the resource body on 409s from resource creation so the caller can
  still see the tag for the existing resource. (chore(dcfdemo mapping_v_2) #78)
• Consolidate the resource create code together so subresources aren't a
  special case. (update dcf diicitonary to version 2.3 #77)
• Include logged-in group by default in the list of groups for queried
  users. (new version of the dictionary #73)
• Simplify the DB schema by not using an internal "root" resource. (chore(dump new arranger config from QA) #59)
• Add more tests for auth endpoints. All the tests. (NIAID - one more variable #52)
• Improved the auth request query (fix(qa-mickey): new qa environment #49)
• README startup guide (chore(dcp new arranger config) #48)
• Commandline argument documentation (chore(dcp new arranger config) #48)
• Use test "fixtures" in a few additional places (not hardcoding test case
  content everywhere it's used) (niaid file etl #45)

New Features

• Support using resource tags directly in auth request (brain mapping and arranger project #86)
• Add QS param ?p to POST /resource which will make arborist create
  parent resources for your request if they didn't exist already. (fix(dcp_1): arranger config dcp_1 #81)
• Add coveralls integration for code coverage analytics. (adding arranger project dcf_1 #80)
• Add QS parameter ?tags for /auth/resources which will make it output tags
  instead of paths. (update dictinoary to 2.2 #70)
• The /user/{name}/resources endpoint also supports this. (update dictinoary to 2.2 #70)
• db migration & utilities (updating manifest for dcp #42)
• Add db migration automation and use this for the travis setup so travis is
  actually running tests on postgres backend (updating manifest for dcp #42)
• Add documentation on migration commands & utility scripts (updating manifest for dcp #42)
• Add CRUD endpoints for users & groups as well as granting policies (updating manifest for dcp #42)
• Add tests for REST API & database operations (updating manifest for dcp #42)
• Add documentation & examples on tests (updating manifest for dcp #42)
• Add resource rules lexer & parser for handling simple logic in auth
  requests (updating manifest for dcp #42)
• Added client (updated dictionary_url for bhcdictionary release v0.4.0 #53)
• Check client permission (updated dictionary_url for bhcdictionary release v0.4.0 #53)
• Add anonymous and logged-in groups which can be given their own
  policies (though they can't contain users). Policies granted to the
  anonymous group allow to all users, whether or not they have a token, and
  the ones granted to logged-in apply to all users with a JWT. (new release 3.1.18 #58)
• Also update the README a bit; there was a duplicated section that's removed
  now. (new release 3.1.18 #58)
• Add tag field to the resources which should be used in cases such as in
  indexd where we want to maintain a persistent reference to a resource in
  arborist, whose name & path are potentially mutable. (chore(niaid_release_1):mapping and arranger #56)
• Add method for revoking all user policies (for supporting fence sync) (New NIAID config #50)
• API documentation on users/groups (chore(dcp new arranger config) #48)
• Add groups API (niaid file etl #45)
• Add API for granting/revoking policies on users & groups (niaid file etl #45)
• Add tests for new APIs (niaid file etl #45)
• Add user CRUD endpoints (feat(kidsfirst): add kf arranger config #44)
• Add tests for user endpoints (feat(kidsfirst): add kf arranger config #44)
• Added resource rules yacc lexer and parser (Update manifest.json dictionary #40)
• Implemented auth/request and auth/proxy (Update manifest.json dictionary #40)
• Added lazy prepared statement cache (Update manifest.json dictionary #40)
• Add more db migration automation and use this for the travis setup (chore(arranger config):cp from QA to prod #41)
• db_version table now stores integer ID and text for the version, not
  postgres date (more robust for applying new migrations) (chore(arranger config):cp from QA to prod #41)
• Add documentation on migration commands & utility scripts (chore(arranger config):cp from QA to prod #41)
• Add tests for REST API & database operations (chore(arranger config):cp from QA to prod #41)
• Add documentation & examples on tests (chore(arranger config):cp from QA to prod #41)

Deployment Changes

• Database deployment (updating manifest for dcp #42)
• See migrations folder for database schema and setup utilities (updating manifest for dcp #42)

Bug Fixes

• Auth resources listings include all the resources below the authorized
  resources. (update ndh dictionary to 3.1.21 #87)
• Fix resource insert trigger to not make a new tag for existing resources.
  (fix(arranger): make arranger pointing at dcp_1 #85)
• Don't alter resource tag on PUT request (Chore/dictionary update #84)
• Support ?p flag on PUT /resource/ (Chore/dictionary update #84)
• Fixed a few bugs in the OpenAPI docs (feat(nginx): Switch to ctds nginx #79)
• Listing resources for a user or doing an auth resources request includes
  resources granted as the result of membership in groups. (chore(remove tube index name block) #75)
• Fix error handling on auth request (chore(niaid_arranger_3) #72)
• Fixes a bug causing the subresources to not submit. Submitting resources
  should work as expected now. (fix(cleanup): remove qa and dev commons #55)
• Separates the Resource model into a ResourceIn (which is the format
  which should be accepted from user input) and a ResourceOut (which is how
  it should look when returned by the arborist API). The difference is that
  inputs can specify the entire resource subtree. (fix(cleanup): remove qa and dev commons #55)
• Include group policies (fix(qa-mickey): new qa environment #49)
• defer did not have intended effect inside test runs, fixed to have
  correct behavior (feat(kidsfirst): add kf arranger config #44)
• couple small fixes in db schema, run make down and make up to reset
  this version (feat(kidsfirst): add kf arranger config #44)
• Fix resource name formatting in policies (root.a.b.c in db vs /a/b/c
  responses & input) (chore(arranger config):cp from QA to prod #41)
• GET /resource will return 404 correctly (chore(arranger config):cp from QA to prod #41)
• Server.MakeRouter now takes io.Writer input so HTTP access logs can go
  somewhere other than stdout (chore(arranger config):cp from QA to prod #41)
• Fix a couple status codes (201 vs 204 on deletes) and use http.Status*
  instead of numerals (chore(arranger config):cp from QA to prod #41)

fence # Release Notes

For: uc-cdis/fence

Notes since tag: 2.8.2

Notes to tag/commit: 3.2.4

Generated: 2019-07-10

Dependency Updates

• bump sqlalchemy 0.9.9 to 1.3.3 (Update manifest.json #630)
• bump userdatamodel 1.5.0 to 1.5.1 (Update manifest.json #630)
• bump authlib from 0.9 to 0.11 (Update manifest.json #628)

New Features

• dbgap syncing now updates resources in arborist (Gkuffel branch #641)
• SESSION_COOKIE_DOMAIN is now configurable (Update manifest.json #640)
• Client policies can be managed in the user.yaml as long as the client
  already exists in fence (Update manifest.json #642)
• use standard base image (Update etlmapping.yml #638)
• Configure new clients with arborist policies using --arborist and
  --policies flags for fence-create (chore/edc #608)
• Use arborist for authorization checks on downloading data files from
  indexd, as long as the authz field is present in the index record (chore/edc #608)
• swagger document for multipart upload presigned url (chore(niaid-dat) #616)
• Allow setting policies for anonymous and logged-in groups in the
  user.yaml. (patch update for fence #625)
• Use arborist to check permission for indexd record upload/download on
  records when supported (send "rbac" field from the indexd record to
  arborist) (updated QA BRAIN dd #606)

Improvements

• dbgap sync will no longer create duplicate entries in arborist database for
  user with lower & upper case name (chore(data upload anvilstaging) #655)
• service account validation will only verify that users on the Google
  Project have access to the projects the active registered service account
  have access to (in other words, expired service account project access
  won't be checked since they don't actually have access to data) (updated BloodPAC data dictionary #651)
• fence will now consider authz field on indexd record to determine if a file
  is public or not (for different signed url behavior). previously only
  checked acl field (Changed fieldMapping for NDE AIDs Exploration page #653)
• allow not providing a user.yaml (e.g. just a pure dbgap sync). previously
  you had to pass an empty user.yaml (Gkuffel branch #641)
• dbgap study to arborist resource namespace configuration (Gkuffel branch #641)
• specify upstream idp to require user re-auth. caveat is some IdPs wouldn't
  support it (Update manifest.json #643)
• The login endpoints now verify that the redirect provided is valid
  according to fence: it redirects back to the Gen3 application, to an OAuth
  client, or to some other approved URL from the configuration. (updated BRAIN dd #540)
• Deployment changes (updated BRAIN dd #540)
• Added optional new LOGIN_REDIRECT_WHITELIST config variable to allow for
  redirecting to specific domains for login/logout. (updated BRAIN dd #540)
• Base class for different IDPs for Login (chore/pfb-stage #593)
• Update default configuration with details about setting up ORCID &
  Microsoft OAuth 2 Clients (chore/pfb-stage #593)
• Better logging for sftp connections for dbgap syncing (fix(login.bionimbus): canary busted #637)
• filter SAWarning about not reflecting partial indices (chore(peregrine_timeout aids) #636)
• Support providing alternative identity provider in oidc flow (Update manifest.json #635)
• use new gen3config instead of having all config code in fence (chore/pfb-stage #592)
• validate user.yaml files via gen3users tests before running usersync (Chore/update stage #629)
• Update user sync for compatibility with changes to arborist. (Modify NIAID DAT display fields #613)
• Policies now owned by arborist (Modify NIAID DAT display fields #613)
• Arborist needs (read-only) copy of users. (Modify NIAID DAT display fields #613)
• User info endpoint will return policy list from arborist for that user, if
  available. (Pelican new version for STAGE #604)

Breaking Changes

• remove --verbose option to fence-create CLI, prefers checking the
  DEBUG value from config now to determine whether or not to output debug
  logs (updated DCF dd #647)
• Users' RBAC policies are now owned by arborist, not fence, so the tables
  are removed from fence/userdatamodel. (chore/edc #608)
• The usersync is updated to work specifically with the new arborist version.
  (chore/edc #608)
• In general, this and following versions of fence should be deployed only
  with
  arborist>=2.0.0
  (chore/edc #608)
• Intended to work only with arborist >= 2.0 (using new database). Won't work
  with older arborist API. (Modify NIAID DAT display fields #613)
• Remove (unused) RBAC blueprint (updated QA BRAIN dd #606)
• Questions: (updated QA BRAIN dd #606)
• stick with "read-storage"/"write-storage" ? (updated QA BRAIN dd #606)

Bug Fixes

• fix exception when dbgap resources and user.yaml have a matching root level
  resource without any subresources and create regression test (Update tube to 0.3.10 #656)
• google-user-sa-validation calls dockerrun to start service now instead of
  forcibly calling apache (was getting errors b/c fence is using nginx now)
  (chore(anvil-prod): update everything #654)
• fix issue with service account registration where validation checks new
  access and previous access when trying to update an SA (so if updating to
  restrict to a subset of previous access, validation may fail and not allow
  the update) (updated BloodPAC data dictionary #651)
• Re-add openssh back to image for sftp access to work correctly (fix/move dat config out of file explorer config #644)
• fix error with logger if no user.yaml provided (feat(add-bhcdatastaging): added bhcdatastaging #639)
• can now successfully call fence-create sync --sync_from_dbgap True
  without providing a valid user.yaml (e.g. just a dbgap sync) (fix(login.bionimbus): canary busted #637)
• Fix typos in exception handler (chore(anvil-staging): add sower #626)

indexd # Release Notes

For: uc-cdis/indexd

Notes since tag: 1.0.10

Notes to tag/commit: 02d6ed8d672137ee056faf019d562230b3acc348

Generated: 2019-07-10

Breaking Changes

• Functionality of POST /dataobjects/list has been moved to GET /dataobjects, and the parameters provided to this endpoint are now
  specified as query parameters instead of in the request body. (Fix - demo for NIAID #188)

Improvements

• Specification of required and optional parameters has been properly
  serialized in the schema. (Fix - demo for NIAID #188)
• checksum parameter is now optional for ListDataBundles and
  ListDataObjects. (Fix - demo for NIAID #188)
• Minor changes for PEP8 compliance. (Fix - demo for NIAID #188)
• indexd records created during the data upload flow will contain the file
  names. (Update to latest versions of services #197)
• update swagger docs (upgrading tube version to 1.0.5 #186)

Dependency Updates

• bump sqlalchemy from 1.0.8 to 1.3.3 (Feat/anvilstaging #211)
• bump sqlalchemy-utils from 0.32.21 to 0.33.11 (Feat/anvilstaging #211)
• gnupg2 is now installed when running tests with Docker (reverting etl_mapping and tube changes made to dcp #189)

Bug Fixes

• Remove extra leading / in resource paths in migration (test(jenkins): testing shared lib #221)
• Fix log message (test(jenkins): testing shared lib #221)
• Make window size formatting string clearer (test(jenkins): testing shared lib #221)
• Fix Swagger doc BulkInputInfo model (chore(ndh_fence_2.5.3) #208)
• Allow ACLs to be duplicated in submissions & updates but reduced to a
  unique set. Fixes adding new arranger project and updating manifest #204 (feat(unaligned_read_count): add unaligned read #207)
• Fix Docker testing harness failing on start because gnupg2 is not
  installed. (reverting etl_mapping and tube changes made to dcp #189)
• Add bullet list to README (Updated BRAIN Commons dictionary #194)

Deployment Changes

• Add script to run for filling out records' authz fields based on the
  acl fields. (fix(stage-etl): etl busted in stage #218)
• cc @david4096 (Fix - demo for NIAID #188)
• The database table index_record_ace's primary key is (did, acl), so we
  shouldn't allow duplicates in the ACL list when making an IndexRecord.
  Using a set removes the duplicates. (feat/niaid qc #198)

New Features

• Add coveralls integration for code coverage analytics (fix(portal): upgrade portal to 2.4.0 #226)
• Now Indexd calls Arborist for permission check (Update manifest.json #215)
• added new authz field (previously rbac field) (Change NIAID portal to new release #214)
• Changing from apache 2 to nginx (chore(dcf): update services to match staging #213)
• updating to uwsgi (chore(dcf): update services to match staging #213)
• Record now have a new rbac field (chore(ndh_fence_2.5.3) #209)
• DRS v0.5.0 compliance (Fix - demo for NIAID #188)
• Update the blank record creation endpoint in indexd to support providing
  the file name. (Update to latest versions of services #197)

pidgin # Release Notes

For: uc-cdis/pidgin

Notes since tag: 1.1.0

Notes to tag/commit: a4aa1d95a09f573b1f9202e70d7bb2cad6798205

Generated: 2019-07-10

Deployment Changes

• use the latest version of the python 3 base image (updating the manifest #29)

Improvements

• add schemaorg example to swagger documentation (feat(brain): data and config index v1 #28)

portal # Release Notes

For: uc-cdis/data-portal

Notes since tag: 2.8.0

Notes to tag/commit: fd6be3020958c359fb9b78714fe68d22836f6687

Generated: 2019-07-10

Deployment Changes

• New manifest.json variable: logout_inactive_users. Defaults to true if not
  provided. (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)
• Feat/logout user opt out cloud-automation#818 (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)
• enable source map tool so that developers could have better debugging under
  dev mode (Feat/added tiered access #514)

Bug Fixes

• Fix for issue where Guppy's downloadRawDataByTypeAndFilter() is returning
  different data formats in different commons. (chore: update dictionary in dcf demo and canine #551)
• Guppy export to workspace flow was using incorrect manifest format and was
  including entries with undefined object_ids, resulting in a 400 response
  from the manifest service (feat(stage-staging): match prod config/manifest #548)
• For the data upload process, we were casting everything to a string. This
  created issues when the required input was a number. Fixes to parse as an
  when needed integer. (Chore/niaid nav #545)
• Also fixes UI bug with the floating table header on the Data Upload page
  (Chore/niaid nav #545)
• Fixes some padding inconsistencies for the responsive layout (chore/update tube ver #544)
• fix index chart crash bug when less than 3 nodes (Update manifest.json #543)
• fixed manifest format for downloading by flatting its content (chore(add-container): Added higher memory container to edc #541)
• The old download manifest logic was replaced by a similar logic for
  creating manifest for export to workspace (Brain #538)
• Fix a bug that guppy explorer's "export to workspace" button not enabled at
  first time (Updated QA BRAIN ETL mapping for new DD #537)
• When files are re-uploaded/edited and submitted again, submission status is
  reset. Previously, errors persisted, even when corrected in the file. (feat(add-pfb): added export to pfb to canine #534)
• fixed missing slash for query page (Update manifest.json #532)
• add default value for tier access limit for local script (Update manifest.json #531)

• The query page now calls updateSession() before executing a query. This
  fixes the bug where query results come back empty with a 200 response in
  the case where the user's session has expired. (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)

• The refreshSession function now hits /user/user instead of just /user, so
  that we can notify the user with the ReduxAuthTimeoutPopup if their session
  is over (/user is public, /user/user is not). (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)

• Makes charts even width and not overflow in explorer, no overflow in facet
  texts, and puts buttons on login page in column flow (Automated manifest pr for genomel.bionimbus.org at 07/24/2019 14:18  #524)
• fix the error filter to get right error entities for submission result
  (chore(ibdgc): fence 2.8.2 for multipart upload #518)
• make submission status shows the overall status (chore(ibdgc): fence 2.8.2 for multipart upload #518)
• add arranger query limit to first 10000 to avoid err response (fix/stage #515)
• Empty lines will not be included in chunks now for submission (updated BloodPAC Workspace VM images #508)
• Page no longer jumps, as the iframe no longer expands enough to scroll.
  (chore(bump versions) #506)
• Updated Brain to be acronym (Automated manifest pr for data.bloodpac.org at 07/10/2019 15:45  #504)

Code Changes

• The SessionMonitor no longer logs out users. If a user has been inactive
  and the config has logoutInactiveUsers == true, we just decline to refresh
  their session and allow Fence to log them out. At this point, we start
  using Sheepdog to check their auth status instead of Fence, so that we can
  check their auth status without refreshing the access token. (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)

• Users are allowed to keep a workspace tab open for the duration of the
  Fence session lifetime without being logged out (likely 8 hours). We can't
  detect user activity within the workspace iframe, so a solution to detect
  their activity is not trivial. (Note that once a Jupyter pod is spawned, it
  stays alive regardless of whether or not we redirect their browser to the
  login page.) Allowing "inactivity" on this page will make it easier for
  users to perform long-running analysis operations in a notebook overnight
  (the redirect didn't work before anyway, because the browser displays that
  "You have unsaved changes" popup which can be declined). (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)

• The session is now updated every 5 minutes, rather than every 10 minutes.
  If a user's auth has expired and the session is not updated, the user won't
  know they've been logged out until they navigate to a new page (or until
  they notice that pages are suddenly refusing to serve them data).
  Increasing the updateSession frequency is helpful in this situation. (Automated manifest pr for data.bloodpac.org at 07/24/2019 14:19  #526)

Improvements

• File explorer that uses ES index instead of slow graphQL queries (Enable download manifest button in STAGE #547)
• Better bar charts for guppy explorer with "show more"/"show less" buttons
  (Updated QA BRAIN ETL mapping for new DD #537)
• Revise some guppy explorer options (Updated QA BRAIN ETL mapping for new DD #537)
• Add canine portal config (Updated QA BRAIN ETL mapping for new DD #537)
• Updated BRAIN logo (Brain #539)
• Updated logo (fix(change-niaddata-fence): change fence niaddata to version that all… #536)
• Relevant homepage stats (fix(change-niaddata-fence): change fence niaddata to version that all… #536)
• Windmill is more usable on tablet/mobile (Deploy file explorer to BRAIN #519)
• remove guppyConfig for now (Update manifest.json #531)
• Overall UI/UX improvement to meet design requirements for tiered access
  data explorer page (Update manifest.json #517)
• remove useless post-install just to speed up local developing (Revert "Bump versions in NDEF" #530)
• Before starting, check if the homepage chart config in portal config is
  correct (Bump versions in NDEF #529)
• improve export to workspace performance (Chore/update niaid etl #520)
• Text for HIV Classifier app is clearer (Update manifest.json #516)
• UI looks cleaner (Update manifest.json #512)
• Updating NIAID configuration for new explorer facets and export to
  workspace button (fix typo #510)
• DataSTAGE branding (Rename charlie to naids #509)
• Using the button component from UI library (updated BloodPAC Workspace VM images #507)

New Features

• Merges the file explorer with the data explorer (Enable download manifest button in STAGE #547)
• Use Hatchery to allow users to pick from Docker containers (Automated manifest pr for acct.bionimbus.org at 07/24/2019 14:19  #527)
• Add support for external Fence (Feat/niaid prod guppy #542)
• Windmill looks better on tablet/mobile (Deploy file explorer to BRAIN #519)
• Implemented blocking table for authenticated but not authorized view (Update manifest.json #517)
• Utilized ConnectFilter from Guppy (Update manifest.json #517)
• Data explorer now supports tiered access by using Guppy (Update manifest.json #517)
• More relevant Explorer. (feat(update-sower): Updated sower version for export to pfb fix #522)
• More relevant number on homepage for mickey (chore(ibdgc): update versions #521)
• Use guppy for data explorer instead of arranger (Automated manifest pr for acct.bionimbus.org at 07/10/2019 15:45  #505)
• Apps!! (Automated manifest pr for gen3.datastage.io at 07/10/2019 15:45  #501)
• Updated packages, most notably Webpack v4 (chore(persistent disk):niaid #491)
• Adds configuration for chart colors (updated BloodPAC Workspace VM images #507)

spark # Release Notes

For: uc-cdis/gen3-spark

Notes since tag: 1.0.0

Notes to tag/commit: 00ef588fd7d64415afe396bc754a6b39d20d7638

Generated: 2019-07-10

tube # Release Notes

For: uc-cdis/tube

Notes since tag: 0.1.3

Notes to tag/commit: 0.3.7

Generated: 2019-07-10

Deployment Changes

• You may need to change the config map to use the new mapping file if the
  commons still use the old version of mapping file. (fixing qa-brain etl mapping #43)

New Features

• follow_up document for niaid (new version of the dictionary #73)
• Dictionary of properties with key as (doc_name, prop_name) (new version of the dictionary #73)
• sliding window (fix(arranger): arranger config version 3 #71)
• min/max aggregation (fix(arranger): arranger config version 3 #71)
• File can belong to multiple cases (update dcf dictionary to 2.1 #68)
• File can connect directly to project, not case (update dcf dictionary to 2.1 #68)
• Support one file belongs to multiple cases in file index (Feat/dcfdemo explorer #67)
• Forced running etl without new data (chore(niaid etl beta data) #66)
• Facet for DCP (update to 3.1.19 #61)
• add timestamp to the es index (fix(cleanup): remove qa and dev commons #55)
• fix list type (change to master branch - NIAID #54)
• add set function on an aggregation node (chore(manifest):sync up with prod current version #51)
• Default empty dataset when no data in database (fix(qa-mickey): new qa environment #49)
• Implemented ordering and get first element of flatten_props (support
  many-to-one) (chore(dcp new arranger config) #48)
• Update latest version of gdcdatamodel that fixes subgroup nested (fix(qa-dcp): add revproxy image version info #47)
• Implemented Versioning for the Index (fixing qa-brain etl mapping #43)

Improvements

• Support working with many-to-one relationship (chore(dcp new arranger config) #48)

Bug Fixes

• fix backward compatible for arranger index (new version of the dictionary #73)
• fix bug create type for auth_resource_path (new version of the dictionary #73)
• File with case_id null (update dcf dictionary to 2.1 #68)
• data_format and data_type empty (update dcf dictionary to 2.1 #68)
• missing category field (DCF and BloodPAC etlMapping.yaml files added #62)
• There is a bug in timestamp feature branch that used hard-code index name
  called etl. It is fixed by this branch (new release 3.1.18 #58)