Merge pull request #366 from raft-tech/raft-tdp-main

Multi-Issue PR

.circleci/config.yml

@@ -8,7 +8,7 @@ orbs:
 executors:
   docker-executor:
     docker:
-      - image: cimg/python:3.8.10
+      - image: cimg/python:3.10.1
         user: root
   machine-executor:
     machine:
@@ -43,6 +43,12 @@ commands:
           name: Build and spin-up React application
           command: cd tdrs-frontend; docker-compose up -d --build
 
+  disable-npm-audit:
+    steps:
+      - run:
+          name: Disable npm audit warnings in CI
+          command: npm set audit false
+
   # This allows us to use the orb stanza for node/install within other commands
   # NOTE: This doesn't work correctly on machine executors
   install-nodejs: node/install
@@ -214,11 +220,10 @@ commands:
         type: string
     steps:
       - install-nodejs:
-          install-yarn: true
           node-version: "16.13"
+      - disable-npm-audit
       - install-nodejs-packages:
           app-dir: tdrs-frontend
-          pkg-manager: yarn
       - get-app-deploy-strategy:
           appname: <<parameters.frontend-appname>>
       - run:
@@ -361,21 +366,18 @@ jobs:
     steps:
       - checkout
       - install-nodejs-machine
-      - node/install-yarn
+      - disable-npm-audit
       - install-nodejs-packages:
           app-dir: tdrs-frontend
-          cache-path: ~/.cache
-          cache-version: v2
-          pkg-manager: yarn
       - run:
           name: Run ESLint
-          command: cd tdrs-frontend; yarn lint
+          command: cd tdrs-frontend; npm run lint
       - run:
           name: Run Pa11y Accessibility Tests
-          command: cd tdrs-frontend; mkdir pa11y-screenshots/; yarn test:accessibility
+          command: cd tdrs-frontend; mkdir pa11y-screenshots/; npm run test:accessibility
       - run:
           name: Run Jest Unit Tests
-          command: cd tdrs-frontend; yarn test:ci
+          command: cd tdrs-frontend; npm run test:ci
       - upload-codecov:
           component: frontend
           coverage-report: ./tdrs-frontend/coverage/lcov.info

.github/dependabot.yml

@@ -11,6 +11,8 @@ updates:
     labels:
       - "dependencies"
       - "frontend"
+      - "a11y-review"
+      - "dev"
       - "raft review"
     reviewers:
       - "raft-tech/tdrs-developers"
@@ -34,6 +36,7 @@ updates:
     labels:
       - "dependencies"
       - "backend"
+      - "dev"
       - "raft review"
     reviewers:
       - "raft-tech/tdrs-developers"

docs/Background/Current-TDRS.md

@@ -33,7 +33,7 @@ See below for a visual of this data flow.
 ![dataflow_tdrs](https://i.imgur.com/1bK7HMJ.jpg)
 [Figma link to data flow](https://www.figma.com/file/irgQPLTrajxCXNiYBTEnMV/TDP-Mockups-For-Feedback?node-id=5617%3A47733)
 
-We've also outlined what we've observed from [user research](https://github.com/HHS/TANF-app/blob/main/docs/User-Experience/Research/2020%2C%20Fall%20-%20Understanding%20STT%20Roles%2C%20Source%20of%20Truth%2C%20and%20Metadata.md) about how data is prepared, transmitted, and used. This includes details about: 
+We've also outlined what we've observed from [user research](https://github.com/raft-tech/TANF-app/blob/raft-tdp-main/docs/User-Experience/Research-Syntheses/2020%2C%20Fall%20-%20Understanding%20STT%20Roles%2C%20Source%20of%20Truth%2C%20and%20Metadata.md) about how data is prepared, transmitted, and used. This includes details about: 
  * the kinds of data STTs can submit
  * tools STTs use to prepare data files (fTANF, etc.)
  * resources OFA provides to STTs on how to prepare and validate their files
@@ -164,7 +164,7 @@ Data file sizes vary by section of the TANF/SSP-MOE reports and are as follows:
 
 "Encryption" is an artifact of STTs using executables like FTANF.exe and SSPMOE.exe to prepare their data transmissions files. Encryption in this context means that the values in the SSN position of Section 1 and Section 2 data files are replaced with other alphanumeric characters. 
 
-TDRS has a decryption script that runs on transmitted files if (and only if) the files have an "E" as the [encryption indicator in the header record](https://github.com/raft-tech/TANF-app/blob/raft-tdp-main/docs/User-Experience/Research/2020%2C%20Fall%20-%20Flat%20File%20Metadata%20Guide.md#header-structure). For future reference: in the absence of checking the header record, if the SSN includes special characters, this is also evidence of encryption.
+TDRS has a decryption script that runs on transmitted files if (and only if) the files have an "E" as the [encryption indicator in the header record](https://github.com/raft-tech/TANF-app/blob/raft-tdp-main/docs/User-Experience/Research-Syntheses/2020,%20Fall%20-%20Flatfile%20Metadata%20Guide.md#header-structure). For future reference: in the absence of checking the header record, if the SSN includes special characters, this is also evidence of encryption.
 
 This decryption ensures that SSNs stored in the database can be used to match to other administrative data sources (e.g. matching to wage records to track employment outcomes for individuals receiving TANF). 
 

docs/Sprint-Review/sprint-38-summary.md

@@ -0,0 +1,62 @@
+# Sprint 38 Summary
+** 1/5/2021 - 1/18/2021 **
+
+## Sprint 38 Goals
+
+* Documentation (reorganizing directories)
+* Resubmission research synthesis
+* ACF AMS
+
+## Closed/Merged/Completed (Done/Demo, Closed)
+- [#1136](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1136) - [Backend] Accept AMS OpenId
+- [#1520](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1520) - As technical lead, I need ACF AMS integrated for direct DAC access
+- [#1525](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1525) - Update TANF-app/doc/Sprint-Review/
+- [#1524](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1524) - Update TANF-app/doc/Security-Compliance/
+- [#573](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/573) - As a dev, I want a README for the /scripts folder that describes what each script is for and how it is used in the CI process.
+- [#1082](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1082) - Update ADRs
+- [#1483](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1483) - Update Current TDRS documentation
+- [#1482](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1482) - Update workflow.md
+- [#1346](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1346) - [SPIKE] Test hero images in code & adjust spec accordingly
+- [#1418](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1418) - [Research] Code docs synthesis of Resubmission Research
+- [#1522](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1522) - Update TANF-app/doc/Product-Strategy/
+- [#1521](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1521) - Update TANF-app/doc/Background/
+- [#1523](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1523) - Update TANF-app/doc/How-We-Work/
+
+## Submitted (QASP Review, OCIO Review)
+- n/a
+
+## Moving to Next Sprint (Raft Review, In Progress, Current Sprint Backlog)
+- n/a
+
+**Blocked**
+- n/a
+
+**Raft Review**
+- [#1170](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1170) - [Backend] As ACF staff, I must use ACF AMS for authentication.
+- [#782](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/782) - As a dev, I want to be on the latest stable version of Python
+- [#1353](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1353) - Create ADR to document decision to move forward with Elasticsearch
+- [#1419](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1419) - Sidenav traps focus on close button
+- [#1449](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1449) - As OFA tech lead, I need the dependabot PR a11y review process documented and config file updated
+- [#1413](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1413) - Django Admin - Tooltip improvements for wider compatibility
+- [#854](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/854) - As technical lead, I want to switch from Yarn to NPM
+
+**In Progress**
+
+- [#1417](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1417) - As a TDP user, I want to easily navigate to specific pages of the TDP website
+- [#1477](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1477) - Update readmes - **EPIC**
+    - [1527](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1527) - Update TANF-app/doc/User-Experience/
+- [#1461](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1461) - Live Comms (Jan 2022)
+- [#1545](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1545) - Migrate Design's Live Comms Site into the TANF-app repo.
+- [#470](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/470) - As a developer, I want a consistent set of commands to perform essential development operations
+
+
+**Current Sprint Backlog**
+- n/a
+
+
+
+## Agenda for Sprint 38 Demo 
+- Documentation Efforts  - Val and Andrew
+- [1520](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1520) - As technical lead, I need ACF AMS integrated for direct DAC access - **can't demo, but Andrew will address**
+- [1418](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1418) - Code docs synthesis of resubmission research - **Miles and Sreedevi**
+- [1136](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1136) - [Backend] Accept AMS OpenId - **Andrew and Jorge**

docs/Technical-Documentation/Architecture-Decision-Record/017-elastisearch.md

@@ -0,0 +1,45 @@
+# 17. Elasticsearch Architecture
+
+Date: 2021-01-12 (yyyy-mm-dd)
+
+## Status
+
+Accepted
+
+## Context
+
+To modernize the eventual reporting stack for TDP, we have a need for a usable data backend. To establish parity with the legacy TDRS system, we originally planned to store parsed TANF and SSP data in our PostGreSQL database; however, the data required cleaning that is burdensome to perform prior to storage so a faster, scalable solution is desired. 
+
+## Decision
+
+We will use an Elastisearch and Kibana stack for their modern feature set and scalability with large data sets. The Elastisearch/Kibana cluster will be an application hosted in Cloud.gov with the rest of our TDP application so we can leverage all the work surrounding authentication, security, and data compliance.
+
+## Consequences
+
+### Benefits
+
+ * Significant performance increase at large scales.
+   * Elasticsearch retains near real-time search capabilities even with datasets measured in hundreds of Terabytes.
+   * PostgreSQL encounters table/index bloatings with very large data sets (>=1 MM rows) which negatively affect performance without advanced DBA operations.
+ * Built in Reporting and Analytics capabilities - with UI capabilities to create Saved Searches, Visualizations and Dashboards.
+ * Built in CSV export of Visualization data.
+ * Access over REST API or Kibana UI, no database client needed.
+   * This has the added security benefit of ensuring there is not direct database access and users must be passed through our standard authentication which for OFA users will include PIV/CAC card.
+ * Auto-generated index mappings, which can be tweaked to gain further performance advantages. These are also much more flexible than traditional schemas used by relational databases.
+ * Capabilities to perform ML and AI analytics on data sets.
+ * Cloud.gov includes the ES service with FISMA moderate pricing.
+ * Automated Index Lifecycle Management policies can be configured to move data into cold storage, etc to satisfy retention requirements.
+
+### Risks
+ * New query language and interface to learn for OFA staff members who will have access
+   - This is mostly mitigated through a SQL Workbench provided in Kibana where you can use regular SQL syntax to query records.
+ * More infrastructure to manage.
+   * This is mostly mitigated due to using a Cloud.gov managed service for ES and Terraform, this greatly simplifies scaling the cluster and abstracts away a lot of the difficult cluster management tasks we would have to do if we didn't use a managed service.
+   * Additional overhead to run a proxy application to control access to ES + Kibana
+
+## Notes
+
+These changes are slated for later in our releases: currently release 3 but might not be fully implemented until v4.
+
+Please also see these notes: https://gist.github.com/jtwillis92/a6840a412676fc2d3f58c0dccbf10da1
+

docs/Technical-Documentation/circle-ci.md

@@ -57,7 +57,7 @@ These all have defaults set in their respective settings modules, but may be ove
 ## Frontend CI build process
 
 ### test-frontend
-* Runs most steps directly on the machine executor, utilizing `yarn` commands defined in package.json
+* Runs most steps directly on the machine executor, utilizing `npm` commands defined in package.json
 * The exception to the above is the zap scanner step - which runs the frontend via docker-compose, using the nginx target instead of the local dev target
 * Major steps:
     * Run ESLint - ensures styling standards are followed
@@ -75,7 +75,7 @@ These all have defaults set in their respective settings modules, but may be ove
         * `REACT_APP_BACKEND_HOST`
         * `REACT_APP_BACKEND_URL`
         * Only difference in values is whether `/v1` is at the end
-    * Runs `yarn build` which generates the HTML needed to serve to end users
+    * Runs `npm run build` which generates the HTML needed to serve to end users
     * Copies in the nginx configuration for build packs
     * Uploads the build output to Cloud.gov using `cf push`
     * Creates and maps the frontend route

docs/User-Experience/Dev-Ready-Designs/README.md

@@ -0,0 +1,3 @@
+# Designs for Dev
+
+_forthcoming_