Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-Issue PR #326

Merged
merged 46 commits into from
Jul 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
7f05899
Merge branch 'epics/89/issues/416/download-files-frontend' into feat/…
jtwillis92 Jun 30, 2021
baac2cf
Merge branch 'backend/833/download-file-endpoint' into feat/834-uploa…
jtwillis92 Jul 2, 2021
2590b70
Merge branch 'epics/89/issues/416/download-files-frontend' into feat/…
jtwillis92 Jul 2, 2021
6a5aa68
Submit data files to actual API, started on updating retrieval
jtwillis92 Jul 2, 2021
0834d3a
Merge branch 'epics/89/issues/416/download-files-frontend' into feat/…
jtwillis92 Jul 2, 2021
1698c5a
Merge branch 'epics/89/issues/416/download-files-frontend' into feat/…
jtwillis92 Jul 6, 2021
fcd6c61
Implemented filtering for ReportFile viewset; update redux reducer fo…
jtwillis92 Jul 6, 2021
066f706
Merge branch 'epics/89/issues/416/download-files-frontend' into feat/…
jtwillis92 Jul 6, 2021
395d97f
Hooked up download endpoint to the frontend; implemented CLEAR_FILE_L…
jtwillis92 Jul 7, 2021
b63a87c
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 8, 2021
38d1908
All endpoints now connected and functional, needs some cleanup and te…
jtwillis92 Jul 8, 2021
418c792
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 9, 2021
9936faf
Endpoint filtering improvements; fix ability to select different STT …
jtwillis92 Jul 9, 2021
a99455c
Updated buildpack documentation for restaging process [draft]
Jul 12, 2021
4b86402
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 13, 2021
b927e94
Fix backend tests & linter errors
jtwillis92 Jul 13, 2021
babe942
Fixed frontend tests
jtwillis92 Jul 13, 2021
15fe344
Fleshing out of process
Jul 15, 2021
7ea732c
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 15, 2021
5db8656
Updates to commands for dev restaging steps
Jul 15, 2021
e02d908
Add missing AWS setting for django-storages
jtwillis92 Jul 16, 2021
6f64d6e
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 19, 2021
bdcaf81
create sprint-25-summary.md
amilash Jul 20, 2021
825058c
update sprint-25-summary.md
amilash Jul 20, 2021
59c2884
Added changelog initial commit
Jul 21, 2021
3a9cdc9
Updates from comments in PR
Jul 21, 2021
bba070d
Update docs/Sprint-Review/sprint-25-summary.md
amilash Jul 21, 2021
7aea1dd
Update docs/Sprint-Review/sprint-25-summary.md
amilash Jul 21, 2021
d3cb094
Update docs/Sprint-Review/sprint-25-summary.md
amilash Jul 21, 2021
9c9e113
Update docs/Sprint-Review/sprint-25-summary.md
amilash Jul 21, 2021
bd935fd
Update docs/Sprint-Review/sprint-25-summary.md
amilash Jul 21, 2021
6c3e3d3
Update sprint-25-summary.md
amilash Jul 21, 2021
040ed40
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 22, 2021
191c6ea
Fix linter errors
jtwillis92 Jul 22, 2021
a1c5002
Update docs/Sprint-Review/sprint-25-summary.md
lfrohlich Jul 26, 2021
414d825
Update docs/Sprint-Review/sprint-25-summary.md
lfrohlich Jul 26, 2021
4cf699c
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 26, 2021
504e075
Merge branch 'raft-tdp-main' into amilash-patch-1
amilash Jul 26, 2021
10baed2
Update 011-buildpacks.md
ADPennington Jul 26, 2021
4dcf39a
Merge branch 'raft-tdp-main' into docs/1045-restaging-for-buildpack-u…
ADPennington Jul 26, 2021
5bc722d
Merge pull request #1128 from raft-tech/amilash-patch-1
andrew-jameson Jul 26, 2021
ecea405
Update docs/Architecture Decision Record/011-buildpacks.md
ADPennington Jul 26, 2021
d4ada01
Merge branch 'raft-tdp-main' into docs/1045-restaging-for-buildpack-u…
ADPennington Jul 26, 2021
b541ab1
Merge pull request #1077 from raft-tech/docs/1045-restaging-for-build…
andrew-jameson Jul 27, 2021
be4194c
Merge branch 'raft-tdp-main' into feat/834-upload-download-connect-re…
jtwillis92 Jul 27, 2021
1d5aaeb
Merge pull request #1061 from raft-tech/feat/834-upload-download-conn…
andrew-jameson Jul 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 54 additions & 1 deletion docs/Architecture Decision Record/011-buildpacks.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 11. Implement Cloud.gov Buildpacks

Date: 2021-02-26
Date: 2021-02-26 (updated 2021-07-26)

## Status

Expand Down Expand Up @@ -33,3 +33,56 @@ Our recommendation is to move to Cloud.gov buildpacks at this time. They are alr
**Notes**
- Docker containers will still need to be maintained for local development and CI/CD
- Docker containers will still need to be hardened for CI/CD

## Restaging for updated buildpacks

As described in [#1045](https://github.com/raft-tech/TANF-app/issues/1045), cloud.gov will inform us that buildpack(s) we use have been updated to a newer version via e-mail to all users with 'developer' role. The e-mail provides specific CloudFoundry CLI steps needed but we have already captured our deployment strategy process/commands in scripts/deploy-backend.sh. Running that script is the preferred methodology. Presently, the e-mail does not provide any specifics about the update, just that there was an update.

Below is the restaging process in full:
1. Upon receipt of email from cloud.gov, restage against dev:
```bash
user@host$ cf login -a api.fr.cloud.gov --sso
API endpoint: api.fr.cloud.gov

Temporary Authentication Code ( Get one at https://login.fr.cloud.gov/passcode ):
Authenticating...
OK


Targeted org hhs-acf-prototyping.

Select a space:
1. tanf-dev
2. tanf-staging

Space (enter to skip): 1
Targeted space tanf-dev.

API endpoint: https://api.fr.cloud.gov
API version: 3.101.0
user: abottoms@goraft.tech
org: hhs-acf-prototyping
space: tanf-dev
$ cf restage tdp-backend-a11y
$ cf restage tdp-backend-raft
$ cf restage tdp-backend-qasp
$ cf restage tdp-backend-sandbox
OR
$ cf restage tdp-frontend-a11y
$ cf restage tdp-frontend-raft
$ cf restage tdp-frontend-qasp
$ cf restage tdp-frontend-sandbox

```
1. Inspect dev environment in cloud.gov for new buildpack versions after restage
1. Inspect relevant official changelog(s):
* https://github.com/cloudfoundry/nginx-buildpack/blob/master/CHANGELOG
* https://github.com/cloudfoundry/python-buildpack/blob/master/CHANGELOG
1. On a new branch, update docs/Technical-Documentation/buildpack-changelog.md with information of the following format:
```
## Buildpacks Changelog
- MM/DD/YYYY [name v#.#.##](link)
- 07/13/2021 [python-buildpack v1.7.43](https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.7.43)
```
1. Open a pull request to 'raft-tdp-main' and assign to Technical Lead
1. Merging pull request shall trigger rolling deploy of the updated buildpack(s) to staging & (eventually) prod without downtime
68 changes: 68 additions & 0 deletions docs/Sprint-Review/sprint-25-summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Sprint 25 Summary
**07/06/2021- 07/20/2021**

### Summary Updates
- The Engineering team continued to work on Secret Key Epic tickets and various devops related tickets to finishing out ATO functionality as well as accessibility work for Django admin and other areas.
- The UX Research team focused on Parsing Error blockers and understanding Regional Staff user journeys, and began investigations into the user access request journey
- NextGen XMS and ACF AMS teams have given us technical information in order to assess and estimate our technical implimentation and recommendation to TDP Product Owner and tech lead. Our next task is to write integration tickets and estimate this work and provide a recommendation to OFA.


## Sprint 25 Goals
- Finish closing ATO functionality tickets
- Focus on Secret Key Leakage Mitigation Epic
- UX Ideation for Parsing Blocker Communications

**[Next Sprint (26) Goals](https://github.com/raft-tech/TANF-app/milestone/29)**
-

## Merged/Completed (Done/Demo, Closed)
- Django Admin a11y Fixes (Sprint 1) [#973](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/973)
- (UX Sprint 24) Current State Analysis of Error Communication and Regional Staff Workflow Validation [#1018](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1018)
- Documentation of current staging environments for TDP [#1051](https://app.zenhub.com/workspaces/tdrs-product-backlog-5f2c6cdc7c0bb1001bdc43a5/issues/raft-tech/tanf-app/1051)
SPIKE: File transfer options for Tribal MVP [#1011](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1011)
## Submitted (QASP Review, OCIO Review)
- Dependabot Mass Merge / Improvements [#1023](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1023)
- Verify Admin Permissions Hierarchy and Roles [#1058](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1058)

## Moving to Next Sprint (Raft Review, In Progress, Current Sprint Backlog)
**Raft Review**
- [EPIC] As an OFA admin, I can download raw file [#89](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/89)
- Groups: Rename `Data Prepper` to `Data Analyst` [#1071](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1071)
- Perform scheduled OWASP scans against deployed site(s) [#1032](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1032)
- [Devops] Allow pa11y to scan views that require authorization [#1044](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1044)
- [Frontend] Hook upload and download to real API endpoints [#834](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/834)


**Blocked**
- Update ATO docs and corresponding code docs [#962](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/962)
-

**In Progress**
- Django Admin a11y Fixes (Sprint 2)[#1053](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1053)
- As a tadpole, I want to know the platform I use to login to TDP (new TDRS)[#379](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/379)
- As tech lead, I want to know the steps that will be followed to use updated buildpacks for TDP apps [#1045](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1045)
- As an OFA Admin, I want an accessible, 508-compliant user interface for managing permissions [#892](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/892)
- Deployed environments should pull AWS credentials from Cloud.gov provided environment variables [#971](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/971)
- SPIKE: Authentication Feasibility Research [#1046](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1046)

**Current Sprint Backlog**
- As a dev, I need to know which authentication service we're using (login.gov vs. NextGen XMS) [#638](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/638)
- [DevOps] Generate a new, random DJANGO_SECRET_KEY on initial Cloud.gov deployments or rebuilds [#967](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/967)
- As a dev, I want an automated tool to prevent me from committing secret keys to the repo [#965](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/965)
- [EPIC] Secret Key Leakage Mitigation [#972](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/972)
- As TDP SO/TL, I need a basic security awareness training developed for IS users (AT-02)[#953](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/953)
- I want a client-side Content Security Policy to protect me from XSS and other client side attacks [#907](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/907)
- Audit Config & Inspection for Production Environment [#897](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/897)
- As a dev, I want Terraform changes to be reflected in label driven deployments (GitHub Action) [#1059](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1059)
- [DevOps] Perform validation on Codecov Bash Uploader script during CI steps [#968](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/968)


## Agenda for Sprint 25 Demo
- (UX Sprint 24) Current State Analysis of Error Communication and Regional Staff Workflow Validation [#1018](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1018) - Miles/ Dmitri
- TDP Staging Site [#1051](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1051) - Jorge
- Django Admin a11y Fixes (Sprint 1) [#973](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/973) - Jorge, locally
- (Pre QASP)[Frontend] Hook upload and download to real API endpoints [#834](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/834) - John
- (Pre QASP)[Devops] Allow pa11y to scan views that require authorization [#1044](https://app.zenhub.com/workspaces/tdrs-sprint-board-5f18ab06dfd91c000f7e682e/issues/raft-tech/tanf-app/1044) - Aaron


[Link to Sprint 25 Milestone Details](https://github.com/raft-tech/TANF-app/milestone/28)
3 changes: 3 additions & 0 deletions docs/Technical-Documentation/buildpack-changelog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Buildpacks Changelog

- 7/13/2021 [python-buildpack v1.7.43](https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.7.43)
29 changes: 29 additions & 0 deletions tdrs-backend/tdpservice/reports/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import logging

from django.http import StreamingHttpResponse
from django_filters import rest_framework as filters
from rest_framework.parsers import MultiPartParser
from rest_framework.response import Response
from rest_framework.status import HTTP_400_BAD_REQUEST
Expand All @@ -17,15 +18,43 @@
logger = logging.getLogger()


class ReportFileFilter(filters.FilterSet):
"""Filters that can be applied to GET requests as query parameters."""

# Override the generated definition for the STT field so we can require it.
stt = filters.NumberFilter(field_name='stt_id', required=True)

class Meta:
"""Class metadata linking to the ReportFile and fields accepted."""

model = ReportFile
fields = ['stt', 'quarter', 'year']


class ReportFileViewSet(ModelViewSet):
"""Report file views."""

http_method_names = ['get', 'post', 'head']
filterset_class = ReportFileFilter
parser_classes = [MultiPartParser]
permission_classes = [ReportFilePermissions]
serializer_class = ReportFileSerializer

# TODO: Handle versioning in queryset
# Ref: https://github.com/raft-tech/TANF-app/issues/1007
queryset = ReportFile.objects.all()

# NOTE: This is a temporary hack to make sure the latest version of the file
# is the one presented in the UI. Once we implement the above linked issue
# we will be able to appropriately refer to the latest versions only.
ordering = ['-version']

def filter_queryset(self, queryset):
"""Only apply filters to the list action."""
if self.action != 'list':
self.filterset_class = None
return super().filter_queryset(queryset)

@action(methods=["get"], detail=True)
def download(self, request, pk=None):
"""Retrieve a file from s3 then stream it to the client."""
Expand Down
6 changes: 5 additions & 1 deletion tdrs-backend/tdpservice/settings/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ class Common(Configuration):
AWS_S3_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_BUCKET")
AWS_REGION_NAME = os.getenv("AWS_REGION_NAME")
AWS_S3_REGION_NAME = os.getenv("AWS_REGION_NAME")

# Those who will receive error notifications from django via email
ADMINS = (("Admin1", "ADMIN_EMAIL_FIRST"), ("Admin2", "ADMIN_EMAIL_SECOND"))
Expand Down Expand Up @@ -154,7 +155,7 @@ class Common(Configuration):
}

# General
APPEND_SLASH = False
APPEND_SLASH = True
TIME_ZONE = "UTC"
LANGUAGE_CODE = "en-us"
# If you set this to False, Django will make some optimizations so as not
Expand Down Expand Up @@ -302,6 +303,9 @@ class Common(Configuration):
"rest_framework.authentication.SessionAuthentication",
"rest_framework.authentication.TokenAuthentication",
),
"DEFAULT_FILTER_BACKENDS": [
"django_filters.rest_framework.DjangoFilterBackend",
],
"TEST_REQUEST_DEFAULT_FORMAT": "json",
"TEST_REQUEST_RENDERER_CLASSES": [
"rest_framework.renderers.MultiPartRenderer",
Expand Down
13 changes: 11 additions & 2 deletions tdrs-backend/tdpservice/users/permissions.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@
"""Set permissions for users."""
from collections import ChainMap

from rest_framework import permissions


def is_own_stt(request, view):
"""Verify user belongs to requested STT."""
is_data_analyst = is_in_group(request.user, 'Data Analyst')
requested_stt = view.kwargs.get('stt', request.data.get('stt'))

# Depending on the request, the STT could be found in three different places
# so we will merge all together and just do one check
request_parameters = ChainMap(
view.kwargs,
request.query_params,
request.data
)
requested_stt = request_parameters.get('stt')
user_stt = request.user.stt_id if hasattr(request.user, 'stt_id') else None

return bool(
Expand Down Expand Up @@ -76,7 +85,7 @@ def has_object_permission(self, request, view, obj):
This is used in cases where we call .get_object() to retrieve a report
and do not have the STT available in the request, ie. report was
requested for download via the ID of the report. This is not called
on POST requests (creating new reports).
on POST requests (creating new reports) or for a list of reports.
"""
is_ofa_admin = is_in_group(request.user, "OFA Admin")
is_data_analyst = is_in_group(request.user, 'Data Analyst')
Expand Down
Loading