Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

10/16/24 production deploy #1366

Merged
merged 90 commits into from
Oct 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
2f9fea9
actually start deleting old s3 objects
Sep 11, 2024
564ae06
fix schedule
Sep 11, 2024
ada2c3e
Change 18f deploy tool to CG deploy tool
Sep 18, 2024
a3c1663
fix phone number lookup
Sep 24, 2024
881a251
fix phone number lookup
Sep 24, 2024
ec6bfd8
improve report performance
Sep 25, 2024
291890b
revert change to task schedule
Sep 25, 2024
0736c98
Merge pull request #1334 from GSA/update-deploy-tool
ccostino Sep 26, 2024
e5ac50b
add test
Sep 26, 2024
d27d877
Merge pull request #1330 from GSA/fix_test_coverage
terrazoon Sep 26, 2024
ec4e522
Modify input to accepted argument
Sep 26, 2024
8aae0d5
use shared memory instead of expiring dict for jobs cache
Sep 26, 2024
561d813
remove redis_store mocking from s3 tests
Sep 26, 2024
c2f2cbc
fix tests
Sep 26, 2024
3cba7f1
fix tests
Sep 26, 2024
24f2b9f
fix flake8
Sep 26, 2024
b6d37b7
dont do coverage of test directory
Sep 26, 2024
7c1ca17
dont do coverage of test directory
Sep 26, 2024
fceeed2
dont do coverage of test directory
Sep 26, 2024
3259717
dont do coverage of test directory
Sep 26, 2024
b656ad8
dont do coverage of test directory
Sep 26, 2024
67467bb
dont do coverage of test directory
Sep 26, 2024
b4db1b0
revert coverage changes
Sep 26, 2024
1569de2
revert coverage changes
Sep 26, 2024
096e3a3
clean up
Sep 26, 2024
8e6c079
fix flake8
Sep 26, 2024
26ffb93
optimize S3 partitioning
Sep 27, 2024
ca89779
get the lines that are not covered for tests
Sep 27, 2024
8918dbc
increase test coverage
Sep 27, 2024
d6f4aca
fix format
Sep 27, 2024
b265186
Merge pull request #1336 from GSA/notify-admin-1956
terrazoon Sep 27, 2024
d3768e3
Merge pull request #1341 from GSA/shared_memory
terrazoon Sep 27, 2024
580885a
merge from main
Sep 27, 2024
16b357f
add comment
Sep 27, 2024
e99f4bc
suppress warnings
Sep 27, 2024
0d82f89
reduce max connections to 10
Sep 27, 2024
e0dab07
Fix env var syntax
Sep 30, 2024
db7cc32
Merge branch 'main' of https://github.com/GSA/notifications-api into …
Sep 30, 2024
f844da8
update comment
Sep 30, 2024
6b861f3
fix debug
Sep 30, 2024
544e7e6
code review feedback
Sep 30, 2024
8082b09
Merge pull request #1340 from GSA/fix-cfcommand-deploytool
ccostino Sep 30, 2024
f4a0762
Fix syntax on cf command
Sep 30, 2024
c0388d2
Merge pull request #1344 from GSA/fix-cg-deploytool
terrazoon Sep 30, 2024
70dc52b
Go back to brackets with secrets. in command
Sep 30, 2024
93fe137
Merge pull request #1345 from GSA/fix-cgcommand-brackets
terrazoon Sep 30, 2024
1c231b1
Adjust LOGIN_DOT_GOV_REGISTRATION_URL env var in staging deploy
ccostino Sep 30, 2024
92877f8
Merge pull request #1346 from GSA/one-more-staging-depoy-fix
A-Shumway42 Sep 30, 2024
1f4905e
Refer to GitHub Action container environment variable
ccostino Oct 1, 2024
76eb528
add documentation for how to rotate DANGEROUS_SALT
Oct 1, 2024
816c8e9
Merge pull request #1347 from GSA/login-gov-url-fix
ccostino Oct 1, 2024
216528d
Merge pull request #1339 from GSA/faster_s3_downloads
terrazoon Oct 1, 2024
c2ed8f1
fix personalization_bug for one-offs
Oct 1, 2024
f55c437
try fixing tests
Oct 1, 2024
b7a6f4a
try fixing tests
Oct 1, 2024
9c43329
fix test
Oct 1, 2024
641d168
fix properly
Oct 1, 2024
a70b450
revert tests
Oct 1, 2024
05bac16
One more attempt at fixing the Login.gov registration URL
ccostino Oct 1, 2024
ab7e575
don't run coverage on tests
Oct 1, 2024
37e5de3
don't run coverage on tests
Oct 1, 2024
dbfc025
Merge pull request #1350 from GSA/login-gov-url-fix-again
ccostino Oct 1, 2024
b70d47f
Attempt quoting the Login.gov URL
ccostino Oct 1, 2024
04e9836
Merge pull request #1352 from GSA/fix-login-gov-url-with-quotes
ccostino Oct 1, 2024
f8410a2
Attempt quoting the Login.gov URL with var-name
ccostino Oct 1, 2024
bda8b75
Merge pull request #1349 from GSA/fix_personalization_bug
terrazoon Oct 1, 2024
05838f9
Revert back to our original formatting, but include the --strategy ro…
ccostino Oct 1, 2024
5edf0c7
merge from main
Oct 1, 2024
182572a
Forgot to add the push command
ccostino Oct 1, 2024
aba9132
Merge pull request #1353 from GSA/fix-login-gov-url-var-name-with-quotes
ccostino Oct 1, 2024
97ee4fe
cleanup
Oct 2, 2024
c792a24
cleanup
Oct 2, 2024
f720b48
Bug: Invites expiring immediately
Oct 2, 2024
ae5c0d2
Merge pull request #1343 from GSA/notify-api-1299
terrazoon Oct 2, 2024
4abd540
debug s3 partitioning
Oct 3, 2024
f87e710
Merge pull request #1356 from GSA/debug_s3_partitioning
ccostino Oct 3, 2024
e6cf8a9
Merge pull request #1354 from GSA/notify-admin-1976
ccostino Oct 3, 2024
0c100dd
Change references name in deploy tool
Oct 4, 2024
584f243
Switch back repo reference and update deploy tool for demo/prod scripts
Oct 4, 2024
497f91a
Fix deploy config references
Oct 4, 2024
f81785c
Update egress proxy deployment steps
ccostino Oct 5, 2024
37dc593
Merge pull request #1348 from GSA/rotate_secret_doc
ccostino Oct 5, 2024
b230d86
Merge pull request #1357 from GSA/fix-deploy-tool-name
ccostino Oct 9, 2024
c2f2e36
Added missing egress proxy deploy action inputs
ccostino Oct 9, 2024
a1b6047
Merge branch 'main' into update-egress-proxy
ccostino Oct 9, 2024
22bb1d0
Swap config vars for env vars
ccostino Oct 9, 2024
f644f52
Fix environment name references for CF
ccostino Oct 10, 2024
70022c0
Merge pull request #1358 from GSA/update-egress-proxy
ccostino Oct 10, 2024
3f6c362
Finalize updates for cg-cli-tools
ccostino Oct 10, 2024
9b14a63
Merge pull request #1361 from GSA/finalize-cg-tools-update
heyitsmebev Oct 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .ds.baseline
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@
"filename": "tests/app/aws/test_s3.py",
"hashed_secret": "67a74306b06d0c01624fe0d0249a570f4d093747",
"is_verified": false,
"line_number": 27,
"line_number": 29,
"is_secret": false
}
],
Expand Down Expand Up @@ -384,5 +384,5 @@
}
]
},
"generated_at": "2024-09-10T18:12:39Z"
"generated_at": "2024-09-27T16:42:53Z"
}
16 changes: 16 additions & 0 deletions .github/actions/deploy-proxy/action.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
name: Deploy egress proxy
description: Set egress space security groups and deploy proxy
inputs:
cf_org:
description: The org the target app exists in.
required: true
cf_space:
description: The space the target app exists in.
required: true
Expand All @@ -16,6 +19,19 @@ inputs:
runs:
using: composite
steps:
- name: Install cf-cli
shell: bash
run: |
curl -A "cg-deploy-action" -v -L -o cf-cli_amd64.deb 'https://packages.cloudfoundry.org/stable?release=debian64&version=v8&source=github'
sudo dpkg -i cf-cli_amd64.deb
- name: Login to cf-cli
shell: bash
run: |
cf api api.fr.cloud.gov
cf auth
- name: Target org and space
shell: bash
run: cf target -o ${{ inputs.cf_org }} -s ${{ inputs.cf_space }}
- name: Set restricted space egress
shell: bash
run: ./terraform/set_space_egress.sh -t -s ${{ inputs.cf_space }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
- name: Check for dead code
run: make dead-code
- name: Run tests with coverage
run: poetry run coverage run --omit=*/notifications_utils/*,*/migrations/* -m pytest --maxfail=10
run: poetry run coverage run --omit=*/migrations/*,*/tests/* -m pytest --maxfail=10
env:
SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
NOTIFY_E2E_TEST_EMAIL: ${{ secrets.NOTIFY_E2E_TEST_EMAIL }}
Expand All @@ -63,7 +63,7 @@ jobs:
NOTIFY_E2E_TEST_PASSWORD: ${{ secrets.NOTIFY_E2E_TEST_PASSWORD }}
- name: Check coverage threshold
# TODO get this back up to 95
run: poetry run coverage report --fail-under=95
run: poetry run coverage report -m --fail-under=91

validate-new-relic-config:
runs-on: ubuntu-latest
Expand Down
14 changes: 10 additions & 4 deletions .github/workflows/deploy-demo.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
run: poetry export --without-hashes --format=requirements.txt > requirements.txt

- name: Deploy to cloud.gov
uses: 18f/cg-deploy-action@main
uses: cloud-gov/cg-cli-tools@main
env:
DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
SECRET_KEY: ${{ secrets.SECRET_KEY }}
Expand All @@ -64,7 +64,8 @@ jobs:
cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
cf_org: gsa-tts-benefits-studio
cf_space: notify-demo
push_arguments: >-
cf_command: >-
push -f manifest.yml
--vars-file deploy-config/demo.yml
--var DANGEROUS_SALT="$DANGEROUS_SALT"
--var SECRET_KEY="$SECRET_KEY"
Expand All @@ -73,6 +74,7 @@ jobs:
--var NOTIFY_E2E_TEST_EMAIL="$NOTIFY_E2E_TEST_EMAIL"
--var NOTIFY_E2E_TEST_PASSWORD="$NOTIFY_E2E_TEST_PASSWORD"
--var LOGIN_DOT_GOV_REGISTRATION_URL="$LOGIN_DOT_GOV_REGISTRATION_URL"
--strategy rolling

- name: Check for changes to templates.json
id: changed-templates
Expand All @@ -95,6 +97,10 @@ jobs:
- name: Deploy egress proxy
if: steps.changed-egress-config.outputs.any_changed == 'true'
uses: ./.github/actions/deploy-proxy
env:
CF_USERNAME: ${{ secrets.CLOUDGOV_USERNAME }}
CF_PASSWORD: ${{ secrets.CLOUDGOV_PASSWORD }}
with:
cf_space: notify-demo
app: notify-api-demo
cf_org: gsa-tts-benefits-studio
cf_space: notify-staging
app: notify-api-staging
14 changes: 10 additions & 4 deletions .github/workflows/deploy-prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
run: poetry export --without-hashes --format=requirements.txt > requirements.txt

- name: Deploy to cloud.gov
uses: 18f/cg-deploy-action@main
uses: cloud-gov/cg-cli-tools@main
env:
DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
SECRET_KEY: ${{ secrets.SECRET_KEY }}
Expand All @@ -68,7 +68,8 @@ jobs:
cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
cf_org: gsa-tts-benefits-studio
cf_space: notify-production
push_arguments: >-
cf_command: >-
push -f manifest.yml
--vars-file deploy-config/production.yml
--var DANGEROUS_SALT="$DANGEROUS_SALT"
--var SECRET_KEY="$SECRET_KEY"
Expand All @@ -77,6 +78,7 @@ jobs:
--var NOTIFY_E2E_TEST_EMAIL="$NOTIFY_E2E_TEST_EMAIL"
--var NOTIFY_E2E_TEST_PASSWORD="$NOTIFY_E2E_TEST_PASSWORD"
--var LOGIN_DOT_GOV_REGISTRATION_URL="$LOGIN_DOT_GOV_REGISTRATION_URL"
--strategy rolling

- name: Check for changes to templates.json
id: changed-templates
Expand All @@ -99,6 +101,10 @@ jobs:
- name: Deploy egress proxy
if: steps.changed-egress-config.outputs.any_changed == 'true'
uses: ./.github/actions/deploy-proxy
env:
CF_USERNAME: ${{ secrets.CLOUDGOV_USERNAME }}
CF_PASSWORD: ${{ secrets.CLOUDGOV_PASSWORD }}
with:
cf_space: notify-production
app: notify-api-production
cf_org: gsa-tts-benefits-studio
cf_space: notify-staging
app: notify-api-staging
10 changes: 8 additions & 2 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:
run: poetry export --without-hashes --format=requirements.txt > requirements.txt

- name: Deploy to cloud.gov
uses: 18f/cg-deploy-action@main
uses: cloud-gov/cg-cli-tools@main
env:
DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
SECRET_KEY: ${{ secrets.SECRET_KEY }}
Expand All @@ -70,7 +70,8 @@ jobs:
cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
cf_org: gsa-tts-benefits-studio
cf_space: notify-staging
push_arguments: >-
cf_command: >-
push -f manifest.yml
--vars-file deploy-config/staging.yml
--var DANGEROUS_SALT="$DANGEROUS_SALT"
--var SECRET_KEY="$SECRET_KEY"
Expand All @@ -79,6 +80,7 @@ jobs:
--var NOTIFY_E2E_TEST_EMAIL="$NOTIFY_E2E_TEST_EMAIL"
--var NOTIFY_E2E_TEST_PASSWORD="$NOTIFY_E2E_TEST_PASSWORD"
--var LOGIN_DOT_GOV_REGISTRATION_URL="$LOGIN_DOT_GOV_REGISTRATION_URL"
--strategy rolling

- name: Check for changes to templates.json
id: changed-templates
Expand All @@ -101,7 +103,11 @@ jobs:
- name: Deploy egress proxy
if: steps.changed-egress-config.outputs.any_changed == 'true'
uses: ./.github/actions/deploy-proxy
env:
CF_USERNAME: ${{ secrets.CLOUDGOV_USERNAME }}
CF_PASSWORD: ${{ secrets.CLOUDGOV_PASSWORD }}
with:
cf_org: gsa-tts-benefits-studio
cf_space: notify-staging
app: notify-api-staging

Expand Down
5 changes: 3 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,10 @@ test: ## Run tests and create coverage report
poetry run black .
poetry run flake8 .
poetry run isort --check-only ./app ./tests
poetry run coverage run --omit=*/notifications_utils/*,*/migrations/* -m pytest --maxfail=10
poetry run coverage run --omit=*/migrations/*,*/tests/* -m pytest --maxfail=10

poetry run coverage report -m --fail-under=95
## TODO set this back to 95 asap
poetry run coverage report -m --fail-under=91
poetry run coverage html -d .coverage_cache

.PHONY: py-lock
Expand Down
Loading
Loading