LG-14320 Add deactivate IPP cancelled profiles task #11180

shanechesnutt-ft · 2024-08-30T17:06:39Z

🎫 Ticket

Link to the relevant ticket:
LG-14320

🛠 Summary of changes

Added a script to deactivate profiles that are considered to be IPP verification cancelled state.

📜 Testing Plan

Scenario: Running backfill script

Create data using the seed_users_with_profiles.rb seed file

bundle exec rake seed:broken_profiles

Check that number of profile in broken state is greater than 0 using rails console

    InPersonEnrollment.where(status: [:expired, :cancelled, :failed]).
      includes(:profile).
      where.not(profile: { in_person_verification_pending_at: nil }).
      count

Note: It might be good to setup the seeder to create a few pending profiles for testing to ensure those do not get updated.

Run the backfill script to backup data without updating profiles

bundle exec rake profiles:backfill_deactivated_ipp_verification_cancelled UPDATE_PROFILES=false

Verify that log output contains the expected profiles information (id, deactivation_reason, in_person_verification_pending_at) is contained in it
Copy the log output and save it to a file
Run the backfill script data updating profiles

bundle exec rake profiles:backfill_deactivated_ipp_verification_cancelled UPDATE_PROFILES=true

Check the number of profile in broken state is 0 using rails console

    InPersonEnrollment.where(status: [:expired, :cancelled, :failed]).
      includes(:profile).
      where.not(profile: { in_person_verification_pending_at: nil }).
      count

Scenario: Running rollback script

Run the rollback script

bundle exec rake profiles:rollback_backfill_deactivated_ipp_verification_cancelled BACKFILL_OUTPUT="$(cat output)"

Check the number of profile in broken state is back to the original count before the backfill script ran using rails console

    InPersonEnrollment.where(status: [:expired, :cancelled, :failed]).
      includes(:profile).
      where.not(profile: { in_person_verification_pending_at: nil }).
      count

gina-yamada · 2024-08-30T17:49:57Z

lib/tasks/seed_users_with_profiles.rake

+    enrollment.profile = profile
+    enrollment.save!
+  end
+end


For Run the backfill script to backup data without updating profiles - we are just running that to 1. confirm batches are working and 2. ensure the print out is working as expected, correct? What do you think about adding a sentence on the testing plan to confirm both? (I am adding it here so it can be resolved.)

gina-yamada · 2024-09-03T14:35:57Z

lib/tasks/seed_users_with_profiles.rake

@@ -0,0 +1,86 @@
+# frozen_string_literal: true


Do you think we should create data that should not be updated to ensure/check that we are not updating establishing, pending, and passed enrollments and/or those enrollments that don't have a timestamp for the profile? What are your thoughts on this? Overkill?

I like this idea if it's not too much trouble!

lib/tasks/backfill_in_person_pending_at.rake

aduth · 2024-09-03T19:20:17Z

lib/tasks/seed_users_with_profiles.rake

If it doesn't exist already, can we make a follow-up ticket to remove these extra tasks once it's been run successfully in production?

I don't think we have a ticket for this yet. I will make sure it is created.

shanechesnutt-ft · 2024-09-04T14:39:47Z

I just wanted to note that I am going to delete lib/tasks/seed_users_with_profiles.rake before this is merged

gina-yamada · 2024-09-04T14:42:37Z

I created LG-14391 Clean up Backfill In Person Pending At Rake Task to delete this script so it would not get forgotten.

gina-yamada · 2024-09-04T15:48:26Z

lib/tasks/backfill_in_person_pending_at.rake

@@ -75,4 +75,67 @@ namespace :profiles do

    warn "backfill_in_person_verification_pending_at left #{profiles.count} rows"
  end
+
+  ##
+  # Usage:


Checked prod on 09/04/24 at approx. 11:30ET

Expired (3786), Cancelled *813), Failed (587) - Total 5186 in prod

Establishing 1313

jennyverdeyen

Ran through the testing plan and the tasks work as described. Looks great!

eileen-nava

I didn't manually test this, but I reviewed it and everything looks good. Approved. ✅

eileen-nava · 2024-09-04T18:39:03Z

lib/tasks/backfill_in_person_pending_at.rake

+
+    warn "Updating #{profile_data.count} records"
+    profile_data.in_groups_of(1000, false) do |batch|
+      batch.each do |profile_datum|


Nice use of datum. 👏🏻

gina-yamada · 2024-09-05T18:33:54Z

I did locally testing- and it worked as expected. I also worked through this with Shane a bit so happy others approved.

shanechesnutt-ft · 2024-09-05T20:25:10Z

I did a squash and removed commits from my branch. I removed the seed file that was added testing. Need a quick double check then I can merge.

eileen-nava · 2024-09-05T20:40:16Z

@shanechesnutt-ft I did a once-over just now. Looks good. 👍🏻

jennyverdeyen · 2024-09-06T14:16:24Z

Ran through the test plan again after your changes. Looks good, script works as described!

matthinz

LGTM

n1zyy · 2024-09-06T19:11:38Z

lib/tasks/backfill_in_person_pending_at.rake

+          timestamp = profile.in_person_verification_pending_at
+
+          warn "#{profile.id},#{profile.deactivation_reason},#{timestamp}"
+          if update_profiles


TBH I wouldn't change it at this point, but I feel like it would be a tiny bit more ideal if the warn on line 101 occurred inside this conditional, or there was otherwise some indication if update_profiles was false. It'd otherwise be easy to miss that this script didn't change anything if the environment variable wasn't set.

But, it looks like this has already been extensively tested, so it doesn't feel like it's worth changing. I'd just make sure someone confirms the script was successful, which was probably part of the plan all along anyway. 👍

Good callout! I did add checks in the roll plan for verifying the data.

changelog: Internal, Data Normalization, Add a backfill script to deactivate all profiles consider to be in ipp verfication cancelled state. co-authored-by: Andrew Duthie <andrew.duthie@gsa.gov>

gina-yamada reviewed Aug 30, 2024

View reviewed changes

gina-yamada reviewed Sep 3, 2024

View reviewed changes

lib/tasks/backfill_in_person_pending_at.rake Show resolved Hide resolved

shanechesnutt-ft requested a review from aduth September 3, 2024 19:03

aduth approved these changes Sep 3, 2024

View reviewed changes

shanechesnutt-ft marked this pull request as ready for review September 4, 2024 15:48

gina-yamada reviewed Sep 4, 2024

View reviewed changes

jennyverdeyen self-requested a review September 4, 2024 17:04

jennyverdeyen approved these changes Sep 4, 2024

View reviewed changes

eileen-nava approved these changes Sep 4, 2024

View reviewed changes

shanechesnutt-ft force-pushed the sc/LG-14320 branch from bd9f471 to 48326e0 Compare September 4, 2024 19:21

shanechesnutt-ft force-pushed the sc/LG-14320 branch 2 times, most recently from 60a5e0f to 94eb061 Compare September 5, 2024 20:11

shanechesnutt-ft force-pushed the sc/LG-14320 branch from 94eb061 to a3fc305 Compare September 5, 2024 21:03

matthinz approved these changes Sep 6, 2024

View reviewed changes

shanechesnutt-ft force-pushed the sc/LG-14320 branch 2 times, most recently from 5c86f8a to be63601 Compare September 6, 2024 19:07

n1zyy approved these changes Sep 6, 2024

View reviewed changes

LG-14320 Add deactivate IPP cancelled profiles task

367fe4e

changelog: Internal, Data Normalization, Add a backfill script to deactivate all profiles consider to be in ipp verfication cancelled state. co-authored-by: Andrew Duthie <andrew.duthie@gsa.gov>

shanechesnutt-ft force-pushed the sc/LG-14320 branch from be63601 to 367fe4e Compare September 6, 2024 19:47

shanechesnutt-ft merged commit dd28eba into main Sep 6, 2024
2 checks passed

shanechesnutt-ft deleted the sc/LG-14320 branch September 6, 2024 20:00

mdiarra3 mentioned this pull request Sep 10, 2024

Deploy RC 413 to Production #11222

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LG-14320 Add deactivate IPP cancelled profiles task #11180

LG-14320 Add deactivate IPP cancelled profiles task #11180

shanechesnutt-ft commented Aug 30, 2024 •

edited

Loading

gina-yamada Aug 30, 2024

gina-yamada Sep 3, 2024

jennyverdeyen Sep 4, 2024

aduth Sep 3, 2024

shanechesnutt-ft Sep 4, 2024

shanechesnutt-ft commented Sep 4, 2024

gina-yamada commented Sep 4, 2024

gina-yamada Sep 4, 2024 •

edited

Loading

jennyverdeyen left a comment

eileen-nava left a comment

eileen-nava Sep 4, 2024

gina-yamada commented Sep 5, 2024

shanechesnutt-ft commented Sep 5, 2024

eileen-nava commented Sep 5, 2024

jennyverdeyen commented Sep 6, 2024

matthinz left a comment

n1zyy Sep 6, 2024

shanechesnutt-ft Sep 6, 2024

LG-14320 Add deactivate IPP cancelled profiles task #11180

LG-14320 Add deactivate IPP cancelled profiles task #11180

Conversation

shanechesnutt-ft commented Aug 30, 2024 • edited Loading

🎫 Ticket

🛠 Summary of changes

📜 Testing Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanechesnutt-ft commented Sep 4, 2024

gina-yamada commented Sep 4, 2024

gina-yamada Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

jennyverdeyen left a comment

Choose a reason for hiding this comment

eileen-nava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gina-yamada commented Sep 5, 2024

shanechesnutt-ft commented Sep 5, 2024

eileen-nava commented Sep 5, 2024

jennyverdeyen commented Sep 6, 2024

matthinz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanechesnutt-ft commented Aug 30, 2024 •

edited

Loading

gina-yamada Sep 4, 2024 •

edited

Loading