Feature/appeals 43597 release candidate (#22004)

* APPEALS-36759 Add sms_response_content and sms_response_time notifications table

* APPEALS-37003 First pass at job refactor

* APPEALS-37003 Add comments and update kwargs for va notify service

* APPEALS-37003 Update spec files

* APPEALS-37003 Update spec

* APPEALS-37003 Ensure user in job and create email_enabled method

* APPEALS-37003 Remove question mark

* Limit max retry attempts to 5

* Add ? to boolen return methods

* jcohen/APPEALS-43706 (#21444)

* APPEALS-43706 macros added to each model file for the polymorphic relationship to be setup.

* APPEALS-43706 migration made and yet to be run.

* APPEALS-43706 migrations ran and expected columns and indexes are in table.

* Fix some spec failures unrelated to this branch

* Limit schema diff

* Add a missing tab

* Fix more unrelated failures. Also prevent ACD lever seeds from overwriting spec output

* Fix last unrelated failure.

* APPEALS-43706 linting issues in va_notify_service fixed.

---------

Co-authored-by: Jonathan Cohen <cohen_jonathan@bah.com>
Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* [APPEALS-43598] Quarterly Notification Job refactoring (#21505)

* Quarterly Notification Job refactoring

* APPEALS-43598 Fix lint issues for QuarterlyNotificationsJob refactor

* APPEALS-43598 Fix db schema.rb file

* APPEALS-43598 Fix schema.rb file

* APPEALS-43598 Refactoring QuarterlyNotificationsJob for readability

* APPEALS-43598 Refactoring QuarterlyNotificationsJob and removing extra methods

* [APPEALS-43598] Update method name

---------

Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* APPEALS-44007: Eliminate Transmission of Notifications to Deceased Veterans (#21481)

* APPEALS-44007: Writing tests to ensure adding the 'failure due to deceased' information is properly set and added to the db

* updates to pr

* pushing changes for tonight

* updating spec file to handle enabling and disabling feature flags

* Update format_message_status

* Lint roll

* fixing conflicts

* Updates to send notification spec

* Fix one of the legacy test cases

* Remove accidental push

* Fix remaining test

---------

Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* MattT/APPEALS-45235: Remove Temporary PDF Files From tmp Directory After All eFolder Upload Attempts (#21557)

* Cleanup files after uploading to VBMS

* Only delete file if it exists

* Resolve linter issue

* Update specs

* Trigger CodeClimate

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Add appeal alias for notifiable

* APPEALS-45346: Populate Polymorphic Association Columns When Creating New Notifications (#21575)

* APPEALS-45346: Populate Polymorphic Association Columns When Creating New Notifications

* fixing some errors

* updating factory

* updating tests

* updating tests and factory

* rubocop

---------

Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* APPEALS-43596 Adding JobMessageDeletionMiddleware to delete SQS messages for certain jobs

* APPEALS-43596 Adding tests

* APPEALS-43596 Fix codeclimate

* APPEALS-43714: Refactor VANotify Modules to Not Rely on Class Variables (#21596)

* APPEALS-43714: Refactor VANotify Modules to Not Rely on Class Variables

* pushing changes

* updating tests

* rubocop

* rubocop'n

* deleting repeat constants

* removing file in favor of EVENT_TYPE_FILTERS.json

* removing instances of using VA_NOTIFY_TEMPLATE_NAMES and using EVENT_TYPE_FILTERS instead

* replacing all instances of strings with constants

* fixing bug

* rubocop'n

* rubocop'n

* rubocop

---------

Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* Clean notification_events after notification report specs finish (#21662)

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* MattT/APPEALS-43725: Extract `notify_appellant` Call from `QuaterlyNotificationsJob` to Decrease Runtime (#21643)

* Group notification types as scopes

* Add new job

* Fix typo

* Add back batching

* Batch send messages

* Multithread job queueing

* Reconfigure bulk enqueueing to utilize send_notifications.fifo queue

* Update the comment on QuaterlyNotificationsJob#perform

to adhere to the RDoc standard

* Use constants for template name

* Provide a more descriptive var name

* Remove unused env var

* Fix linting issues

* Prevent error from being thrown early in init job

* Add RDoc comments to appeal_state

* Add rdoc comment to init job

* Add another rdoc comment to the init job

* Use a more specific error class whenever an appeal

cannot be located within the init job

* Add a more specific error to CaseflowJob#enqueue_batch_of_jobs

* Add test for SendNotificationJob.queue_name_suffix

* Update QuarterlyNotificationsJob tests

* Update appeal_state_spec.rb to cover new scopes

* Add spec file for NotificationInitializationJob

* Add CaseflowJob specs

* Add CaseflowJob specs

* Address rubocop issue (by ignoring it)

* Kicking off CodeClimate

* Fix some lint issues

* Freeze constant

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Move spec file to proper location

* APPEALS-43599 (#21696)

* APPEALS-43599 Refactoring SendNotificationJob

* APPEALS-43599 Remove comments

* APPEALS-43599 Suppress code smells from reek

* APPEALS-43599 Suppress code smells from reek

* jcohen/APPEALS-43727 (#21589)

* APPEALS-43727 Branch created. Extracted all the 'when' conditions in the case statement within appellant_notification.rb update_appeal_state method.

* APPEALS-43727 appellant_notification_service is created, logic from crowded method in appellant_notificaion.rb placed in newly created file, allowing redability.

* APPEALS-43727 added logic to appeal_state.rb, appellant_notification.rb updated with less logic, readable code through well named methods.

* APPEALS-43727 commit before merge from teammates work

* APPEALS-43727 removed freeze method on default status hash in appeal_state.rb. removed some rubocop disabling in appellant_notificaion.rb

* APPEALS-43727 extracted more logic from the appellant_notification module and placed it into the appeal_state class. Also got rid of rubocop declarations in app_notif and app_state as well. finally removed feature_toggle calls in appell_notif.

* APPEALS-43727 tests passing, code works

* APPEALS-43727 adding tests to appeal_state_spec to test functionality

* APPEALS-43727 rspec tests in appeal_state.

* APPEALS-43727 tests pass.

* Fixing code climate duplicate code issue

* fixing case statement

* code climate things

* rubocop

* APPEALS-43727 work completed.

* APPEALS-43727 PR comments applied. tests not in a working state.

* APPEALS-43727 PR comments applied. appeal_state_spec in working state.

* APPEALS-43727 PR comments applied. appellant_notification_spec.rb all green.

* Adjust spacing of comments in appeal_state.rb

* Utilize #external_id method over ternary

* Swap or conditional for blank?

* Fix another whitespace gap

* APPEALS-43727 PR comments applied. again after reset.

* APPEALS-43727 PR comments applied. flipped conditionals in appeal_state.rb

* Update app/models/concerns/appeal_concern.rb

* APPEALS-43727 code/test edits

* APPEALS-43727 bulk_task_reasssignment_spec reverted to orginal state, with the exception of an expectation that was on line 98

* APPEALS-43727 lint fix

---------

Co-authored-by: Jonathan Cohen <cohen_jonathan@bah.com>
Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* APPEALS-46179 (#21712)

* APPEALS-46179 Create JobExecutionTimes table

* APPEALS-46179 Add comment

* APPEALS-46179 Update schema

* APPEALS-46193 (#21753)

* APPEALS-46193 Add record to JobExecutionTime table when job is performed

* APPEALS-46193 Updating JobExecutionTimes table with an upsert query

* APPEALS-46193 Ignore JobExecutionTimes for certain jobs

* APPEALS-46193 Update ignore_job_execution_time? method

* APPEALS-46193 Adding tests

* Fix lint issues

* Update AWS dependencies

* Rollback gem reduction due to UAT issue

* Alter queue priority

* Forward error in SendNotificationJob

* Allow for notification audit records to remain in ProdTest

* Fix spec

* Fix typo in test fixture's name

* Fix lint infraction related to re-raising exceptions

* Address duplication CC error

* Reenable-privacy-mail-notifications (#21859)

* Add conditional for FOIA and Privacy Act mail

task to receives 'Privacy Act Pending' notifications

* Update spec

* Adjust some specs

* Add missing param

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Remove byebug from spec

* Revert queue tweak

* Remove duplicate method

* APPEALS-47382: Refactor Job Behavior Filtering Boolean Methods (#21898)

* initial commit

* refactoring methods to have constants instead of methods

* adding comments back in

---------

Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>
Co-authored-by: nhansen3 <noah.hansen1323@gmail.com>

* APPEALS-48342: Prevent Intentional Retries of SendNotificationJobs in ProdTest from Causing Duplicate Notification Table Entries (#21927)

* initial commit

* rubocop'n

* rubcop'n

* fixing tests

---------

Co-authored-by: nhansen3 <noah.hansen1323@gmail.com>
Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com>

* Resolve missed conflict

* MattT/APPEALS-48514 (#21959)

* Prototype of privacy act fix

* Leave privacy act states intact upon decision mailed

* Remove unused param

* Attempt updating specs to cover new behavior

* Fix attr name typo

* Add missing bang

* Skip reek issue

The issue has been deemed to be minor enough to not warrant having to reinvent everything for now.

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Fix quarterly specs by add in necessary hearings

* Adjust spacing/ordering of gems (#21978)

* MattT/APPEALS-49085 (#21960)

* Filter out appeals whose hearings have been held from hearing_scheduled scope

* Fix whitespace issues

* Fix failing specs caused by an oversight on my part

* Fix lint issue

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* MattT/APPEALS-49212 (#21991)

* Add module to set hearing_scheduled to false whenever hearings are marked as held

* Add module to set hearing_scheduled to false whenever hearings are marked as held

* Add step to NightlySyncsJob to sync hearing dispositions and appeal stateS

* Add comment to new method

* Use constant for disposition type

* Fix typo in accessor

* Add spec for new appeal state action

* Create user in top scope

* Expand nighlty syncs job to check all scheduled hearing disposition types.

No show was left off as we don't have a requirement around it yet.

* Add nightly sync job spec

* Add disp task spec

* Add specs targeting the hearings controller

* Add sync script

* Eager load constants

* Add distinct

* Remove appeal_docketed from being reset

* Add script to adjust ama hearing statuses

* Correct mistype

* Add appeal_type clause

* Fix specs

* Lint fixes

* Flip conditionals in specs

* Fix var name

* Fix var

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Backfill addition to script

* Add AppealState record for newly docketed LegacyAppeals (#21999)

* APPEALS-48966 Adding AppealState record for recently docketed appeals

* APPEALS-48966 Refactor code and add comments

* Delete extraneous files

* Revert to using constants value

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Fix typo

* Fix typo

* Fix seed script

* Fix seed script

* Ensure cleanup

* Change copy for Appeal Docketed status (#22006)

* Change copy for Appeal Docketed status

* Change case

---------

Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>

* Use status text value instead of key

---------

Co-authored-by: Jeff Marks <jeff.t.marks@gmail.com>
Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com>
Co-authored-by: Jonathan Cohen <121630615+JCohDev@users.noreply.github.com>
Co-authored-by: Jonathan Cohen <cohen_jonathan@bah.com>
Co-authored-by: prernadevbah <132498915+prernadevbah@users.noreply.github.com>
Co-authored-by: noahhansen-gov <166541737+noahhansen-gov@users.noreply.github.com>
Co-authored-by: Noah Hansen <nhansen@Noahs-MacBook-Pro.local>
Co-authored-by: Prerna Devulapalli <prerna.devulapalli@va.gov>
Co-authored-by: nhansen3 <noah.hansen1323@gmail.com>
Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com>

app/jobs/quarterly_notifications_job.rb

@@ -21,13 +21,15 @@ def perform
 
     begin
       NOTIFICATION_TYPES.each_key do |notification_type|
+        status_text = NOTIFICATION_TYPES[notification_type.to_sym]
+
         jobs = AppealState.eligible_for_quarterly.send(notification_type).pluck(:appeal_id, :appeal_type)
           .map do |related_appeal_info|
           NotificationInitializationJob.new(
             appeal_id: related_appeal_info.first,
             appeal_type: related_appeal_info.last,
             template_name: Constants.EVENT_TYPE_FILTERS.quarterly_notification,
-            appeal_status: notification_type.to_s
+            appeal_status: status_text
           )
         end
 

client/constants/QUARTERLY_STATUSES.json

@@ -6,5 +6,5 @@
   "ihp_pending": "VSO IHP Pending",
   "hearing_to_be_rescheduled": "Hearing to be Rescheduled",
   "hearing_to_be_rescheduled_privacy_pending": "Hearing to be Rescheduled / Privacy Act Pending",
-  "appeal_docketed": "Appeal Docketed"
+  "appeal_docketed": "docketed"
 }

lib/tasks/appeal_state_synchronizer.rake

@@ -121,23 +121,38 @@ namespace :appeal_state_synchronizer do
     end
   end
 
-  def backfill_appeal_information
-    updates_to_make = {}
-
-    Notification.where(notifiable_id: nil)
-      .or(Notification.where(notifiable_type: nil))
-      .find_in_batches(batch_size: 10_000) do |notification_batch|
-      notification_batch.index_by(&:id).each do |id, notification_row|
-        appeal = Appeal.find_appeal_by_uuid_or_find_or_create_legacy_appeal_by_vacols_id(notification_row.appeals_id)
-        updates_to_make[id] = { id: id, notifiable: appeal }
-      end
+  def ama_backfill_sql
+    <<-SQL
+      UPDATE notifications n
+      SET notifiable_id = a.id,
+        notifiable_type = 'Appeal'
+      FROM appeals a
+      WHERE n.appeals_id = a.uuid::text
+    SQL
+  end
 
-      Notification.update(updates_to_make.keys, updates_to_make.values)
+  def legacy_backfill_sql
+    <<-SQL
+      UPDATE notifications n
+      SET notifiable_id = la.id,
+        notifiable_type = 'LegacyAppeal'
+      FROM legacy_appeals la
+      WHERE n.appeals_id = la.vacols_id
+    SQL
+  end
+
+  def backfill_appeal_information
+    begin
+      # Temporarily Bump Up Timeout
+      ActiveRecord::Base.connection.execute("SET statement_timeout = '600s'")
 
+      ActiveRecord::Base.connection.execute(ama_backfill_sql)
+      ActiveRecord::Base.connection.execute(legacy_backfill_sql)
     rescue StandardError => error
-      Rails.logger.error("#{notification.id} couldn't have its appeal set because of #{error.message}")
+      Rails.logger.error("#{error.class}: #{error.message}\n#{error.backtrace}")
     ensure
-      updates_to_make = {}
+      # Set Timeout Back
+      ActiveRecord::Base.connection.execute("SET statement_timeout = '30s'")
     end
   end
 end