Feature Tracking PR for APPEALS-50887 (#22244)

* Remove transaction_wrapper * Remove extra newline * APPEALS-51045: Remove ReceiveNotificationJob (#22207) Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> * MattT/APPEALS-51115: Initialize FIFO SQS Queues Locally (#22182) * Set local env to utilize FIFO queues where appropriate FIFO queues are configured with the same attributes as our queues in higher environments. * Resolve failing test * Remove defunct test * Update make commands --------- Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com> * APPEALS-51847: Institute Database Migration for Establishing 'sms_status_reason' and 'email_status_reason' Columns (#22208) * adding and running migration * adding comments to migration and re running db:migrate * pushing up original scheam * committing schema * small edit to migration and deleting unnecessary columns from schema --------- Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com> * removing job and all mentions of job (#22212) Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> Co-authored-by: Matthew Thornton <99351305+ThorntonMatthew@users.noreply.github.com> * MattT/APPEALS-51059: Alter Select Quarterly Statuses (#22239) * Alter status messages * Save more info in fake response * Swap keys * Add missing param * Test fixes. Also, setting appeal_docketed to false whenever appeals are decided or cancelled * Fix fasterer issue --------- Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com> * MattT/APPEALS-51101: Rewrite ProcessNotificationStatusUpdateJob to Consume Messages from an SQS Queue (#22247) * Update fake response body * Try enabling localstack in CI * Init SqsService and alter most/all of the ProcessNotificationStatusUpdatesJob * Create SQS queues in test env * Change port formatting * Mount docker socket * Try localstack hostname * Wait for localstack in CI * Add more yarddoc comments * Log errors and number of messages processed * Add custom error classes * Add yarddoc comments to SqsService class * Create SqsService spec file * Redo much of ProcessNotificationStatusUpdatesJob's spec file * Disable line length checks for yarddoc comments --------- Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com> * APPEALS-51087: Rewrite Api::V1::VaNotifyController#notification_update Controller Action to No Longer Utilize Redis (#22295) * initial push * rubocop'n * fixing tests? * rubocop'n * addressing most of matt's comments * removing yard doc files and keeping comment * rubocop --------- Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> * Remove extra attribute * Fix incorrect log counts * Fix log statement * Fix typo * APPEALS-53603: Modify logic to always show recipient information if present (#22414) * initial commit * making rspec tests happy * linting * adding tests back and refactoring * small char error --------- Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> * Sync decided appeals with states table (#22434) * APPEALS-52882 Sync decided appeals decision_mailed status * APPEALS-52882 Fix lint issues * APPEALS-52882 Fix codeclimate issues * APPEALS-52882 Address PR feedback * APPEALS-52882 Address PR feedback * Jcohen/APPEALS-52861 (#22450) * APPEALS-52861 Bug fix implemented. * APPEALS-52861 cleaned up some code through aliasing a common method name between appeal_types, eliminating a check that we do in the AppealDecisionMailed module. * APPEALS-52861 fixed method name error. * APPEALS-52861 fixed some tests. * APPEALS-52861 fixed More tests. * APPEALS-52861 fixed More tests. * APPEALS-52861 feature branch changes merged in with working branch and method name changes extracted. * APPEALS-52861 feature branch changes merged in with working branch and method name changes extracted, included case statement. * APPEALS-52861 moved case statement to another file to be the boolean check for constested status on an appeal. --------- Co-authored-by: Jonathan Cohen <cohen_jonathan@bah.com> Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com> * APPEALS-52892: For all legacy appeal states/notifications check status in VACOLS (#22455) * initial commit * addressing comments * small change * making tests happy * fixing spec tests * rubocop'n * reverting --------- Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com> * APPEALS-55159: Parameter Requirements Are Too Stringent Given What We Will Expect from VA Notify (#22561) * APPEALS-55159 * updating tests * APPEALS-55159 Refactor permitted and required params * APPEALS-55159 Fix linting issue --------- Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> Co-authored-by: msteele <Marc.Steele@va.gov> Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com> * Adding rake task to sync decided appeals (#22492) * APPEALS-52872 Add rake task to sync decided appeals * APPEALS-52872 Remove byebug * APPEALS-52872 Fixing lint errors * APPEALS-52872 Addressing performance issues * APPEALS-52872 Addressing performance issues * APPEALS-52872 Fix ENV variable update * APPEALS-52872 Parallelizing AppealState updates * APPEALS-52872 Remove byebug --------- Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com> * Prevent Prodtest from contacting VA Notify Staging (#22629) * APPEALS-54918 Update VA Notify fake service and initializer * APPEALS-54918 Add CASEFLOW_BASE_URL env var in demo/dev * APPEALS-54918 Add seed file for API Key and added to main seed execution * APPEALS-54918 Add missing argument to method definition * APPEALS-54918 Update "to" params and fix linting issues * APPEALS-54918 Prevent post requests in test env * Rename ApiKeys seed file * Fix linting error on require statement * Add argument for call within sync_review_job.rb and minor refactor * Inherit seed from base, align with AMA for seed file placement --------- Co-authored-by: Matthew Thornton <ThorntonMatthew@users.noreply.github.com> Co-authored-by: noahhansen-gov <166541737+noahhansen-gov@users.noreply.github.com> Co-authored-by: nhansen3 <noah.hansen1323@gmail.com> Co-authored-by: prernadevbah <132498915+prernadevbah@users.noreply.github.com> Co-authored-by: msteele <Marc.Steele@va.gov> Co-authored-by: Jonathan Cohen <121630615+JCohDev@users.noreply.github.com> Co-authored-by: Jonathan Cohen <cohen_jonathan@bah.com> Co-authored-by: Marc Steele <71673522+msteele96@users.noreply.github.com>
department-of-veterans-affairs · Sep 9, 2024 · 386a1ab · 386a1ab
1 parent 0d05a95
commit 386a1ab
Show file tree

Hide file tree

Showing 51 changed files with 1,280 additions and 983 deletions.
diff --git a/.github/workflows/workflow.yml b/.github/workflows/workflow.yml
@@ -45,6 +45,15 @@ jobs:
         ports:
           - 1521:1521
 
+      localstack:
+        image: localstack/localstack:0.14.5
+        ports:
+          - 4566:4566
+        env:
+          SERVICES: "sqs"
+        volumes:
+          - /var/run/docker.sock:/var/run/docker.sock
+
     strategy:
       fail-fast: false
       matrix:
@@ -188,6 +197,9 @@ jobs:
       - name: "Wait for database"
         run: dockerize -wait tcp://postgres:5432 -timeout 1m
 
+      - name: "Wait for localstack"
+        run: dockerize -wait tcp://localstack:4566 -timeout 30s
+
       - name: "Wait for FACOLS"
         run: ./ci-bin/capture-log "bundle exec rake local:vacols:wait_for_connection"
 

diff --git a/Makefile.example b/Makefile.example
@@ -287,7 +287,7 @@ one-test:  ## run the rspec test passed in
 	bundle exec rspec $(RUN_ARGS)
 
 run-all-queues: ## start shoryuken with all queues
-	bundle exec shoryuken -q caseflow_development_send_notifications caseflow_development_high_priority caseflow_development_low_priority  -R
+	bundle exec shoryuken -q caseflow_development_send_notifications.fifo caseflow_development_high_priority caseflow_development_low_priority  -R
 
 run-low-priority: ## start shoryuken with just the low priority queue
 	bundle exec shoryuken -q caseflow_development_low_priority  -R
@@ -296,7 +296,7 @@ run-high-priority: ## start shoryuken with just the high priority queue
 	bundle exec shoryuken -q caseflow_development_high_priority  -R
 
 run-send-notifications: ## start shoryuken with just the send_notification queue
-	bundle exec shoryuken -q caseflow_development_send_notifications  -R
+	bundle exec shoryuken -q caseflow_development_send_notifications.fifo  -R
 
 jest: ## Run jest tests
 	cd client && yarn jest

diff --git a/app/controllers/api/v1/va_notify_controller.rb b/app/controllers/api/v1/va_notify_controller.rb
@@ -1,46 +1,63 @@
 # frozen_string_literal: true
 
 class Api::V1::VaNotifyController < Api::ApplicationController
-  # Purpose: POST request to VA Notify API to update status for a Notification entry
+  # Purpose: POST request to VA Notify API to update status for a Notification entry.
   #
   # Params: Params content can be found at https://vajira.max.gov/browse/APPEALS-21021
   #
   # Response: Update corresponding Notification status
   def notifications_update
-    send "#{required_params[:notification_type]}_update"
+    send_sqs_message
+    render json: {
+      message: "#{params['notification_type']} Notification successfully updated: ID #{params['id']}"
+    }
+  rescue StandardError => error
+    log_error(error, params["id"], params["notification_type"])
+    render json: { error: error.message }, status: :bad_request
   end
 
   private
 
-  # Purpose: Finds and updates notification if type is email
-  #
-  # Params: Params content can be found at https://vajira.max.gov/browse/APPEALS-21021
-  #
-  # Response: Update corresponding email Notification status
-  def email_update
-    redis.set("email_update:#{required_params[:id]}:#{required_params[:status]}", 0)
-
-    render json: { message: "Email notification successfully updated: ID #{required_params[:id]}" }
+  def va_notify_params
+    params.permit(:id, :notification_type, :status, :status_reason, :to)
   end
 
-  # Purpose: Finds and updates notification if type is SMS
-  #
-  # Params: Params content can be found at https://vajira.max.gov/browse/APPEALS-21021
-  #
-  # Response: Update corresponding SMS Notification status
-  def sms_update
-    redis.set("sms_update:#{required_params[:id]}:#{required_params[:status]}", 0)
-
-    render json: { message: "SMS notification successfully updated: ID #{required_params[:id]}" }
+  def build_message_body
+    id_param, notification_type_param, status_param = va_notify_params.require([:id, :notification_type, :status])
+
+    {
+      external_id: id_param,
+      notification_type: notification_type_param,
+      recipient: va_notify_params[:to],
+      status: status_param,
+      status_reason: va_notify_params[:status_reason]
+    }
+  rescue StandardError => error
+    raise error
   end
 
-  def required_params
-    id_param, notification_type_param, status_param = params.require([:id, :notification_type, :status])
+  def build_sqs_message
+    message_body = build_message_body.to_json
+
+    {
+      queue_url: SqsService.find_queue_url_by_name(name: "receive_notifications"),
+      message_body: message_body,
+      message_deduplication_id: Digest::SHA256.hexdigest(message_body),
+      message_group_id: Constants.VA_NOTIFY_CONSTANTS.message_group_id
+    }
+  rescue StandardError => error
+    raise error
+  end
 
-    { id: id_param, notification_type: notification_type_param, status: status_param }
+  def send_sqs_message
+    sqs = SqsService.sqs_client
+    sqs.send_message(build_sqs_message)
   end
 
-  def redis
-    @redis ||= Redis.new(url: Rails.application.secrets.redis_url_cache)
+  def log_error(error, external_id, notification_type)
+    Rails.logger.error("#{error.message}\n#{error.backtrace.join("\n")}\n \
+                       external_id: #{external_id}\n \
+                       notification_type: #{notification_type}")
+    Raven.capture_exception(error)
   end
 end
diff --git a/app/helpers/sync_decided_appeals_helper.rb b/app/helpers/sync_decided_appeals_helper.rb
@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+##
+# Helper to sync the decided appeals and their decision_mailed status
+
+module SyncDecidedAppealsHelper
+  VACOLS_BATCH_PROCESS_LIMIT = ENV["VACOLS_QUERY_BATCH_SIZE"] || 800
+
+  # Syncs the decision_mailed status of Legacy Appeals with a decision made
+  def sync_decided_appeals
+    begin
+      # Join query to retrieve Legacy AppealState ids and corresponding vacols_id
+      appeal_state_ids = AppealState.legacy.where(decision_mailed: false)
+        .joins(:legacy_appeal).preload(:legacy_appeal)
+        .pluck(:id, :vacols_id)
+
+      appeal_state_ids_hash = appeal_state_ids.to_h
+
+      vacols_decision_dates = get_decision_dates(appeal_state_ids_hash.values).to_h
+
+      ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
+        Parallel.each(appeal_state_ids_hash, in_threads: 4) do |appeal_state_hash|
+          appeal_state_id = appeal_state_hash[0]
+          vacols_id = appeal_state_hash[1]
+          # If there is a decision date on the VACOLS record,
+          # update the decision_mailed status on the AppealState to true
+          if vacols_decision_dates[vacols_id].present?
+            AppealState.find(appeal_state_id).decision_mailed_appeal_state_update_action!
+          end
+        end
+      end
+    rescue StandardError => error
+      Rails.logger.error("#{error.class}: #{error.message}\n#{error.backtrace}")
+
+      # Re-raising the error so it can be caught in the NightlySyncsJob report
+      raise error
+    end
+  end
+
+  # Method to retrieve the decision dates from VACOLS in batches
+  # params: vacols_ids
+  # Returns: Hash containing the key, value pair of vacols_id, decision_date
+  def get_decision_dates(vacols_ids)
+    begin
+      decision_dates = {}
+
+      # Query VACOLS in batches
+      vacols_ids.in_groups_of(VACOLS_BATCH_PROCESS_LIMIT.to_i) do |vacols_id|
+        VACOLS::Case.where(bfkey: vacols_id).each do |vacols_record|
+          decision_dates[vacols_record[:bfkey]] = vacols_record[:bfddec]
+        end
+      end
+
+      decision_dates
+    rescue ActiveRecord::RecordNotFound
+      []
+    end
+  end
+
+  def get_vacols_ids(legacy_appeal_states)
+    begin
+      vacols_ids = {}
+
+      legacy_appeal_states.each do |appeal_state|
+        legacy_appeal = LegacyAppeal.find(appeal_state.appeal_id)
+
+        # Find the VACOLS record associated with the LegacyAppeal
+        vacols_ids << { appeal_state.id.to_s => (legacy_appeal[:vacols_id]).to_s }
+      end
+
+      vacols_ids
+    rescue ActiveRecord::RecordNotFound
+      {}
+    end
+  end
+end
diff --git a/app/jobs/nightly_syncs_job.rb b/app/jobs/nightly_syncs_job.rb
@@ -6,6 +6,7 @@
 class NightlySyncsJob < CaseflowJob
   queue_with_priority :low_priority
   application_attr :queue # arbitrary
+  include SyncDecidedAppealsHelper
 
   def perform
     RequestStore.store[:current_user] = User.system_user
@@ -16,6 +17,7 @@ def perform
     sync_vacols_users
     sync_decision_review_tasks
     sync_bgs_attorneys
+    sync_all_decided_appeals
 
     slack_service.send_notification(@slack_report.join("\n"), self.class.name) if @slack_report.any?
   end
@@ -84,6 +86,14 @@ def sync_bgs_attorneys
     @slack_report << "*Fatal error in sync_bgs_attorneys:* #{error}"
   end
 
+  def sync_all_decided_appeals
+    begin
+      sync_decided_appeals
+    rescue StandardError => error
+      @slack_report << "*Fatal error in sync_decided_appeals* #{error}"
+    end
+  end
+
   def dangling_legacy_appeals
     reporter = LegacyAppealsWithNoVacolsCase.new
     reporter.call
@@ -105,5 +115,7 @@ def sync_hearing_states
         state.scheduled_in_error_appeal_state_update_action!
       end
     end
+  rescue StandardError => error
+    @slack_report << "*Fatal error in sync_hearing_states* #{error}"
   end
 end
diff --git a/app/jobs/process_decision_document_job.rb b/app/jobs/process_decision_document_job.rb
@@ -4,10 +4,10 @@ class ProcessDecisionDocumentJob < CaseflowJob
   queue_with_priority :low_priority
   application_attr :intake
 
-  def perform(decision_document_id, mail_package = nil)
+  def perform(decision_document_id, contested, mail_package = nil)
     RequestStore.store[:application] = "idt"
     RequestStore.store[:current_user] = User.system_user
 
-    DecisionDocument.find(decision_document_id).process!(mail_package)
+    DecisionDocument.find(decision_document_id).process!(contested, mail_package)
   end
 end