feat: Cancel Job Attachments session action when transfer rates drop below threshold

[gahyusuh]

What was the problem/requirement? (What/Why)

During the SYNC_INPUT_JOB_ATTACHMENTS session actions, there might be a potential for the download operation to get stuck or stall due to unexpected issues. Currently, there is no mechanism to monitor or detect such scenarios. This poses a risk, especially when downloading many/large files or using slow/unstable network.

What was the solution? (How)

• Leverages the on_downloading_files callback function, which is passed to the AssetSync's sync_inputs function to track the the file transfer progress. This callback is intended to be called at regular intervals - 1 second at the longest.
• Monitors the transfer rate from the progress report delivered via the callback function. If it detects a series of alarmingly low transfer rates (< LOW_TRANSFER_RATE_THRESHOLD), and if the count exceeds the specified threshold (>= LOW_TRANSFER_COUNT_THRESHOLD), cancels the download and fails the current (SYNC_INPUT_JOB_ATTACHMENTS) action.

What is the impact of this change?

Worker agent can now detect and halt potentially stuck SYNC_INPUT_JOB_ATTACHMENTS session actions. This ensures that resources are not wasted on prolonged stalled operations, and that users are timely notified about the issues.

How was this change tested?

• hatch run lint && hatch run test

• Run an end-to-end test (submitting a non-rendering job bundle --> running a CMF)

  • Happy-path: ensured that there were no issues/errors until output files were generated as expected.
  • When Job Attachment's AssetSync (intentionally) reported the download speed lower than the speed threshold for the time longer than the time threshold:

  Canceled sessionaction-674e8fc017e24d7b94f8dd98eefbbf4f-1 as NEVER_ATTEMPTED
  INFO     [session-674e8fc017e24d7b94f8dd98eefbbf4f] [sessionaction-674e8fc017e24d7b94f8dd98eefbbf4f-0] (job.sync_input_job_attachments()): Action completed as FAILED
  INFO     Synchronizing with service (sending UpdateWorkerSchedule)
  INFO     Updating actions: {'sessionaction-674e8fc017e24d7b94f8dd98eefbbf4f-0': {'startedAt': datetime.datetime(2024, 1, 30, 17, 43, 49, 135243, tzinfo=datetime.timezone.utc), 'completedStatus': 'FAILED', 'progressMessage': 'Input syncing failed due to successive low transfer rates (< 10 Kb/s). The transfer rate was below the threshold for the last 1 minute.', 'endedAt': datetime.datetime(2024, 1, 30, 17, 44, 16, 518941, tzinfo=datetime.timezone.utc)}, 'sessionaction-674e8fc017e24d7b94f8dd98eefbbf4f-1': {'completedStatus': 'NEVER_ATTEMPTED', 'progressMessage': 'Input syncing failed due to successive low transfer rates (< 10 Kb/s). The transfer rate was below the threshold for the last 1 minute.'}}
  

  Also, confirmed that a telemetry data capturing this sync-inputs failure was sent correctly.

  {
    "event_timestamp":1706636627000,
    "event_type":"com.amazon.rum.deadline.worker_agent.sync_inputs_failure",
    "event_id":"16c2f820-6d7d-40b5-b777-df2f565a93d1",
    "event_version":"1.0.0",
    "log_stream":"2024-01-30T10",
    "application_id":"00cf4815-a611-42f0-85ac-1e05b995e6e6",
    "metadata": {
      "version":"0.20.0",
      "osName":"Linux",
      "osVersion":"5.10.205-172.807.amzn2int.x86_64",
      "domain":"*.madstudio.aws.dev",
      "service":"deadline-cloud-worker-agent",
      "python_version":"3.10.8",
      "countryCode":"US",
      "subdivisionCode":"OR"
    },
    "user_details":{
      "sessionId":"779308b4-9b63-4c94-8e60-0da0a0155169",
      "userId":"e726a3a4-d7c4-4e32-9610-dc333200a625"
    },
    "event_details":{
      "queue_id":"queue-a0c93fade6de47009999b273cc5b6249",
      "failure_reason":"Insufficient download speed: Input syncing failed due to successive low transfer rates (< 10 Kb/s). The transfer rate was below the threshold for the last 1 minute."
    }
  }
  

Was this change documented?

No.

Is this a breaking change?

No.

[mwiebe]

Can you put this rate in based on the LOW_TRANSFER_RATE_THRESHOLD value? If that constant is changed, this message will be inaccurate, and that could be confusing to diagnose.

[gahyusuh]

Sure, I have addressed it in the revision.

[mwiebe]

The same applies to the "1 minute" here, if the constant defining that is changed, this could be confusing.

[mwiebe]

Same comment about the message to constant consistency here, maybe it should be taking the message from the ActionStatus.

[gahyusuh]

I have updated these option names, as I was getting the below warnings when running hatch run fmt or hatch run lint

warning: The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the following options in `pyproject.toml`:
  - 'ignore' -> 'lint.ignore'
  - 'isort' -> 'lint.isort'

[marofke]

LGTM, great to see some extra safeguards being added! 🚢 🦈