SFTP queue setup #50

somesylvie · 2024-06-10T20:40:33Z

Queue and trigger setup for SFTP ingestion

This PR adds local and cloud queues, plus a queue reader to read and handle messages from the queue. In cloud environments, when a file is added to an import folder, an event automatically enqueues a file create event on the queue.

Issue

CDCgov/trusted-intermediary#1076

Checklist

I have added tests to cover my changes
I have added logging where useful (with appropriate log level)
I have added GoDocs where required
I have updated the documentation accordingly

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

Co-Authored-By: halprin <halprin@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com> Co-Authored-By: rajeev-flexion <154462164+rajeev-flexion@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: halprin <halprin@users.noreply.github.com>

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

src/senders/json_responses.go

Co-authored-by: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

sonarcloud · 2024-06-12T19:29:59Z

Quality Gate passed

Issues
18 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

somesylvie and others added 30 commits June 4, 2024 12:39

Set up local testing queue

9567c2a

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

let's use a local queue

78f1fab

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

let's read a message from our local queue

f8f6131

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

clean up error messages and returns

5751e00

Co-Authored-By: halprin <halprin@users.noreply.github.com>

use the message to read the file

0e6ce2f

Refactored to drive through use case

41de16f

Start refactoring

56f47f8

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Parse Event.subject to get blob path

148a8d7

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Refactor and add some unit tests

ede82f2

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

Write a unit tests with mocks

ffe545c

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

Adding terraform for queue and event grid

22d3c14

Merge branch 'main' into SFTP-queue-setup

77b9d3f

Fixing refactor issue

25c03d1

use the name when it should be a name

5002afe

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

reference system topic name

da2511e

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

point to storage account ID

8658115

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

apparently we send dead letter messages to a container?

61d257e

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

dashes not underscores

2219879

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

identity and permissions for event

103c2fa

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

another attempt at eventgrid roles

94f2914

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

maybe this?

5aeeabe

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

add tests for message handling and deletion

54c1aa4

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

try a dependency?

b3c5a78

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

???

5baa3eb

change role scope

6cbf7e5

Co-Authored-By: halprin <halprin@users.noreply.github.com>

change principal

9cf1eed

Co-Authored-By: halprin <halprin@users.noreply.github.com>

enable platform logs

2ff0feb

Co-Authored-By: halprin <halprin@users.noreply.github.com>

move TODOs to cards, don't throw if error dequeueing

3204dd7

Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com>

somesylvie temporarily deployed to pr June 10, 2024 22:26 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 10, 2024 22:28 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 10, 2024 22:29 — with GitHub Actions Inactive

Updates for SonarCloud

52a07dd

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

somesylvie temporarily deployed to pr June 11, 2024 15:28 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 15:31 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 15:32 — with GitHub Actions Inactive

Merge branch 'main' into SFTP-queue-setup

035caae

somesylvie temporarily deployed to pr June 11, 2024 17:43 — with GitHub Actions Inactive

somesylvie had a problem deploying to pr June 11, 2024 17:46 — with GitHub Actions Failure

somesylvie temporarily deployed to pr June 11, 2024 18:12 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 18:14 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 18:16 — with GitHub Actions Inactive

somesylvie and others added 2 commits June 11, 2024 13:43

Add ADR for queues

004d504

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

Add code structure/organization ADR

6cab99c

Co-Authored-By: pluckyswan <96704946+pluckyswan@users.noreply.github.com> Co-Authored-By: jcrichlake <145698165+jcrichlake@users.noreply.github.com> Co-Authored-By: halprin <halprin@users.noreply.github.com>

somesylvie temporarily deployed to pr June 11, 2024 18:56 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 18:59 — with GitHub Actions Inactive

somesylvie temporarily deployed to pr June 11, 2024 19:01 — with GitHub Actions Inactive

jcrichlake reviewed Jun 12, 2024

View reviewed changes

src/senders/json_responses.go Outdated Show resolved Hide resolved

src/senders/json_responses.go Outdated Show resolved Hide resolved

somesylvie and others added 2 commits June 12, 2024 14:26

Update src/senders/json_responses.go

703ec5b

Co-authored-by: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

Update src/senders/json_responses.go

d5f4ada

Co-authored-by: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

jcrichlake approved these changes Jun 12, 2024

View reviewed changes

Merge branch 'main' into SFTP-queue-setup

8ea66dc

jcrichlake temporarily deployed to pr June 12, 2024 19:30 — with GitHub Actions Inactive

jcrichlake temporarily deployed to pr June 12, 2024 19:32 — with GitHub Actions Inactive

jcrichlake deployed to pr June 12, 2024 19:34 — with GitHub Actions View deployment

somesylvie merged commit 61a8640 into main Jun 12, 2024
13 checks passed

somesylvie deleted the SFTP-queue-setup branch June 12, 2024 19:52

somesylvie temporarily deployed to pr June 12, 2024 19:52 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SFTP queue setup #50

SFTP queue setup #50

somesylvie commented Jun 10, 2024 •

edited

Loading

sonarcloud bot commented Jun 12, 2024

SFTP queue setup #50

SFTP queue setup #50

Conversation

somesylvie commented Jun 10, 2024 • edited Loading

Queue and trigger setup for SFTP ingestion

Issue

Checklist

sonarcloud bot commented Jun 12, 2024

Quality Gate passed

somesylvie commented Jun 10, 2024 •

edited

Loading