Scheduled polling v1 (#78)

- Added terraform for an Azure Function to use for polling
- Added terraform for polling queue and DLQ
- Added function deploy github action
- Added function code to, on a CRON schedule, enqueue a message
- Updated README


Co-authored-by: Samuel Aquino <saquino@flexion.us>
Co-authored-by: pluckyswan <96704946+pluckyswan@users.noreply.github.com>
Co-authored-by: James Herr <jherr@flexion.us>
Co-authored-by: Bella Luz Quintero <bquintero@flexion.us>
Co-authored-by: jcrichlake <145698165+jcrichlake@users.noreply.github.com>

.github/workflows/functions-deploy.yml

@@ -0,0 +1,90 @@
+# TODO - currently this is only called from the `internal` deploy job. It should be part of all envs
+name: Deploy Node.js project to Azure Function App
+
+on:
+  workflow_call:
+    inputs:
+      ENVIRONMENT:
+        required: true
+        type: string
+      REGISTRY:
+        required: true
+        type: string
+      REPO:
+        required: true
+        type: string
+      APP:
+        required: true
+        type: string
+    secrets:
+      AZURE_CLIENT_ID:
+        required: true
+      AZURE_TENANT_ID:
+        required: true
+      AZURE_SUBSCRIPTION_ID:
+        required: true
+      AZURE_FUNCTIONAPP_PUBLISH_PROFILE:
+        required: true
+
+permissions:
+  contents: read
+
+env:
+  # AZURE_FUNCTIONAPP_NAME has to match the azurerm_linux_function_app's name in functions.tf
+  AZURE_FUNCTIONAPP_NAME: 'polling-function-${{ inputs.ENVIRONMENT }}'
+  AZURE_FUNCTIONAPP_PACKAGE_PATH: 'azure_functions'
+  NODE_VERSION: '20.x'
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    environment:
+      ${{ inputs.ENVIRONMENT }}
+    permissions:
+      id-token: write
+      contents: read
+
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@v4
+
+      - name: 'Setup Node ${{ env.NODE_VERSION }} Environment'
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: 'Resolve Project Dependencies Using Npm'
+        shell: bash
+        run: |
+          pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
+          npm install
+          npm run build --if-present
+          npm run test --if-present
+          popd
+
+      - name: 'Login via Azure CLI'
+        uses: azure/login@v2
+        with:
+          client-id: ${{ secrets.AZURE_CLIENT_ID }}
+          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
+          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
+
+      - name: 'Get Publish Profile'
+        run: |
+          PUBLISH_PROFILE=$(az webapp deployment list-publishing-profiles -g "csels-rsti-${{ inputs.ENVIRONMENT }}-moderate-rg" -n "polling-function-${{ inputs.ENVIRONMENT }}" --xml)
+          echo "::add-mask::$PUBLISH_PROFILE"
+          echo "PUBLISH_PROFILE=$PUBLISH_PROFILE" >> $GITHUB_OUTPUT
+        id: getPublishProfile
+
+      - name: 'Run Azure Functions Action'
+        uses: Azure/functions-action@v1
+        id: azure-function-deploy
+        with:
+          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
+          package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
+          publish-profile: ${{ steps.getPublishProfile.outputs.PUBLISH_PROFILE }}
+
+# TODO: Add a step to report failure on Azure Function App Func creation
+# Currently, if there's a bug in the function code or it tries to use an environment variable that doesn't exist
+# or has an invalid value, the Github Action Step will complete successfully but the function code will NOT be
+# deployed, and any previous version of the function will be removed

.github/workflows/internal-deploy.yml

@@ -1,6 +1,10 @@
 name: Deploy to Internal Environment
 
 on:
+  workflow_call:
+    secrets:
+      AZURE_FUNCTIONAPP_PUBLISH_PROFILE_INTERNAL:
+        required: true
   push:
     branches:
       - internal
@@ -32,3 +36,18 @@ jobs:
       AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
       AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
       AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
+
+  function-deploy:
+    name: Function Deploy
+    needs: terraform-deploy
+    uses: ./.github/workflows/functions-deploy.yml
+    with:
+      ENVIRONMENT: internal
+      REPO: report-stream-sftp-ingest
+      APP: ${{ needs.terraform-deploy.outputs.APP }}
+      REGISTRY: ${{ needs.terraform-deploy.outputs.REGISTRY }}
+    secrets:
+      AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
+      AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
+      AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
+      AZURE_FUNCTIONAPP_PUBLISH_PROFILE: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE_INTERNAL }}

.gitignore

@@ -26,10 +26,113 @@ reportstream-sftp-ingestion
 
 # IDE files
 .idea/
+.vscode/
 
 # Ignore terraform state (as it is persisted via Azure Storage)
 terraform.tfstate*
 .terraform*
 
 # Local blob storage data
 /localdata
+
+
+# Items added by creating the Azure function:
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# TypeScript v1 declaration files
+typings/
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+.env.test
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+
+# next.js build output
+.next
+
+# nuxt.js build output
+.nuxt
+
+# vuepress build output
+.vuepress/dist
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TypeScript output
+dist
+out
+
+# Azure Functions artifacts
+bin
+obj
+appsettings.json
+local.settings.json
+
+# Azurite artifacts
+__blobstorage__
+__queuestorage__
+__azurite_db*__.json

README.md

@@ -74,7 +74,7 @@ In the cloud, EventGrid monitors the blob storage container and sends file creat
 In the local Azurite tool, there are no events to connect the blob storage container to the queue.
 To mimic the deployed behavior so our app can read queue messages and access the file specified in the message:
 1. Upload a file to your local Azurite sftp container
-2. In Azure Storage Explorer, find the `blob-message-queue` that the service currently reads from
+2. In Azure Storage Explorer, find the `message-import-queue` that the service currently reads from
 3. Add a file create event message to that queue. You can start with this base message and edit the `subject` to
 match your newly-created file
    ```json
@@ -174,6 +174,7 @@ occur when a release is published.
 
 ## Related documents
 
+* [Azure Functions and Typescript](/azure_functions/src/README.md)
 * [Open Practices](/docs/open_practices.md)
 * [Rules of Behavior](/docs/rules_of_behavior.md)
 * [Thanks and Acknowledgements](/docs/thanks.md)

adr/009-typescript-and-azure-functions.md

@@ -0,0 +1,18 @@
+# 9. Typescript and Azure Functions
+
+Date: 2024-07-15
+
+## Decision
+
+We're using Azure Functions for scheduled tasks. Azure Functions support a CRON
+expression trigger, which most other Azure resources do not. Azure Webjobs are
+supposed to work with CRON triggers, but don't seem to work well with Linux servers since they are in preview.
+
+We're using Typescript for Azure Functions.
+Go is not well-supported (it requires special handlers). Of the well-supported
+language options, Typescript is lighter-weight than Java or C# and more
+familiar to the team than Python.
+
+## Status
+
+Accepted.

azure_functions/.funcignore

@@ -0,0 +1,11 @@
+*.js.map
+*.ts
+.git*
+.vscode
+__azurite_db*__.json
+__blobstorage__
+__queuestorage__
+local.settings.json
+test
+tsconfig.json
+node_modules/

azure_functions/host.json

@@ -0,0 +1,15 @@
+{
+  "version": "2.0",
+  "logging": {
+    "applicationInsights": {
+      "samplingSettings": {
+        "isEnabled": true,
+        "excludedTypes": "Request"
+      }
+    }
+  },
+  "extensionBundle": {
+    "id": "Microsoft.Azure.Functions.ExtensionBundle",
+    "version": "[4.*, 5.0.0)"
+  }
+}