Event Stream Service (#1444)

* Event Stream Service Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * event stream service api for ux (in progress) Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * UX code to support Event Stream Service configuration Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * Add appStore to all tests that use formStore (there is now a dependency to check for features). Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * lint fix. Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * add rate limits to routes Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * sigh - cut/paste error... fix Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * tweak ux when config is new Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * helm charts for event stream service updates to connect via websockets Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * merge fix Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * add event stream config notes, add to deployment Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * string instead of bool Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * naming consistency Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * add cicd param files for dev/test/prod Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * add WEBSOCKETS flag, allow to connect to either NATS protocol or WS Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * update values-dev.yaml for deployment to dev namespace with more resources Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * updating packages so i can merge Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * rename migration file to move past others added to main Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * FORMS-1431 - feature flags tests Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * Event Service Config unit tests, remove unused code Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * import computed after merge.. Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * FORMS-1429: tests for encryption keys some file clean up and merge fix. Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * add event stream service tests Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * FORMS-1428: event stream service tests Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * rename migration file. Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * FORMS-1427: add tests to form service for event stream messages Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * Frontend tests Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * FORMS-1561: add form metadata to event stream message Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * Forms-1548: update open api spec Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * changes to event stream pullConsumer example Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * review changes -paths * fix for adding eventstreamconfig to existing forms. * reset the probeId interval to 10s * remove feature flags allow event stream service to run without connection add flags to enable form level event stream add flags to send event stream meta over webhook * event stream supporting webhook Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * remove rate limiter Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * Update app/src/docs/v1.api-spec.yaml Co-authored-by: Walter Moar <Walter.Moar@gov.bc.ca> * remove api access from encryptionkey routes Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * remove features from config map Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> * put reduced limits on chefs stream * changes from review * Add stream configurations to deployment config map Add specific user for consuming messages Enforce a size on messages. * update params files for openshift. * openshift can't automatically marshall int to string? * logging/info on event stream configuration * remove defaults use param files * need param file included in PR oc command * updated example, updated chart values * ESS helm charts - preserve secret CHEFS - update stream configuration only if needed * Update event-stream-service/charts/event-stream-service/values-test.yaml Co-authored-by: Walter Moar <Walter.Moar@gov.bc.ca> * Update event-stream-service/charts/event-stream-service/values.yaml Co-authored-by: Walter Moar <Walter.Moar@gov.bc.ca> * ess - add axios update secrets * update nginx config, add hpa update resource allocation for prod * connection changes for publishing from CHEFS chart changes to remove NGINX --------- Signed-off-by: Jason Sherman <tools@usingtechnolo.gy> Co-authored-by: Walter Moar <Walter.Moar@gov.bc.ca>
bcgov · Nov 27, 2024 · 721022a · 721022a
1 parent 39252c5
commit 721022a
Show file tree

Hide file tree

Showing 119 changed files with 6,689 additions and 1,200 deletions.
diff --git a/.devcontainer/chefs_local/local.json.sample b/.devcontainer/chefs_local/local.json.sample
@@ -55,13 +55,28 @@
         "public": {
           "limitApiKey": "120",
           "limitFrontend": "500",
-          "windowMs": "60000",
+          "windowMs": "60000"
         }
       },
       "encryption": {
         "proxy": "352f7c24819086bf3df5a38c1a40586045f73e0007440c9d27d59ee8560e3fe7"
       }
     },
+    "eventStreamService": {
+      "servers": "localhost:4222,localhost:4223,localhost:4224",
+      "websockets": "false",
+      "streamName": "CHEFS",
+      "source": "chefs-local",
+      "domain": "forms",
+      "username": "chefs",
+      "password": "password",
+      "maxAge": "300000",
+      "maxBytes": "1048576",
+      "maxMsgs": "10",
+      "maxMsgSize": "51200",
+      "duplicateWindow": "60000",
+      "numReplicas": "3"
+    },
     "serviceClient": {
       "commonServices": {
         "ches": {

diff --git a/.devcontainer/chefs_local/test.json b/.devcontainer/chefs_local/test.json
@@ -61,6 +61,20 @@
       "proxy": "5fb2054478353fd8d514056d1745b3a9eef066deadda4b90967af7ca65ce6505"
     }
   },
+  "eventStreamService": {
+    "servers": "localhost:4222,localhost:4223,localhost:4224",
+    "streamName": "CHEFS",
+    "source": "chefs",
+    "domain": "forms",
+    "username": "chefs",
+    "password": "password",
+    "maxAge": "900000",
+    "maxBytes": "26214400",
+    "maxMsgs": "500",
+    "maxMsgSize": "50000",
+    "duplicateWindow": "60000",
+    "numReplicas": "3"
+  },
   "serviceClient": {
     "commonServices": {
       "ches": {

diff --git a/.github/actions/deploy-to-environment/action.yaml b/.github/actions/deploy-to-environment/action.yaml
@@ -94,6 +94,15 @@ runs:
       run: |
         oc process --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f openshift/app.cm.yaml -p NAMESPACE=${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -p APP_NAME=${{ inputs.acronym }} -p JOB_NAME=${{ inputs.job_name }} -p SERVER_HOST=${{ inputs.server_host }} -o yaml | oc apply --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f -
 
+    - name: Deploy event stream ConfigMaps
+      shell: bash
+      run: |
+        if [[ "${{ inputs.job_name }}" == pr-* ]]; then
+          oc process --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f openshift/ess.cm.yaml -p APP_NAME=${{ inputs.acronym }} -p JOB_NAME=${{ inputs.job_name }} --param-file=openshift/ess.dev.param -p SOURCE=${{ inputs.job_name }} -o yaml | oc apply --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f -
+        else
+          oc process --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f openshift/ess.cm.yaml -p APP_NAME=${{ inputs.acronym }} -p JOB_NAME=${{ inputs.job_name }} --param-file=openshift/ess.${{ inputs.namespace_environment }}.param -o yaml | oc apply --namespace ${{ inputs.namespace_prefix }}-${{ inputs.namespace_environment }} -f -
+        fi
+
     - name: Deploy App
       shell: bash
       run: |

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -48,7 +48,7 @@
       "type": "node",
       "request": "launch",
       "name": "Jest: current file",
-      //"env": { "NODE_ENV": "test" },
+      "env": { "NODE_ENV": "test" },
       "program": "${workspaceFolder}/app/node_modules/.bin/jest",
       "args": [
         "${file}",
@@ -77,6 +77,12 @@
       ],
       "smartStep": true,
       "console": "integratedTerminal"
+    },
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Node.js - Debug Current File",
+      "program": "${file}"
     }
   ],
   "version": "0.2.0"

diff --git a/app/app.js b/app/app.js
@@ -13,14 +13,18 @@ const v1Router = require('./src/routes/v1');
 
 const DataConnection = require('./src/db/dataConnection');
 const dataConnection = new DataConnection();
+const { eventStreamService } = require('./src/components/eventStreamService');
+
 const apiRouter = express.Router();
 const state = {
   connections: {
     data: false,
+    eventStreamService: false,
   },
   ready: false,
   shutdown: false,
 };
+
 let probeId;
 const app = express();
 app.use(compression());
@@ -62,6 +66,14 @@ apiRouter.use('/config', (_req, res, next) => {
     // we will need to pass
     const uploads = config.get('files.uploads');
     const feConfig = { ...frontend, uploads: uploads };
+    let ess = config.util.cloneDeep(config.get('eventStreamService'));
+    if (ess) {
+      delete ess['username'];
+      delete ess['password'];
+      feConfig['eventStreamService'] = {
+        ...ess,
+      };
+    }
     res.status(200).json(feConfig);
   } catch (err) {
     next(err);
@@ -160,6 +172,7 @@ function cleanup() {
   log.info('Cleaning up...', { function: 'cleanup' });
   clearInterval(probeId);
 
+  eventStreamService.closeConnection();
   dataConnection.close(() => process.exit());
 
   // Wait 10 seconds max before hard exiting
@@ -173,7 +186,7 @@ function cleanup() {
  */
 function initializeConnections() {
   // Initialize connections and exit if unsuccessful
-  const tasks = [dataConnection.checkAll()];
+  const tasks = [dataConnection.checkAll(), eventStreamService.checkConnection()];
 
   Promise.all(tasks)
     .then((results) => {
@@ -183,9 +196,16 @@ function initializeConnections() {
         log.info('DataConnection Reachable', {
           function: 'initializeConnections',
         });
+
+      state.connections.eventStreamService = results[1];
+      const reachable = state.connections.eventStreamService ? 'Reachable' : 'Unreachable';
+      log.info(`EventStreamService ${reachable}`, {
+        function: 'initializeConnections',
+      });
     })
     .catch((error) => {
       log.error(`Initialization failed: Database OK = ${state.connections.data}`, { function: 'initializeConnections' });
+      log.error(`Initialization failed: EventStreamService OK = ${state.connections.eventStreamService}`, { function: 'initializeConnections' });
       log.error('Connection initialization failure', error.message, {
         function: 'initializeConnections',
       });
@@ -195,7 +215,7 @@ function initializeConnections() {
       }
     })
     .finally(() => {
-      state.ready = Object.values(state.connections).every((x) => x);
+      state.ready = state.connections.data; // only need db running
       if (state.ready) {
         log.info('Service ready to accept traffic', {
           function: 'initializeConnections',
@@ -214,17 +234,21 @@ function initializeConnections() {
 function checkConnections() {
   const wasReady = state.ready;
   if (!state.shutdown) {
-    const tasks = [dataConnection.checkConnection()];
+    const tasks = [dataConnection.checkConnection(), eventStreamService.checkConnection()];
 
     Promise.all(tasks).then((results) => {
       state.connections.data = results[0];
-      state.ready = Object.values(state.connections).every((x) => x);
+      state.connections.eventStreamService = results[1];
+
+      state.ready = state.connections.data; // only want no db to halt application
       if (!wasReady && state.ready)
         log.info('Service ready to accept traffic', {
           function: 'checkConnections',
         });
       log.verbose(state);
       if (!state.ready) {
+        log.error(`Database connected = ${state.connections.data}`, { function: 'checkConnections' });
+        log.error(`EventStreamService connected = ${state.connections.eventStreamService}`, { function: 'checkConnections' });
         process.exitCode = 1;
         shutdown();
       }

diff --git a/app/config/custom-environment-variables.json b/app/config/custom-environment-variables.json
@@ -58,6 +58,21 @@
       "proxy": "SERVER_ENCRYPTION_PROXY"
     }
   },
+  "eventStreamService": {
+    "servers": "EVENTSTREAMSERVICE_SERVERS",
+    "websockets": "EVENTSTREAMSERVICE_WEBSOCKETS",
+    "streamName": "EVENTSTREAMSERVICE_STREAMNAME",
+    "source": "EVENTSTREAMSERVICE_SOURCE",
+    "domain": "EVENTSTREAMSERVICE_DOMAIN",
+    "username": "EVENTSTREAMSERVICE_USERNAME",
+    "password": "EVENTSTREAMSERVICE_PASSWORD",
+    "maxAge": "EVENTSTREAMSERVICE_MAXAGE",
+    "maxBytes": "EVENTSTREAMSERVICE_MAXBYTES",
+    "maxMsgs": "EVENTSTREAMSERVICE_MAXMSGS",
+    "maxMsgSize": "EVENTSTREAMSERVICE_MAXMSGSIZE",
+    "duplicateWindow": "EVENTSTREAMSERVICE_DUPLICATEWINDOW",
+    "numReplicas": "EVENTSTREAMSERVICE_NUMREPLICAS"
+  },
   "serviceClient": {
     "commonServices": {
       "ches": {

diff --git a/app/config/default.json b/app/config/default.json
@@ -62,6 +62,21 @@
       "proxy": "352f7c24819086bf3df5a38c1a40586045f73e0007440c9d27d59ee8560e3fe7"
     }
   },
+  "eventStreamService": {
+    "servers": "localhost:4222,localhost:4223,localhost:4224",
+    "websockets": "false",
+    "streamName": "CHEFS",
+    "source": "chefs-local",
+    "domain": "forms",
+    "username": "chefs",
+    "password": "password",
+    "maxAge": "900000",
+    "maxBytes": "966367641",
+    "maxMsgs": "1000",
+    "maxMsgSize": "966367",
+    "duplicateWindow": "60000",
+    "numReplicas": "3"
+  },
   "serviceClient": {
     "commonServices": {
       "ches": {

diff --git a/app/frontend/src/components/designer/FormDesigner.vue b/app/frontend/src/components/designer/FormDesigner.vue
@@ -406,6 +406,7 @@ async function schemaCreateNew() {
     useCase: form.value.useCase,
     labels: form.value.labels,
     formMetadata: form.value.formMetadata,
+    eventStreamConfig: form.value.eventStreamConfig,
   });
   // update user labels with any new added labels
   if (

diff --git a/app/frontend/src/components/designer/FormSettings.vue b/app/frontend/src/components/designer/FormSettings.vue
@@ -1,12 +1,13 @@
 <script setup>
 import { storeToRefs } from 'pinia';
-
 import FormGeneralSettings from '~/components/designer/settings/FormGeneralSettings.vue';
 import FormAccessSettings from '~/components/designer/settings/FormAccessSettings.vue';
 import FormFunctionalitySettings from '~/components/designer/settings/FormFunctionalitySettings.vue';
 import FormSubmissionSettings from '~/components/designer/settings/FormSubmissionSettings.vue';
 import FormScheduleSettings from '~/components/designer/settings/FormScheduleSettings.vue';
 import FormMetadataSettings from '~/components/designer/settings/FormMetadataSettings.vue';
+import FormEventStreamSettings from '~/components/designer/settings/FormEventStreamSettings.vue';
+
 import { useFormStore } from '~/store/form';
 
 defineProps({
@@ -40,6 +41,9 @@ const { form, isFormPublished, isRTL } = storeToRefs(useFormStore());
       <v-col cols="12" md="6">
         <FormMetadataSettings :disabled="disabled" />
       </v-col>
+      <v-col v-if="form.eventStreamConfig.enabled">
+        <FormEventStreamSettings :disabled="disabled" />
+      </v-col>
     </v-row>
   </v-container>
 </template>