From c48848cd572bd503d5b6ffc720375788512f18fd Mon Sep 17 00:00:00 2001 From: MacRae Linton Date: Wed, 23 Oct 2024 10:00:12 -0700 Subject: [PATCH 1/2] ADR for syncing data to Salesforce --- .../029-push-submissions-into-salesforce.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/architectural-decision-records/029-push-submissions-into-salesforce.md diff --git a/docs/architectural-decision-records/029-push-submissions-into-salesforce.md b/docs/architectural-decision-records/029-push-submissions-into-salesforce.md new file mode 100644 index 0000000000..32af464af0 --- /dev/null +++ b/docs/architectural-decision-records/029-push-submissions-into-salesforce.md @@ -0,0 +1,51 @@ +# Push Submission Data Into Salesforce + +In order for CMS users to be able to review submissions in Salesforce, all submission need to be in the Salesforce database. We will first get data into Salesforce by calling Salesforce APIS from app-api on every submission. Other data sync strategies may follow. + +### Decision Drivers + +- Data needs to get into Salesforce automatically, not by manually copying data over from the web portal. +- It needs to be in the Salesforce review system within an hour or so +- We would like to be able to include a link to the Salesforce record in the email we send on submission +- We can rely on Salesforce's builtin change tracking so we don't need to send over revisions, just per-contract and per-rate data +- Source of truth for all submission data will be the web portal, no edits will be allowed in Salesforce, only review information will be entered there. + + +## Options + +### Chosen: Push data into Salesforce from app-api on submission + +Salesforce has a few different API options for sending data into its database from outside. We can make a REST call from app-api as part of our submission handler. This will send data into Salesforce immediately at submission time, there will be no gap. On resubmission we will send over the new data. + +- `+` Data is sync'd to Salesforce immediately +- `+` Get to write a single API call, no diffing with existing data required +- `+` Get data back immediately, can send links to Salesforce in submission email +- `-` No sync, if any calls fail we need to manually fix the missing data +- `-` No sync, so past submissions will not be copied over +- `-` Have to push data on every sync-action: submit, add question, add response + + +### Use pub/sub to sync data between app-api and Salesforce + +Use GraphQL pub/sub system to sync. Salesforce will subscribe to contracts and rates and all new entries will get sent over. pub/sub will handle retry logic when there are errors. + +- `+` Can be used to sync historical data over +- `+` Pub/sub supplies tricky retry logic, should be durable +- `+` If push errors, we still will get data into the system +- `+` Would call existing app-api API, same way we've asked ARMS to +- `-` Not instantaneous, probably can't send Salesforce link in submission email +- `-` requires writing more code in Salesforce +- `-` GraphQL Pub/Sub is an extension to Apollo, not built in + + +### Use use BigMac (Kafka) to sync data between app-api and Salesforce + +Use the existing MACPRO system BigMac to sync data. app-api -> BigMac -> Salesforce. + +- `+` Can be used to sync historical data over +- `+` Kafka supplies tricky retry logic, should be durable +- `+` If push errors, we still will get data into the system +- `+` MACPRO already has multiple data sources going into BigMac, MC Contract data would be another +- `-` adding another system to the sync process is adding complexity, more pieces to fail +- `-` We haven't had reason to integrate with BigMac yet, attaching two systems to it just to talk to each other seems wasteful. + From ea16ca4a4a0cfca526e4c624655a0514af0b80c2 Mon Sep 17 00:00:00 2001 From: MacRae Linton Date: Mon, 4 Nov 2024 13:45:49 -0800 Subject: [PATCH 2/2] update choice and add some more choices --- .../029-push-submissions-into-salesforce.md | 37 ++++++++++++++++--- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/docs/architectural-decision-records/029-push-submissions-into-salesforce.md b/docs/architectural-decision-records/029-push-submissions-into-salesforce.md index 32af464af0..4598c62094 100644 --- a/docs/architectural-decision-records/029-push-submissions-into-salesforce.md +++ b/docs/architectural-decision-records/029-push-submissions-into-salesforce.md @@ -9,21 +9,33 @@ In order for CMS users to be able to review submissions in Salesforce, all submi - We would like to be able to include a link to the Salesforce record in the email we send on submission - We can rely on Salesforce's builtin change tracking so we don't need to send over revisions, just per-contract and per-rate data - Source of truth for all submission data will be the web portal, no edits will be allowed in Salesforce, only review information will be entered there. +- We will need to import Contracts, Rates, Questions, and Responses. +- Can be implemented by the end of the year, we are on a deadline with this project ## Options -### Chosen: Push data into Salesforce from app-api on submission +### Chosen: Write a script to pull data from app-api into Salesforce -Salesforce has a few different API options for sending data into its database from outside. We can make a REST call from app-api as part of our submission handler. This will send data into Salesforce immediately at submission time, there will be no gap. On resubmission we will send over the new data. +Run a script on a scheduled job that imports data into Salesforce from app-api. Probably add an "updatedAfter" flag to most index APIs to thin responses. We will run this script on the regular with no automated error handling. When we want to get into error handling we can start to investigate pub-sub -- `+` Data is sync'd to Salesforce immediately +- `+` Can be used to sync historical data over +- `+` If it errors, we still will get data into the system on the next run, or after we fix the bug +- `+` Would call existing app-api API, same way we've asked ARMS to +- `-` Not instantaneous, probably can't send Salesforce link in submission email +- `-` Hand-rolling sync code, pub/sub would provide a lot of stuff for free + +### Push data into Salesforce from app-api on submission + +Salesforce has a few different API options for sending data into its database from outside. We can make a REST call from app-api as part of all of our create* handlers. This will send data into Salesforce immediately at submission time, there will be no gap. On resubmission we will send over the new data. -- Historical data can be loaded one-time. + +- `+` Data is sync'd to Salesforce immediately - `+` Get to write a single API call, no diffing with existing data required - `+` Get data back immediately, can send links to Salesforce in submission email - `-` No sync, if any calls fail we need to manually fix the missing data - `-` No sync, so past submissions will not be copied over -- `-` Have to push data on every sync-action: submit, add question, add response - +- `-` Have to push data on every syncable-action: submit, add question, add response, approve +- `-` Testing around the API integration is difficult b/c it requires performing a bunch of actions in MC Review to trigger ### Use pub/sub to sync data between app-api and Salesforce @@ -34,9 +46,21 @@ Use GraphQL pub/sub system to sync. Salesforce will subscribe to contracts and r - `+` If push errors, we still will get data into the system - `+` Would call existing app-api API, same way we've asked ARMS to - `-` Not instantaneous, probably can't send Salesforce link in submission email -- `-` requires writing more code in Salesforce - `-` GraphQL Pub/Sub is an extension to Apollo, not built in +- `-` More work to adopt another technology + +### Use "External Objects" to create a view of the Postgres tables + +Salesforce gets a view of the db tables directly. +- `+` Everything is sync'd automatically +- `+` No custom code is required; all configurations are handled through the platform +- `-` Might be very expensive (5k/table/month?) +- `-` Ties Salesforce to our db model, which is not designed for public consumption whereas the API is +- `-` Requires allowing connections from our database to salesforce, a security risk since right now access to the db is very locked down +- `-` Licenses need to be purchased from day one in the demo org, as trial orgs expire in less than two weeks +- `-` It is necessary to verify that the connectors are FedRAMP certified at CMS. +- `-` Bidirectional synchronization must be carefully planned based on the connectors and the external database ### Use use BigMac (Kafka) to sync data between app-api and Salesforce @@ -48,4 +72,5 @@ Use the existing MACPRO system BigMac to sync data. app-api -> BigMac -> Salesfo - `+` MACPRO already has multiple data sources going into BigMac, MC Contract data would be another - `-` adding another system to the sync process is adding complexity, more pieces to fail - `-` We haven't had reason to integrate with BigMac yet, attaching two systems to it just to talk to each other seems wasteful. +- `-` Requires coordination with another team