MNTOR-3776: cronjob to pull in the data brokers and refresh (part 2)

[mansaj]

References:

Jira: MNTOR-3776

Description

Part 2 of the ticket: cronjob to pull in the data brokers and refresh

How to test

npm run dev:cron:db-pull-data-brokers

Checklist (Definition of Done)

• Commits in this PR are minimal and have descriptive commit messages.
• I've added or updated the relevant sections in readme and/or code comments
• If this PR implements a feature flag or experimentation, the Ship Behind Feature Flag status in Jira has been set
• All acceptance criteria are met.

[github-actions]

Preview URL 🚀 : https://blurts-server-pr-5291-mgjlpikfea-uk.a.run.app

[github-actions]

Preview URL 🚀 : https://blurts-server-pr-5291-mgjlpikfea-uk.a.run.app

[rhelmer]

It's not harmful but we no longer have this service enabled in Sentry FYI

[github-actions]

Preview URL 🚀 : https://blurts-server-pr-5291-mgjlpikfea-uk.a.run.app

[rhelmer]

I found an issue while testing this, in

blurts-server/src/app/functions/server/onerep.ts

Line 427 in ba9c0e4

"/data-brokers?per_page=100&page=" + page.toString(),

where per_page is hardcoded.

getAllDataBrokers function is already using fetchAllPages so I think we just need pass the current/total pages correctly. I'll add that to this PR.

[rhelmer]

I found an issue while testing this, in

blurts-server/src/app/functions/server/onerep.ts

Line 427 in ba9c0e4

"/data-brokers?per_page=100&page=" + page.toString(),

where per_page is hardcoded.

getAllDataBrokers function is already using fetchAllPages so I think we just need pass the current/total pages correctly. I'll add that to this PR.

Oh actually I take that back, it's just a bit confusing to read here but per_page is 100 and page is passed correctly, so this should be fine.

[github-actions]

Preview URL 🚀 : https://blurts-server-pr-5291-mgjlpikfea-uk.a.run.app

[github-actions]

Preview URL 🚀 : https://blurts-server-pr-5291-mgjlpikfea-uk.a.run.app

[github-actions]

Cleanup completed - database 'blurts-server-pr-5291' destroyed, cloud run service 'blurts-server-pr-5291' destroyed