On-demand generation of current full dataset for download

[cbaksik]

We would like to be able to download a full dump of data that represents the current data set in POD for a given stream. In other words, a download that contains all of our unique records in their latest state, excluding any records deleted since the stream was created.

Currently, to get this data, we would have to download the last full that we sent to POD, as well as any incremental and delete files since then, and integrate them all together locally. We would like to avoid this, and instead have an option in the POD UI (or via API) to do this for us on demand, so that we can fetch a single full file of our current data set.

We would not expect to use this feature frequently, so there is no expectation that the file needs to be generated quickly (e.g. under 24 hours). If the generation takes more than 24 hours, that is fine. Ideally, the system would send us a notification when the file was available, but that's not part of the requirement. We can just check periodically to see if it's done.

We would like the feature to be available to any of our users. It doesn't have to be restricted to Admins. It should be clear in the UI when the last full was generated, and/or whether one is currently being generated, so that a user doesn't initiate the process if it's been done recently, or is currently in progress. Alternatively, the UI piece could be a later enhancement, in which case the option to generate a full file should be limited to Admins, or be limited to a request via API (admin or any user) to make it less likely that someone would do it by accident.

We're interested in using this feature for our own data. If the feature would allow other partners to generate a new full set of our data, this is fine too.

USE CASE:

We have a project partner (unrelated to POD) and would like to supply them with a recent full set of our data. The last full dump we sent to POD as a new stream was in June 2022. We send daily incremental and deletes to POD, so we have sent hundreds of files since mid-2022. We would like to use POD to generate a full set of current data, without having to go through the process of incorporating those hundreds of incremental and deletes into the full file from mid-2022.