[Feature] Export Data Based on Backup S3 Bucket

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Description

This feature relies on the Backup Agent functionality outlined in #12876. The backup agent transfers the data to a S3 bucket, the Export Command LIne will enable exporting data based on this bucket, allowing users to extract and utilize specific sets of data efficiently.

Requirements

1. Integration with Backup Snapshots:

   • Leverage the S3 bucket created by the Backup Agent feature as the data source for export.
   • Ensure the export process clean up the snapshot is created for the exportation.

2. Export Configuration:

   • Time Range Export: Allow users to define a specific time range from the snapshot for export.
   • Query-Based Export: Allow users to filter and export data using specific query criteria (e.g., tags, metrics, identifiers).

3. Export Formats:

   • Support commonly used data formats like JSON and CSV.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Is Export Manager going to be a separate role node?

[hanahmily]

Is Export Manager going to be a separate role node?

It is a command-line tool provided by bydbctl, which can be scheduled to run automatically using cron on a daily basis or at specified intervals.

[wu-sheng]

So, the tool is going to grap snapshots from mutiple data nodes(through liaison)?
Where the filter happens? Are they processed on data nodes?

[wu-sheng]

About the feature, the exportation should support tags/fields selection if possible. This could reduce the volume costs of the files.

[hanahmily]

I want to create a design to explain the details. Perhaps we can set up exporting to the backup process.