Time gaps, chunking, and compression

[del42]

Following are some of the suggestions from the meeting discussion:

Time Gap Filling

1. Data Continuity: Ensures a continuous dataset, crucial for time-series analysis and forecasting models.
2. Improved Analysis: Filling gaps can enable better statistical analysis and machine learning model performance.

Chunking

1. Manageability: Breaks large datasets into smaller, more manageable pieces, easing processing and analysis.
2. Parallel Processing: Enables distributed or parallel processing, enhancing computational efficiency.
3. Loss of Context: Individual chunks might lose the context of the whole dataset, affecting analysis.
4. Computational Cost: Requires compute for dividing and reassembling data, which can introduce overhead.

Compression

1. Reduced Storage Space: Compresses data to occupy less storage, beneficial for large datasets.
2. Improved Transmission Speed: Compressed data can be transmitted more quickly over networks.
3. Loss of Information: Lossy compression techniques can result in loss of data fidelity.
4. Computational Cost: Compressing and decompressing data requires computational resources, potentially impacting performance.

ℹ️ NOTE: This is not yet final decision.
❗ Important: CHoRUS is purely Cloud environment. Thus the compute, storage, and data redundancy should to be taken serious.