Historical Channel

Goals

• Reuse the Hub's temporal APIs for data created in the past.
• Preserve the immutability and ordering contracts for stable data.
• Allow multithreaded writes of historical data.
• Provide a way to unwind changes or delete historical data that isn't stable yet.
• Provide a mechanism to verify historical data before it is made stable.

Proposed Change

• A channel can now have a mutableTime attribute, which is a valid time.
• Once mutableTime is set, items can be inserted into the channel before or equal to the mutableTime.
• All items after the mutableTime are considered "stable".
• Normal Hub channel rules and expectations apply to the stable portion of the data (e.g. normal inserts, webhooks, replication, etc.)
• Items can be inserted or deleted in any order, allowing for multithreading, before the mutableTime.
• Once the changes are final you can move the mutableTime earlier in time, making all items after the new mutableTime "stable".
• mutableTime channels can not have ttlDays and maxItems, as they are intended for long term storage.
• Existing channels can be converted to this type by someone with admin access.
• This would replace the current notion of "Historical Channels"

How this all works

A new channel is created with the mutability time set to "now". This is effectively the same as a normal channel creation. New data can be inserted linearly moving forward in time.

time -------------------|>
                       now

While the real-time inserts are happening a multithreaded writer is inserting and deleting items at various points in time prior to the mutability time.

               unstable |  stable  |
time -------------------|---------->
                                  now

Once you're happy with the historical data, the mutability time can be moved backwards, making the previously unstable data "stable".

             |       stable        |
time --------|--------------------->
                                  now