feat(batch): support system column _rw_timestamp for tables #19232
Conversation
| // `get_row` doesn't support select `_rw_timestamp` yet. | ||
| assert!(self.epoch_idx.is_none()); |
There was a problem hiding this comment.
Can we extend the get interface to also return the user key (thus epoch)?
There was a problem hiding this comment.
Yes. We do have a plan to add an extended interface (get_with_epoch) to return the epoch from the storage. Let's leave it for a later PR.
| .batch_chunk_iter_with_pk_bounds( | ||
| epoch.into(), | ||
| &pk_prefix, | ||
| range_bounds, | ||
| false, | ||
| 1, | ||
| PrefetchOptions::new(false, false), | ||
| ) | ||
| .await?; | ||
| pin_mut!(iter); | ||
| let chunk = iter.next().await.transpose().map_err(BatchError::from)?; | ||
| if let Some(chunk) = chunk { | ||
| let row = chunk.row_at(0).0.to_owned_row(); | ||
| Ok(Some(row)) | ||
| } else { | ||
| Ok(None) | ||
| } |
There was a problem hiding this comment.
Refactoring this into a separate method like get_row_with_rw_timestamp, and calling it here seems more readable.
After column pruning, this column will be pruned. |
Ohhh SORRY for misreading |
Yes, it works for MVs. |
| "selecting `_rw_timestamp` in a streaming query is not allowed".to_string(), | ||
| "please run the sql in batch mode or remove the column `_rw_timestamp` from the streaming query".to_string(), |
There was a problem hiding this comment.
It's theoretically feasible, right? For example, use the epoch of the persisted entry for historical data, and the current epoch for incremental data.
Do we plan to support it in the future? If so, can we open an issue for it?
There was a problem hiding this comment.
and the current epoch for incremental data.
Yes, it is theoretically possible. We need to fetch the old epoch for delate mesage from the storage as well. Whethr we plan to support it in the future is depended on the requirement. If just for debugging purpose, I think batch query is enough, and it won't be invasive to the streaming proto.
There was a problem hiding this comment.
I think the problem here is the same as when we try to specify the proctime in the streaming query, which generates inconsistent delete messages. We don't know what the previous time value was when processing the delete message
There was a problem hiding this comment.
We don't know what the previous time value was when processing the delete message
Makes sense to me. Some ideas:
- For append-only table / materialized views, there's no confusion and it sounds feasible.
- For table with on-conflict checks, since we already have to lookup the storage for every deletes, we can obtain the deleted timestamp at the same time.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
rw_timestampfor any tables #11629_rw_timestamp(datatype is timestamptz) for tables. We will add a hidden column_rw_timestampto every table catalog when loading it from a proto, but we never persist this column info._rw_timestampin a batch query. Using it in a streaming query will cause an error.epoch_idxto the storage table which indicates the position where we should put the epoch into._rw_timestampis selected.Example:
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.
_rw_timestampsystem column for tables. Users can use a batch query to select this column to check the internal epoch/timestamp of each row which is useful when you want to know when the rows have been updated recently.