Ingestion partition change #683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…of using server time
… of using server time
…i -- to allow ingestion/query using timestamp column from the log data instead of server time / p_timestamp store the time_partition field name in stream.json and in memory STREAM_INFO in ingest api - check if timestamp column name exists in the log data, if no, throw exception also, check if timestamp value can be parsed into datetime, if no, throw exception arrow file name gets the date, hr, mm from the timestamp field (if defined in stream) else file name gets the date, hr, mm from the server time parquet file name gets a random number attached to it -- this is because a lot of log data can have same date, hr, mm value of the timestamp field and with this random number, parquet will not get overwritten in the console, query from and to date will be matched against the value of the timestamp column of the log data (if defined in the stream), else from and to date will be matched against the p_timestamp column
|
Fixes #685 |
Eshanatnight
suggested changes
Mar 5, 2024
server/src/handlers/http/ingest.rs
Outdated
Comment on lines
138
to
171
| let mut ingestion_prefix_timestamp = Utc::now().naive_utc(); | ||
|
|
||
| if time_partition.is_some() { | ||
| let body_timestamp = body.get(&time_partition.clone().unwrap().to_string()); | ||
| if body_timestamp.is_some() { | ||
| if body_timestamp | ||
| .unwrap() | ||
| .to_owned() | ||
| .as_str() | ||
| .unwrap() | ||
| .parse::<DateTime<Utc>>() | ||
| .is_ok() | ||
| { | ||
| ingestion_prefix_timestamp = body_timestamp | ||
| .unwrap() | ||
| .to_owned() | ||
| .as_str() | ||
| .unwrap() | ||
| .parse::<DateTime<Utc>>() | ||
| .unwrap() | ||
| .naive_utc(); | ||
| } else { | ||
| return Err(PostError::Invalid(anyhow::Error::msg(format!( | ||
| "field {} is not in the correct datetime format", | ||
| body_timestamp.unwrap().to_owned().as_str().unwrap() | ||
| )))); | ||
| } | ||
| } else { | ||
| return Err(PostError::Invalid(anyhow::Error::msg(format!( | ||
| "ingestion failed as field {} is not part of the log", | ||
| time_partition.unwrap() | ||
| )))); | ||
| } | ||
| } |
There was a problem hiding this comment.
This could be simplified,
.
.
.
let ingestion_prefix_timestamp = if let Some(time_partition) = time_partition {
let body_timestamp = body.get(&time_partition).ok_or(PostError::Invalid(anyhow::Error::msg(format!(
"ingestion failed as field {} is not part of the log",
time_partition
))))?;
body_timestamp.as_str().unwrap().parse::<DateTime<Utc>>().map_err(|_| PostError::Invalid(anyhow::Error::msg(format!(
"field {} is not in the correct datetime format",
body_timestamp.to_owned().as_str().unwrap()
))))?.naive_utc()
} else {
Utc::now().naive_utc()
};
.
.
.
.There was a problem hiding this comment.
the logic has changed now, please review again
review comments incorporated
Eshanatnight
suggested changes
Mar 7, 2024
nikhilsinhaparseable
added a commit
to nikhilsinhaparseable/parseable
that referenced
this pull request
Apr 20, 2024
…seablehq#683) This PR adds enhancement to use a user provided timestamp for partition in ingesting logs instead of using server time. User needs to add custom header X-P-Time-Partition (optional) at stream creation api to allow ingestion/query using timestamp column from the log data instead of server time p_timestamp This is time_partition field name is stored in stream.json and in memory STREAM_INFO in ingest api. Server checks if timestamp column name exists in the log event, if not, throw exception. Also, checks if timestamp value can be parsed into datetime, if not, throw exception arrow file name gets the date, hr, mm from the timestamp field (if defined in stream) else file name gets the date, hr, mm from the server time parquet file name gets a random number attached to it. This is because a lot of log data can have same date, hr, mm value of the timestamp field and with this random number, parquet will not get overwritten in the console, query from and to date will be matched against the value of the timestamp column of the log data (if defined in the stream), else from and to date will be matched against the p_timestamp column. Fixes parseablehq#671 Fixes parseablehq#685
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add custom header X-P-Time-Partition (optional) at stream creation api -- to allow ingestion/query using timestamp column from the log data instead of server time / p_timestamp
store the time_partition field name in stream.json and in memory STREAM_INFO
in ingest api - check if timestamp column name exists in the log data, if no, throw exception
also, check if timestamp value can be parsed into datetime, if no, throw exception
arrow file name gets the date, hr, mm from the timestamp field (if defined in stream) else file name gets the date, hr, mm from the server time
parquet file name gets a random number attached to it -- this is because a lot of log data can have same date, hr, mm value of the timestamp field and with this random number, parquet will not get overwritten
in the console, query from and to date will be matched against the value of the timestamp column of the log data (if defined in the stream), else from and to date will be matched against the p_timestamp column