Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement]: Abandon the time travel feature #26070

Closed
1 task done
jiaoew1991 opened this issue Aug 1, 2023 · 4 comments
Closed
1 task done

[Enhancement]: Abandon the time travel feature #26070

jiaoew1991 opened this issue Aug 1, 2023 · 4 comments
Assignees
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@jiaoew1991
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

In the early versions, Milvus added the time travel feature. However, it did not achieve the expected results in terms of community feedback and actual effectiveness. Instead, it introduced many obstacles to stability and other feature designs. For example, when designing functions such as TTL, compaction, and garbage collection, compatibility with time travel must be considered, resulting in significant complexity.

Considering the reasons mentioned above, it is recommended to remove the time travel feature in version 2.3.0 upgrade.

Why is this needed?

No response

Anything else?

No response

@jiaoew1991 jiaoew1991 added the kind/enhancement Issues or changes related to enhancement label Aug 1, 2023
@jiaoew1991
Copy link
Contributor Author

jiaoew1991 commented Aug 1, 2023

We should follow the following steps to execute:

@xiaofan-luan
Copy link
Contributor

Agreed.
Remove timetravel can improve the search performance by

  1. pre calculate delete bitset
  2. sort the data by pk instead of timestamp and save pk index.

and maybe more ....

@jiaoew1991
Copy link
Contributor Author

The current handling of search and query for travel_ts is different.

  1. If the user does not set travel_ts, search will use max_ts. Previously, we attempted to change the default value to the query time, but it had a significant impact on the merging of search tasks because the merging conditions include whether the travel_ts of two search tasks are equal. Therefore, the current search does not guarantee mvcc.

  2. If the user does not set travel_ts in a query, proxy will use the query time, which ensures mvcc.

We can delete all travel_ts on the search path because consistency is not necessary for search, and performance is more important. However, the travel_ts of the query needs to be retained but renamed as mvcc. Consistency is necessary for subsequent scenarios such as deduplication of primary keys and delete by expr where querying is done before writing.

@jiaoew1991
Copy link
Contributor Author

finished it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Development

No branches or pull requests

2 participants