Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3.4] mvcc: reduce count-only range overhead #15099

Merged
merged 4 commits into from
Jan 18, 2023

Conversation

fuweid
Copy link
Member

@fuweid fuweid commented Jan 13, 2023

Backports

The #11771 's test case needs #11743 change. So I backport two prs in one pr.

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.


Related to #15072

mlmhl and others added 3 commits January 13, 2023 16:31
(cherry picked from commit aa7b056)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 730f3f1)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 3594ab9)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
@fuweid fuweid closed this Jan 13, 2023
@fuweid fuweid reopened this Jan 13, 2023
@fuweid fuweid closed this Jan 13, 2023
@fuweid fuweid reopened this Jan 13, 2023
@fuweid fuweid closed this Jan 13, 2023
@fuweid fuweid reopened this Jan 13, 2023
@serathius
Copy link
Member

We don't backport features.

@fuweid
Copy link
Member Author

fuweid commented Jan 14, 2023

@serathius Thanks for the comment.

Before I file #15072 pr, I checked the contributor document about the backport policy.

The commits should be restricted to bug fixes and security patches.

Except that it introduces new feature --count-only into the etcdctl, the CountRevisions is new internal interface to prevent unnecessary memory allocation. It won't change the existing interface to end-user. So from the point of view of interface, could the enhancement be considered to be bugfix? :) Thanks

And I'm also trying to backport #12933 and wondering how to handle this case.

We are running etcd-3.4 (upgraded from 3.3) and it provides concurrent read transaction feature.
But with the high read-tx ratio, the etcd-3.4 needs more memory than etcd-3.3. It is easy to run into memory spike.
It seems that it is kind of memory regression. If the following release has a patch to fix the memory issue and the patch is very suitable to older release, is it possible to consider the patch as bugfix?

Look forward to the release managers' comments. Thanks!

@ahrtr
Copy link
Member

ahrtr commented Jan 16, 2023

In principle, we do not backport features. But we should always make judgement based on benefit and risk.

The etcdctl CLI flag --count-only is new to 3.4, but clientv3.WithCountOnly() is already in release-3.4 there for a long time. It's an issue to me that previous implementation of this feature (CountOnly) in 3.4 isn't complete. So I am not against to make it complete as long as it's a simple & minor change.

The simple improvement in (ti *treeIndex) CountRevisions can definitely reduce the memory usage when users perform only Count-Only operation, because we don't need to cache all K/V pairs any more. Again, it's a simple & minor change. So I am not against adding it in release-3.4. But @fuweid please backport the related unit test case (e.g. TestIndexRevision) or add a new unit test as well.

It is kind of backport from etcd-io#14124.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@fuweid
Copy link
Member Author

fuweid commented Jan 16, 2023

Thanks @ahrtr for the comment! Updated the TestIndexRevision ut and it is ready to review. PTAL.

Copy link
Member

@ahrtr ahrtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thank you @fuweid

Please also add a changelog item for 3.4.24.

@fuweid
Copy link
Member Author

fuweid commented Jan 16, 2023

Please also add a changelog item for 3.4.24.

Will do. Thanks!

@fuweid
Copy link
Member Author

fuweid commented Jan 17, 2023

ping @serathius @ptabor ~ is it acceptable?

@ahrtr ahrtr merged commit a1d1af5 into etcd-io:release-3.4 Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants