Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Protect get-snapshots APIs against excessive memory use #104607

Open
DaveCTurner opened this issue Jan 22, 2024 · 2 comments
Open

Protect get-snapshots APIs against excessive memory use #104607

DaveCTurner opened this issue Jan 22, 2024 · 2 comments
Labels
>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@DaveCTurner
Copy link
Contributor

The get-snapshots APIs accumulate the details of all matching snapshots in memory before responding. In a high-index-count cluster with a long retention period on snapshots this can add up to GiB+ of heap on the elected master to request all snapshots.

We should make this API more efficient for sure, see #95345, but we should also protect the cluster against accidental use of this API to request too many snapshots by rejecting such requests before they can cause harm.
@DaveCTurner DaveCTurner added >bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Jan 22, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 22, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@geekpete geekpete mentioned this issue Feb 5, 2024
Open
@DaveCTurner
Copy link
Contributor Author

In fact once #104851 is done we could also make this API run in near-constant memory, streaming its results in chunks all the way from the master node to the client.

@DaveCTurner DaveCTurner mentioned this issue Feb 19, 2024
Merged
@DaveCTurner DaveCTurner mentioned this issue Apr 8, 2024
Merged
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Aug 11, 2024
@DaveCTurner
Add GetSnapshotsIT#testAllFeatures
7b54f9f 
The features of get-snapshots API are all tested in isolation or small
combinations, but there's no one test which pins down exactly how they
all interact. This commit adds such a test, to verify that any future
optimization work preserves the observable behaviour.

Relates elastic#95345
Relates elastic#104607
@DaveCTurner DaveCTurner mentioned this issue Aug 11, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue Aug 13, 2024
@DaveCTurner
Add GetSnapshotsIT#testAllFeatures (#111786)
d2bd374 
The features of get-snapshots API are all tested in isolation or small
combinations, but there's no one test which pins down exactly how they
all interact. This commit adds such a test, to verify that any future
optimization work preserves the observable behaviour.

Relates #95345 Relates #104607
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Sep 4, 2024
@DaveCTurner @cbuescher
Add GetSnapshotsIT#testAllFeatures (elastic#111786)
e1ea785 
The features of get-snapshots API are all tested in isolation or small
combinations, but there's no one test which pins down exactly how they
all interact. This commit adds such a test, to verify that any future
optimization work preserves the observable behaviour.

Relates elastic#95345 Relates elastic#104607
davidkyle pushed a commit to davidkyle/elasticsearch that referenced this issue Sep 5, 2024
@DaveCTurner @davidkyle
Add GetSnapshotsIT#testAllFeatures (elastic#111786)
2125deb 
The features of get-snapshots API are all tested in isolation or small
combinations, but there's no one test which pins down exactly how they
all interact. This commit adds such a test, to verify that any future
optimization work preserves the observable behaviour.

Relates elastic#95345 Relates elastic#104607
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DaveCTurner @elasticsearchmachine