Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v23.2.x] Fixed large allocation in kafka::wait_for_leaders #16435

Conversation

vbotbuildovich
Copy link
Collaborator

Backport of PR #16287
Fixes: #16433, Fixes: #16434,

Previously we used a simple `std::vector` of futures to make
waiting for partition leaders concurrent. Using a vector has a drawback
when dealing with large number of topics and partitions since it may be
required to allocate large contiguous chunk of memory for a future
vector. In this particular case we may not use a fragmented vector or
chunked fifo as the `when_all` uses a plain vector internally.

To make sure no large chunk of memory is allocated to wait for the
partition leaders changed the logic to use
`seastar::max_concurrent_for_each`.

Fixes: redpanda-data#15908

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 8d0b584)
Sometimes it may happen that producer swarm is stopped after topic is
recreated leading to a test failure. Added check restarting the producer
if necessary

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit a18fdcd)
@vbotbuildovich vbotbuildovich added this to the v23.2.x-next milestone Feb 1, 2024
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label Feb 1, 2024
@piyushredpanda
Copy link
Contributor

/ci-repeat

@piyushredpanda
Copy link
Contributor

Hmm, this is hitting a new failure: TopicDeleteCloudStorageTest.topic_delete_unavailable_test @mmaslankaprv

@mmaslankaprv
Copy link
Member

one of the failures (more recent build) is not related and fixed here: #16495

@mmaslankaprv
Copy link
Member

The failure in this test is unrelated, it was closed some time ago here: #13235

@piyushredpanda piyushredpanda merged commit 1b807e4 into redpanda-data:v23.2.x Feb 8, 2024
22 of 24 checks passed
@piyushredpanda piyushredpanda modified the milestones: v23.2.x-next, v23.2.25 Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants