Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][cli] Add command line option for configuring the memory limit #20663

Merged
merged 23 commits into from
Oct 10, 2023
Merged

[feat][cli] Add command line option for configuring the memory limit #20663

merged 23 commits into from
Oct 10, 2023

Conversation

JooHyukKim
Copy link
Contributor

@JooHyukKim JooHyukKim commented Jun 27, 2023

Fixes #15912

Motivation

Cli tools such as pulsar-perf don't contain a way to configure the Pulsar client memory limit.

As described in #15912

Modifications

Verifying this change

  • Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: https://github.com/JooHyukKim/pulsar/pull/14

@JooHyukKim JooHyukKim marked this pull request as draft June 27, 2023 16:21
@github-actions github-actions bot added the doc-required Your PR changes impact docs and you will update later. label Jun 27, 2023
@JooHyukKim
Copy link
Contributor Author

Apologies, my mistake 🙏🏼. I was suppposed to open PR in my personal repo first.

@@ -76,6 +76,9 @@ public static class RootParams {

@Parameter(names = { "--tlsTrustCertsFilePath" }, description = "File path to client trust certificates")
String tlsTrustCertsFilePath;

@Parameter(names = { "-m", "--memory", }, description = "Configure the Pulsar client memory limit")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this consistent with the performance commands?

Such as

@Parameter(names = { "-ml", "--memory-limit", }, description = "Configure the Pulsar client memory limit "
            + "(eg: 32M, 64M)")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gaoran10 Yes for sure! I was going to actually. Thank you for the reminder tho 🙏🏼

@JooHyukKim JooHyukKim requested a review from gaoran10 June 28, 2023 14:01
@JooHyukKim JooHyukKim marked this pull request as ready for review June 28, 2023 14:03

@Parameter(names = { "-ml", "--memory-limit", }, description = "Configure the Pulsar client memory limit "
+ "(eg: 32M, 64M)", converter = MemoryUnitToByteConverter.class)
public long memoryLimit = 0L;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one we might be able to utilize over at PR #20646

@JooHyukKim
Copy link
Contributor Author

We might want to proceed with #20691, so we can get rid of converters added here. Or refactor after merging might do.

…-line-option-for-configuring-the-memory-limit
@JooHyukKim JooHyukKim marked this pull request as draft July 16, 2023 10:49
@JooHyukKim
Copy link
Contributor Author

Let's wait til #20782 gets merged so we can make use of Converter utils.

…-line-option-for-configuring-the-memory-limit
@JooHyukKim
Copy link
Contributor Author

Blocked by #20782

@Technoboy- Technoboy- added this to the 3.2.0 milestone Jul 31, 2023
@JooHyukKim JooHyukKim marked this pull request as ready for review August 23, 2023 12:49
@JooHyukKim JooHyukKim marked this pull request as draft August 23, 2023 13:24
@JooHyukKim JooHyukKim closed this Aug 23, 2023
@JooHyukKim
Copy link
Contributor Author

Closing Note

Temporarily closing due to PIP-280 work is taking longer than expected.
Anyone can pick up where this PR left off, not much left to do, just import pulsar-cli-utils module.

@JooHyukKim JooHyukKim reopened this Sep 4, 2023
@JooHyukKim JooHyukKim marked this pull request as ready for review September 4, 2023 11:37
Copy link
Contributor

@gaoran10 gaoran10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work!

@JooHyukKim
Copy link
Contributor Author

OWASP dependency workflow failing here, and also in the other PR. As a regular commiter, this many number of CVE's to fix is overwhelming 😭

@tisonkun
Copy link
Member

@JooHyukKim If the OWASP failures are unrelated to this patch, it's not a blocker. Don't worry.

I just too busy to give a review 🤣

@JooHyukKim
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@JooHyukKim
Copy link
Contributor Author

JooHyukKim commented Sep 22, 2023

I just too busy to give a review

Right (wink)(wink) 😆 thanks always! 🙏🏽🙏🏽

@JooHyukKim JooHyukKim marked this pull request as draft September 30, 2023 13:13
@JooHyukKim
Copy link
Contributor Author

Converted to draft PR as we have couple of failures to fix.
Anyone is welcome to pick up and go on.

@tisonkun tisonkun marked this pull request as ready for review October 10, 2023 07:57
@tisonkun
Copy link
Member

Let me invesitgate the license issue.

Signed-off-by: tison <wander4096@gmail.com>
@codecov-commenter
Copy link

Codecov Report

Merging #20663 (dc5eed4) into master (8438e43) will increase coverage by 36.35%.
Report is 7 commits behind head on master.
The diff coverage is 86.20%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #20663       +/-   ##
=============================================
+ Coverage     36.84%   73.20%   +36.35%     
- Complexity    12229    32500    +20271     
=============================================
  Files          1699     1888      +189     
  Lines        130559   140251     +9692     
  Branches      14264    15444     +1180     
=============================================
+ Hits          48106   102670    +54564     
+ Misses        76121    29480    -46641     
- Partials       6332     8101     +1769     
Flag Coverage Δ
inttests 24.14% <48.27%> (-0.19%) ⬇️
systests 24.67% <37.93%> (-0.06%) ⬇️
unittests 72.50% <86.20%> (+40.55%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...org/apache/bookkeeper/mledger/LedgerOffloader.java 72.22% <ø> (+72.22%) ⬆️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java 80.63% <100.00%> (+32.31%) ⬆️
...e/bookkeeper/mledger/impl/NullLedgerOffloader.java 35.71% <ø> (+7.14%) ⬆️
...areness/IsolatedBookieEnsemblePlacementPolicy.java 87.02% <100.00%> (+87.02%) ⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java 71.81% <100.00%> (+30.29%) ⬆️
...n/java/org/apache/pulsar/broker/service/Topic.java 36.36% <ø> (+28.03%) ⬆️
...org/apache/pulsar/client/cli/PulsarClientTool.java 68.45% <100.00%> (+12.90%) ⬆️
...esystem/impl/FileSystemManagedLedgerOffloader.java 68.04% <ø> (ø)
...d/jcloud/impl/BlobStoreManagedLedgerOffloader.java 75.18% <ø> (ø)
...ava/org/apache/pulsar/broker/service/Producer.java 82.09% <75.00%> (+19.66%) ⬆️

... and 1441 files with indirect coverage changes

@@ -41,7 +41,7 @@ if [ -z $TARBALL ]; then
exit 1
fi

JARS=$(tar -tf $TARBALL | grep '\.jar' | grep -v 'trino/' | grep -v '/examples/' | grep -v '/instances/' | grep -v pulsar-client | grep -v pulsar-common | grep -v pulsar-package | grep -v pulsar-websocket | grep -v bouncy-castle-bc | sed 's!.*/!!' | sort)
JARS=$(tar -tf $TARBALL | grep '\.jar' | grep -v 'trino/' | grep -v '/examples/' | grep -v '/instances/' | grep -v pulsar-client | grep -v pulsar-cli-utils | grep -v pulsar-common | grep -v pulsar-package | grep -v pulsar-websocket | grep -v bouncy-castle-bc | sed 's!.*/!!' | sort)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh... this was it. Thank you @tisonkun ! 😭

@tisonkun tisonkun merged commit 903e223 into apache:master Oct 10, 2023
44 of 45 checks passed
vinayakmalik95 pushed a commit to tmdc-io/pulsar that referenced this pull request Oct 12, 2023
…pache#20663)

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-required Your PR changes impact docs and you will update later. ready-to-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Cli tools] Add command line option for configuring the memory limit
5 participants