[#859][Improvement] Set MALLOC_ARENA_MAX in start-shuffle-server.sh

[rhh777]

What changes were proposed in this pull request?

Reduce memory usage for some versions of glibc. Prevent oom.

Why are the changes needed?

Fix: #859

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT

[jerqi]

Will it influence the performance of writing data? Have you test the patch in your local environment?

[jerqi]

Why do we choose 4 as our default value?
The comment is a little misunderstanding.

Default value is 8 * CORES

[rhh777]

Why do we choose 4 as our default value? The comment is a little misunderstanding.

Default value is 8 * CORES

Comments may be confusing. Default value is 8 * CORES. Is the default value for glibc. ShuffleServer set to 4 by default is a reference to hadoop/presto configuration.

[jerqi]

Why do we choose 4 as our default value? The comment is a little misunderstanding.

Default value is 8 * CORES

Comments may be confusing. Default value is 8 * CORES. Is the default value for glibc. ShuffleServer set to 4 by default is a reference to hadoop/presto configuration.

Could you modify the misunderstanding comment?

[rhh777]

Why do we choose 4 as our default value? The comment is a little misunderstanding.

Default value is 8 * CORES

Comments may be confusing. Default value is 8 * CORES. Is the default value for glibc. ShuffleServer set to 4 by default is a reference to hadoop/presto configuration.

Could you modify the misunderstanding comment?

yes

[codecov-commenter]

Codecov Report

Merging #860 (90be432) into master (0125fe3) will increase coverage by 2.14%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #860      +/-   ##
============================================
+ Coverage     56.57%   58.71%   +2.14%     
- Complexity     2174     2175       +1     
============================================
  Files           327      307      -20     
  Lines         15979    13619    -2360     
  Branches       1263     1263              
============================================
- Hits           9040     7997    -1043     
+ Misses         6430     5185    -1245     
+ Partials        509      437      -72     

see 21 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jerqi]

Will it influence the performance of writing data? Have you test the patch in your local environment?

According to prestodb/presto#8993, there will be subtle difference among different values of MALLOC_ARENA_MAX.

[jerqi]

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

[rhh777]

Will it influence the performance of writing data? Have you test the patch in your local environment?

Theoretically, in some cases, it will affect the performance. In my environment self-test, there is no obvious change, which may be due to the data volume. However, the situation in OOM no longer occurs.

[rhh777]

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

Does the Resident Set Size of the shuffle server process continuously increase in your environment？

[jerqi]

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

Does the Resident Set Size of the shuffle server process continuously increase in your environment？

We don't use the memory limit. So we don't find this issue. If you verified this pr in your production environment, we can merge it.

[jerqi]

LGTM, thanks @rhh777

[jerqi]

Merged to master.

[rhh777]

Will it influence the performance of writing data? Have you test the patch in your local environment?

I am a little doubtful about the effect of this pr. Because we don't have compress/decompress operation in the shuffle server.

Does the Resident Set Size of the shuffle server process continuously increase in your environment？

We don't use the memory limit. So we don't find this issue. If you verified this pr in your production environment, we can merge it.

Have you ever observed the Resident Set Size of the shuffle server process? In my environment, if I don't limit it, it just keeps growing.

[jerqi]

I never observe the resident set size. If I have time, I will observer it.

[jerqi]

Have you used the Uniffle in your production environment?

[rhh777]

Have you used the Uniffle in your production environment?

Currently deployed in a test environment, running tpcds tests.