Skip to content

Commit

Permalink
[SMALLFIX] Use field property for 'AbstractCmdRunner'
Browse files Browse the repository at this point in the history
### What changes are proposed in this pull request?

Use field property 'mActiveJobs' for 'AbstractCmdRunner' instead of
the default value DEFAULT_ACTIVE_JOBS which is hardcoded with 3000.

### Why are the changes needed?

Currently, the number of active jobs are limited at 3000 which is not
enough when we have lots of loading tasks on huge cluster. We will add
command line args to allow user pass the limit of active jobs
(set the value of 'mActiveJobs') the  in the future.

### Does this PR introduce any user facing changes?

N/A
  • Loading branch information
iAlaska committed Jul 10, 2023
1 parent 73cea83 commit 1ac0270
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ private void submitDistLoad(List<URIStatus> pool, int replication,
Set<String> workerSet, Set<String> excludedWorkerSet,
Set<String> localityIds, Set<String> excludedLocalityIds,
boolean directCache, CmdInfo cmdInfo) {
if (mSubmitted.size() >= DEFAULT_ACTIVE_JOBS) {
if (mSubmitted.size() >= mActiveJobs) {
waitForCmdJob();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ private void copy(AlluxioURI srcPath, AlluxioURI dstPath, boolean overwrite, int
// Submit a child job within a distributed command job.
private void submitDistCp(List<Pair<String, String>> pool, boolean overwrite,
WriteType writeType, CmdInfo cmdInfo) {
if (mSubmitted.size() >= DEFAULT_ACTIVE_JOBS) {
if (mSubmitted.size() >= mActiveJobs) {
waitForCmdJob();
}

Expand Down

0 comments on commit 1ac0270

Please sign in to comment.